Engineering / advanced
Replicate computation and shard training state
Derive gradient normalization, synchronization, bucketing, and parameter-state sharding schedules with explicit memory and communication calculations.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Replicate computation and shard training state
Derive gradient normalization, synchronization, bucketing, and parameter-state sharding schedules with explicit memory and communication calculations.
- 01
Replicated training: agree on the gradient and the update
Derive default DDP gradient averaging and identify the initialization, optimizer, and collective assumptions that keep replicas aligned.
28 min - 02
Unequal token counts require one global denominator
Derive valid-token weighting across workers and accumulation windows, including zero-token cases and compatible count collectives.
28 min - 03
Gradient buckets: overlap follows a dependency schedule
Analyze readiness, stable collective order, and communication tails without assuming asynchronous calls eliminate dependencies or preserve buffer lifetimes automatically.
28 min - 04
Accumulation changes communication, not the intended batch objective
Derive a window-level update and scope DDP no_sync, mixed-precision scaling, clipping, and optimizer boundaries precisely.
28 min - 05
Optimizer-state sharding: assign update ownership explicitly
Calculate persistent state savings while preserving parameter identity, optimizer state, and post-update replica synchronization.
28 min - 06
Gradient sharding: reduce globally and retain the owner slice
Derive reduce-scatter normalization, gradient ownership, and global clipping without confusing a local shard norm with a whole-model norm.
28 min - 07
Parameter sharding: materialize only what the next operation needs
Trace FULL_SHARD parameter availability and all-gather traffic while retaining module-fit, shared-parameter, and optimizer-mapping constraints.
28 min - 08
Prefetching: hide communication only inside a live-memory budget
Analyze allocation lifetimes and overlap windows instead of estimating peak memory from steady-state shards or CPU enqueue timing.
28 min - 09
Equivalence tests: compare the update state, not only the loss
Build canonical small-step references for gradients, parameters, moments, clipping, masks, and precision-dependent drift across replicated and sharded execution.
28 min - 10
Worker failure: restart a coherent training state
Distinguish fixed-group collective failure from uneven inputs and elastic restart, then budget replay and membership-dependent objective changes.
28 min