Distributed training parallelism

Distributed data parallelism, or DDP, places a model replica on each worker; the application must arrange data partitioning. Workers synchronize gradients so compatible optimizer updates keep replicas aligned. PyTorch DDP buckets gradients to overlap communication with backward computation. For equal local batch sizes, averaging local mean gradients produces the global mean. With unequal valid-token or example counts, explicitly normalize for the intended global loss; worker averaging does not automatically supply those weights.

Global batch size is local batch size times data-parallel workers times accumulation steps, assuming equal batches. Increasing it changes optimization, so compare convergence at a matched token or example budget, not just steps per second. Accumulation trades activation memory and synchronization frequency against update frequency. All workers must participate in compatible collective sequences; one rank skipping a backward pass while others synchronize can hang a job rather than simply omit its contribution.

A model's training memory includes weights, gradients, optimizer state, activations, temporary buffers, and communication workspaces. FSDP shards persistent model state over a data-parallel group. In the usual fully sharded execution, workers gather parameters for computation and reduce-scatter gradients afterward. Peak memory still includes temporarily gathered layers and activations. Dividing the checkpoint size by device count is therefore an incomplete peak-memory estimate; the wrapping and prefetch schedule affect which buffers coexist.

Tensor parallelism partitions operations inside a layer, introducing collectives along the forward and backward critical paths. It benefits from fast links between cooperating devices. Pipeline parallelism assigns layers to stages and sends activations between them; microbatches reduce idle pipeline bubbles but introduce scheduling and memory tradeoffs. These methods can be combined in separate dimensions. Put frequent, latency-sensitive communication on fast links and examine stragglers before assuming a larger cross-node group will improve throughput.

For an ideal ring all-reduce, each rank transfers approximately $2(p-1)G/p$ bytes for a gradient buffer of $G$ bytes. Dividing by effective link throughput gives a bandwidth-time estimate. This is a model of a particular algorithm, not a guarantee about a collective library: topology, startup latency, contention, and chunking matter. Use effective bytes per second measured under the job's communication pattern rather than the link's advertised bit rate.

A step's exposed synchronization cost is communication that remains after possible overlap with computation. Perfect overlap is limited by dependency order, not just by total compute time. Profile timelines and compare step time, examples per second, peak memory, and validation behavior. Strong scaling holds global work fixed; weak scaling grows work with workers. Report which experiment you ran. A fast training step that silently drops examples, changes loss normalization, or resumes incorrectly is not a successful infrastructure optimization.