Communication-avoiding training

A collective's cost includes message startup, byte transfer, local reductions, and contention. An ideal ring all-reduce sends and receives chunks over multiple rounds. Its per-rank bytes sent approach twice the gradient buffer size as worker count grows; the same byte count is received. With simultaneous full-duplex transfer, avoid adding both directions and then dividing by a one-direction bandwidth. This is not an exact runtime formula for every algorithm or topology. Small messages can be latency dominated, while a large buffer may be bandwidth dominated or overlap with useful computation.

Exposed communication is the portion that extends the dependency graph's critical path after overlap. Removing a transfer that was completely hidden may leave step time unchanged. Measure a timeline rather than adding all kernel and collective durations, since concurrent intervals cannot be summed as elapsed time. A compression method can also consume compute resources needed for backward execution, reducing overlap. Report both measured communication bytes and total training behavior.

Gradient accumulation sums contributions from several microbatches before one optimizer update. With correct normalization and compatible loss computation, it can reproduce a larger batch gradient at fixed parameters. Batch-dependent layers, stochastic operations, clipping placement, and unequal valid-token counts require care. Accumulating four microbatches is not equivalent to taking four optimizer steps: parameters and optimizer state remain unchanged between those backward passes.

Local training steps go further: each worker changes its own parameters before synchronization, so local gradients are evaluated at different states. Averaging parameters afterward generally does not equal centralized gradient descent. Statistical heterogeneity and step size affect drift. A claim of communication reduction must therefore include convergence evidence. Hold examples or tokens, quality target, and tuning budget explicit rather than comparing only synchronization counts.

Low-rank gradient compression represents a matrix update through thinner factors. Rank determines transmitted size and approximation quality; the useful rank depends on the gradient structure. Error feedback retains a residual so omitted information can influence later updates, but it does not make every individual compressed step exact or guarantee arbitrary training behavior. Compression needs compatible collective operations and may require extra synchronizations for its factors.

A credible comparison reports added compression time, residual memory, synchronization count, model quality, and total examples processed. Test across model layers and network regimes instead of assuming a fixed compression ratio implies a fixed speedup. If communication is already small or overlapped, the extra kernels can make the job slower. Include numerical checks for residual growth and nonfinite values, and compare time to target quality with a tuned uncompressed baseline.