Accumulation changes communication, not the intended batch objective

With equal token counts in every rank and microbatch, dividing each local mean loss by the number of microbatches gives the intended window mean after default DDP averaging. With unequal valid counts, that shortcut fails: use the global eligible-token denominator across the whole window and the local summed-loss scaling derived earlier. Either way, parameters remain at the same pre-update values while those microbatch gradients accumulate. Clear old gradients at the intended window boundary, not between contributing microbatches, and perform the optimizer update only after the complete synchronized contribution is ready.

Calling the optimizer between microbatches evaluates later gradients at different parameters and advances optimizer state more often. Even stateless SGD on a curved loss need not match the accumulated update; momentum, adaptive moments, weight decay, and step-based schedules create additional differences. A smaller final window also needs its actual denominator. Do not blindly divide by a nominal accumulation count when an epoch tail contains fewer contributing tokens. Make that tail policy explicit and compatible on all ranks instead of letting one rank silently stop early.

DDP no_sync suppresses gradient synchronization within its context while allowing local gradient accumulation. The forward pass must be inside the context as well as the corresponding backward work; wrapping only backward is not the documented protocol. A final normal forward/backward outside the context synchronizes accumulated gradients. Every rank must reach compatible synchronization boundaries. Skipping a final synchronized microbatch or clearing the accumulated buffer first defeats the intended update even when earlier local backward calls succeeded.

Communication suppression does not make other collectives disappear. Count reductions, synchronized layers, or application collectives still need matching participation and order. Also distinguish DDP from FSDP: the PyTorch 2.9 FSDP wrapper documents that no_sync can retain full rather than sharded gradients and increase memory. CPU offload introduces further supported-accumulation restrictions. Verify the chosen wrapper and strategy rather than copy a DDP accumulation recipe into a sharded deployment and assume the same memory footprint and synchronization behavior.

For mixed precision, use one consistent loss scale while accumulating the window so that contributions are commensurate. Unscale only after all contributions are present, and apply gradient clipping to the intended synchronized unscaled global gradient before the single optimizer step. Updating the scaler or clipping each microbatch separately can change the algorithm. Ensure all replicas make compatible overflow and skipped-step decisions; otherwise optimizer state, schedules, and parameters can diverge despite a shared nominal step counter. Include those edge cases in a small reference test.

The optimization lab demonstrates why step boundaries matter in a simpler setting. Reset, explicitly set step size 0.1 and initial position two, predict the first two iterates, then Step twice. Compare a fresh Reset with step size 0.2 and initial position two before one Step. The quadratic gives 1.8 then 1.62 versus 1.6. Two sequential updates evaluate a changing gradient. This is not an accumulation implementation, has no microbatches or no_sync, and models no momentum or loss scaler; it only makes the update-boundary distinction concrete.