Microbatch size and gradient semantics

For a token-mean language-model objective, count the valid supervised tokens rather than padded positions or merely sequences. If microbatch m contains n_m valid tokens and reports its own token-mean gradient, its contribution to the whole-batch token mean is weighted by n_m. Averaging those means equally is correct only when the valid counts match. A microbatch with no valid targets needs an explicit handling policy that avoids division by zero while preserving the intended objective and collective participation. If the runtime expects static microbatch shapes, pad and mask a tail or use a supported batching policy rather than assuming a shorter final tensor is legal.

Keep the count and sum consistent across pipeline and data-parallel boundaries. A loss sum accumulated over microbatches can be divided by the complete valid-token count once, or each contribution can be normalized to the same global denominator. Those approaches require matching conventions in the communication layer. If a data-parallel reducer averages replica gradients, an already global-normalized local sum needs the corresponding compensation; blindly averaging again introduces another factor. Write the algebra before adding pipeline scheduling or mixed-precision loss scaling.

PyTorch 2.8 pipeline schedules expose scale_grads, which by default scales accumulated gradients by the number of microbatches. That behavior matches a suitable equal-microbatch mean-loss setup; it does not automatically implement unequal token weighting. A sum-loss design can disable that averaging and apply the intended full-batch normalization exactly once. Check the version-specific schedule and loss implementation together, including any external training loop that rescales gradients. A default is not evidence that a particular masked objective is normalized correctly.

Accumulate all intended contributions before applying the logical optimizer update. Gradient clipping, weight decay, momentum updates, and learning-rate progression performed once per microbatch can define a different algorithm from performing them once per effective batch. Mixed-precision scaling and overflow handling must also agree across the ranks whose update is coupled. Even with mathematically equivalent sums, floating-point reduction order can cause small differences; state a justified numeric tolerance rather than promising bitwise identity merely because the global batch size matches.

At fixed global example batch, increasing M reduces b and can reduce a modeled pipeline bubble while making matrix kernels less efficient. Smaller messages can also increase communication startup overhead relative to payload. Batch normalization computes statistics over its actual execution group, so microbatching can change the function itself even when gradient weighting is correct. Random masks and stateful operations likewise need a defined equivalence target. Validate both losses and gradients using unequal valid counts and a short final microbatch before evaluating throughput.

The Roofline lab isolates an assumed change in arithmetic intensity. Reset, set intensity to 2 and bandwidth to 80, predict the synthetic ceiling, then Step: 160 GFLOP/s. Compare a fresh Reset with both intensity at 8 and bandwidth at 80 before Step: 640 GFLOP/s. Its fixed compute ceiling is 1000. This is not a measured microbatch-size sweep and contains no gradient accumulation, token masks, pipeline bubbles, or optimizer semantics.