Unequal token counts require one global denominator

Define a local sum over exactly the tokens included by the training objective. Exclude ignored targets and padding consistently, but do not remove examples merely because their current loss is inconvenient. Reduce the eligible counts across workers to obtain one positive global denominator. For an accumulation window, that denominator must cover the entire window, not whichever microbatch happened to finish last. Preserve the count and masks with the input fixture so that a single-process reference uses precisely the same token multiset.

The desired gradient is the sum of local loss-sum gradients divided by the global count. DDP adds its own division by world size, so one valid construction scales each local summed loss by world size divided by the global count before backward. Equivalently, a carefully controlled post-reduction rescaling can implement the same algebra, but it must occur before nonlinear transformations such as clipping. Do not divide by the global count and then accidentally leave DDP’s additional world-size division uncompensated.

If a worker with two tokens and another with six tokens each contributes one local mean, a token on the smaller worker receives three times the weight of a token on the larger worker. The reducer cannot infer this mistake from tensor shape or successful communication. All ranks can agree perfectly on the wrong gradient. Equal local batch lengths are also insufficient when loss masks differ. Count the objective’s actual contributing targets and test an intentionally uneven fixture rather than rely on a balanced synthetic benchmark.

Zero eligible tokens need a coordinated policy. If the global count is positive but one rank has none, that rank still needs a compatible forward/backward and collective schedule, with a correctly connected zero contribution where required by the model and reducer configuration. It must not unilaterally skip backward. If the whole window has zero count, every rank must take the same defined skip or error path before division and keep optimizer, scaler, and scheduler state consistent. A divide-by-zero result is not a neutral contribution.

The count reduction is itself a collective with matching order and participation requirements. Every worker must execute it in the same intended window, including ranks whose local count is zero. DDP’s uneven-input join mechanism cannot automatically shadow arbitrary application collectives such as an extra token-count all-reduce. Use a supported coordinated protocol or terminate consistently when inputs are exhausted; do not insert a count collective on only the still-active ranks. Validate both the gradient values and the collective trace for the edge cases.

Use the function lab to isolate scalar weighting. Reset, explicitly set coefficient one and position two, predict y and slope, then Play or Step. For a separate run Reset and explicitly set coefficient 0.5 and position two before predicting again. The fixed parabola’s slope changes from four to two because scalar multiplication affects derivatives as well as loss values. This instrument contains no tokens, masks, global counts, DDP division, or distributed reductions. The numerical token fixture below supplies that accounting rather than asking the parabola to simulate it.