Count the tokens that actually contribute to each update

Let a microbatch contain a known number of eligible prediction targets after serialization, shifting, padding, and boundary masks. Sum those counts across the microbatches and ranks contributing to one optimizer update. This is the update's valid-token batch. A product of data-parallel ranks, accumulation steps, sequences, and sequence length equals that count only under additional full-occupancy and eligibility assumptions. Tensor-parallel replicas holding pieces of the same examples are not additional independently trained tokens.

Also separate attempted updates, successful applied updates, consumed data, and accepted checkpoint progress. An overflow-skipped step can consume data without changing weights; a restart can recompute work already performed but not preserved in the recovered trajectory. Those counters answer different resource and exposure questions. Log the conventions explicitly. A reported tokens-per-second number should identify whether it measures processed input slots, valid targets, or another quantity, and whether it includes recovery and skipped-update work.

For an unsmoothed unweighted token objective, sum local negative log-likelihoods and divide by the global valid-token count. Averaging local means instead weights each local batch equally, even if one has twice as many scored targets. The same distinction applies to gradients at a fixed parameter state. If a data-parallel system averages rank gradients, each rank's local summed-loss gradient needs scaling consistent with that average and the global denominator. Framework-provided averaging and accumulation conventions must be accounted for once, not guessed.

For example, under an explicit average-over-R-ranks reduction, scaling each rank's accumulated loss sum by R/T gives the gradient of the global token mean. If a trainer already divides by accumulation count or uses a different reducer, that formula must be adapted. PyTorch 2.9 documents DDP's gradient behavior and mixed-precision accumulation requirements. Validate the chosen implementation against a small single-process concatenated-batch reference, including deliberately unequal valid counts and the final incomplete accumulation group.

Even with the same global token count, changing sequence lengths, document mixture, dropout realization, or reduction order can alter gradients. Increasing the valid-token batch can reduce some sampling variation but does not imply a universal linear learning-rate scaling rule. Keep the actual objective, optimizer, schedule clock, and token-exposure ledger attached to any batch-size experiment. For remainder groups, choose a declared policy such as correct renormalization, carrying work forward, or dropping it; silently using the full-group denominator changes update magnitude.

The sampling lab isolates one variance relationship under independent identical Bernoulli observations. Reset, set both p=0.5 and n=20, predict mean 0.5 and sample-mean variance 0.0125, then Play or Step. Reset and set both p=0.5,n=100 for variance 0.0025. The bars are exact binomial masses, and playback only highlights outcomes. Tokens and gradients in a document are correlated and heterogeneous, so this instrument does not infer an effective independent sample size or an optimizer scaling law for language-model training.