Control optimizer steps and effective exposure

With D data-parallel workers, an unpacked per-worker microbatch of b sequences, and A accumulation microsteps, a full update processes D×b×A sequences when all workers contribute complete microbatches. This count does not establish an identical gradient to one large batch unless loss normalization, stochastic behavior, and optimizer stepping are equivalent. Last partial batches, dropped records, packing, and variable target lengths can change both the count and the effective weighting. Record realized rather than merely configured exposure.

Selected target tokens determine supervised mass under a token-mean objective. Two microbatches with the same number of sequences can have different target counts because prompt lengths, answer lengths, and masking differ. Averaging each microbatch's mean equally can unintentionally overweight short ones. Use the trainer's supported consistent reduction for the intended objective and inspect it for the pinned version. An accumulation factor is not permission to update optimizer moments on every microstep when the experiment assumes one update after accumulation.

Warmup and decay may be indexed by optimizer steps, epochs, or another implementation-defined measure. Changing accumulation or data size changes how many tokens arrive before the same step number. Optimizer state, weight decay, gradient clipping, precision, and LoRA scaling also affect updates. Standard alpha/r adapter scaling is not a universal replacement for learning-rate selection, and other adapter variants use different rules. Check actual trainable tensors and gradients rather than assuming a configuration file matched every intended target.

Use controlled sweeps with a small, justified search budget and a held-out validation protocol. Record random seeds, early-stop rules, and how checkpoints are selected. Repeatedly picking the best result on one small test set overfits the selection process. A smaller learning rate can still accumulate a large behavioral change over enough steps, and a larger one can damage retention without obvious immediate training instability. There is no single rate or epoch count that guarantees safe adaptation across datasets and bases.

Log completed optimizer updates, selected target tokens, gradient norms, validation outcomes, and peak resource use. Check for skipped updates, nonfinite gradients, and batches with no supervised targets. Preserve the exact data/split manifest and preprocessing configuration so exposure comparisons remain meaningful. If a run resumes, restore the intended optimizer, scheduler, sampler, and random state where reproducibility requires it. Loading adapter weights alone is not necessarily a faithful continuation of the training experiment.

The optimization lab is a bounded smooth illustration. Reset and set both η=0.5, x₀=2, predict first update x=1 and lower quadratic loss. Reset and re-set η=2.5, x₀=2, predict first update x=−3 and higher loss on f(x)=x²/2, then inspect as iterations advance with Play or Step. These reachable controls show this fixed model's unstable regime, not an LLM optimizer or learning-rate recommendation. It has no Adam state, token batches, LoRA scaling, or retention tasks.