Training systems and reproducible experiments

A typical step obtains a batch, computes predictions and loss, accumulates derivatives, updates parameters, and advances the intended schedule. Define where each boundary occurs. Does the learning-rate schedule count examples, optimizer updates, or epochs? Does a skipped nonfinite update advance it? Does the last short batch have the same per-example weight? These choices can change the mathematical experiment even when every line executes without an exception.

Checkpoint enough state to continue that experiment: model parameters and buffers, optimizer moments, scheduler position, random-number generator states, and data-sampling position. Mixed-precision training may also require loss-scaler state. The exact state depends on the pipeline, but restarting with only weights is generally a new optimization history. Associate each checkpoint with the configuration, code revision, data version, and evaluation results rather than trusting a filename such as final-final.

When memory requires microbatches, match the global objective rather than merely repeating backward. If one microbatch contains two valid target tokens and another contains six, their mean token losses should be weighted by two and six. Equal averaging of the two means gives the small microbatch too much weight. Exclude padding consistently from both numerator and denominator, and make a deliberate choice between per-token and per-sequence weighting when sequence lengths differ.

Gradient accumulation reproduces a large-batch gradient only under suitable assumptions: parameters must stay fixed until the combined update, the loss weights must match, and the computation must behave consistently across microbatches. Batch-dependent normalization can change forward values, and stochastic layers or floating-point reduction order can prevent exact equality. Measure numerical differences instead of promising bitwise identity. Separately profile input preparation, transfers, forward, backward, and updates before optimizing the wrong bottleneck.

Set and record relevant random seeds, including data workers and other numerical libraries, and use deterministic algorithms where required and available. A seed does not guarantee identical results across hardware, library versions, or kernels. Save environment details and tolerate justified floating-point differences in regression checks. Repeatability helps isolate bugs; multiple seeds and independent evaluation assess whether an improvement survives ordinary variation. A single reproducible lucky run is still a lucky run.

Evaluation has its own state. In PyTorch, model.eval() changes behavior of modules such as dropout and batch normalization; it does not disable gradient tracking. A no-grad or inference context controls autograd separately, with its own usage constraints. Evaluate the frozen checkpoint under the documented preprocessing, target mask, and decoding policy. Log raw aggregate numerators and denominators, not only batch averages. Finally, compare quality at a stated compute or time budget so an apparent improvement is not just more training.