Complete training-state inventory

Treat the training state as more than a tensor collection. Parameters and persistent model buffers affect the forward computation; optimizer moments, master weights, and step counters affect the update. A scheduler supplies the next learning rate, while mixed-precision scaling can decide whether an update is skipped. If the checkpoint lies inside a gradient-accumulation window, accumulated gradients and the microstep position may matter too. A clean update boundary can simplify this inventory, but only when the actual loop makes those intermediate quantities unnecessary.

Inventory randomness and input progress separately. Python, NumPy, CPU generators, CUDA generators, augmentation workers, samplers, shuffle buffers, sequence packers, and external data readers can each carry state. Record a dataset version and logical consumed position, not just the name of a mutable data directory. A rank-local cursor can be meaningful only with the associated sharding policy. Decide whether recovery repeats or skips prefetched records; a fetch already issued is not necessarily a sample whose gradient contributed to a completed update.

Checkpoint metadata should identify model configuration, parameter names, tokenizer or feature schema, optimizer type and group settings, scheduler convention, software versions, and supported parallel layout. Record training/evaluation mode and any exponential-moving-average weights or custom callback state that affects the next transition. A tensor with the right shape can still represent a differently ordered vocabulary or expert. Compatibility is a tested contract rather than an inference from a successful deserialization call. Explicitly distinguish an intentional optimizer reset for a new experiment from faithful continuation of the previous run, and give those outputs different recovery labels.

PyTorch 2.14 DCP provides distributed state-dictionary machinery; it does not discover every application-specific cursor or configuration dependency. FSDP state handling must map optimizer state to the intended logical parameters, not to incidental object identities from another process. DeepSpeed weight consolidation is useful for exporting parameters but should not be mistaken for a complete restoration of its engine and training history. Extend the application state deliberately and version that schema so a loader can reject an unsupported interpretation before training begins.

A useful inventory names an owner, serialization form, restore action, and oracle for every state category. A scalar global-step field needs a definition: does it count attempted updates, successful optimizer updates, consumed microbatches, or tokens? Those counters can diverge after an overflow or retry. The state of an external side effect, such as a metric stream, may require an idempotency policy instead of being embedded in a tensor file. Record the boundary precisely enough that an operator can explain the next scheduler and sampler actions.

The optimization instrument illustrates dependence on update rules, without saving anything. Reset, set both η=0.5 and x₀=2, predict x₂=0.5, then press Step twice. Separately Reset, set both η=0.25 and x₀=2, predict x₂=1.125, then press Step twice. Each Step advances one iteration of the fixed quadratic descent model. The differing results show why a transition rule belongs in a recovery specification. This toy has no optimizer moments, checkpoint files, sampler, or mixed-precision scaler; an application inventory must cover those separately.