Optimizer, RNG, and data-cursor recovery

Momentum and adaptive optimizers carry quantities that cannot generally be inferred from current parameters. A zeroed moment buffer changes the next update even if the gradient and learning rate are identical. Step counters affect bias corrections and schedules; parameter-group identities affect which settings apply to which tensors. Restore optimizer state with the framework’s supported mapping rather than matching transient object addresses or assuming two same-shaped tensors are interchangeable. Also distinguish the checkpoint’s stored precision from the compute precision and any master-weight representation used for updates.

A scaler’s current scale, growth tracker, and overflow handling can affect the sequence of successful updates. Decide how scheduler advancement relates to skipped steps in the application loop. A checkpoint taken after an attempted update must record whether the optimizer actually changed parameters. The next learning-rate value alone is not always a full scheduler state: warmup, restarts, token counts, or phase transitions may require additional history. Version the training-loop convention so restored counters are interpreted in the same order.

An initial seed selects the start of a random stream; it does not identify the position reached after training has consumed many draws. Save each relevant generator’s current state or an explicitly reproducible logical counter mapping. Libraries and augmentation workers may use separate generators. Restoration order matters because model construction, warmup, or loader setup can consume random values before the first intended resumed operation. Restore at the point required by the application contract and keep test-only random draws from perturbing the production stream.

The data cursor should identify the next logical uncommitted contribution, not merely the furthest record fetched by a worker. Prefetch and decoding can run ahead, while retries or dropped batches can separate read position from successful updates. Preserve dataset identity, shuffle and sampler state, packing remainders, and any required pending accumulation. An explicit policy may replay prefetched but unconsumed records. External readers may need committed offsets or idempotency keys rather than a tensor snapshot alone. Verify sample or token identities after restart, not only a monotonically increasing batch counter.

PyTorch 2.14 warns that reproducibility is not guaranteed across releases and platforms, even with equal seeds. Kernel algorithms, collective reduction order, floating-point rounding, and topology changes can introduce differences beyond random stream state. Define whether your oracle expects exact sample identities, numerically close next updates, or bitwise equality under a tightly fixed environment. Restoring RNG is necessary for some contracts but is not sufficient for all of them. Record the chosen deterministic settings and their compatibility instead of promising universal identical trajectories.

The modular instrument is a transparent state-position analogy, not a random generator. Reset, set both m=8 and a=3, predict residue one after three additions, then Step three times. Separately Reset, set both m=8 and a=1, predict residue three after three additions, then Step three times. Both the rule and position affect the next value. This short periodic sequence neither models a real RNG nor demonstrates its statistical quality. It helps distinguish restoring a current execution position from merely knowing that a process once started at zero.