Restart from a consistent model and input boundary

Specify what a training checkpoint means: completed optimizer steps, gradient-accumulation progress, relevant scheduler state, and the input occurrences represented by that saved model state. A source read cursor can be ahead because of prefetch or asynchronous transforms. A delivered batch can also be pending accumulation rather than included in a completed optimizer update. Decide whether to save those pending tensors and states or replay them from an earlier consistent boundary. A local integer called step is meaningful only with its batch and rank interpretation, especially when multiple ranks contribute to one update.

For a deterministic index stream, one recovery design saves the consumed position and reconstructs later work through deterministic replay. Another saves the source cursor plus all buffered, shuffled, and in-flight state needed to emit the same next occurrence. Both need a documented contract. Advancing only the source cursor while discarding its prefetched records creates a gap. Rewinding input while retaining later model effects can create repeated updates. Coordinate checkpoint publication so a restart cannot accidentally pair a newer input snapshot with an older model file or a partially written state bundle.

A shuffle buffer contains a particular set and order of pending records after a particular number of draws. The initial seed describes how a random process starts, not necessarily its current state or the records already consumed. Stateful transforms may use several libraries' RNGs, and worker scheduling can change which state produces which sample. Capture the required state or use a supported stateless occurrence-based transformation contract. Exact replay may also require deterministic source listing, fixed inputs, and compatible algorithm versions. Do not promise identical model results merely because the input order replays; compute kernels have their own reproducibility limits.

TorchData 0.11.0 StatefulDataLoader documents state_dict and load_state_dict, custom sampler and dataset hooks, and aggregation across local worker processes rather than training ranks. Its iterable-state examples require the same worker count on restore. Default fast-forward behavior and custom efficient state restoration have different costs and guarantees. A topology change therefore needs an explicitly supported repartitioning strategy or a restart policy rather than blind reuse of local worker cursors. Test the exact loader, dataset, transforms, and version combination used by the experiment; API availability is not proof that every custom stateful operation is captured.

Run an identifiable short experiment, save at a known boundary, record the next sequence of occurrence identities and transform outputs, then restore and compare under the declared topology. Include a nonempty shuffle buffer and nonzero prefetch so the test exercises the difficult state. Check dataset and tokenizer versions and reject incompatible restores explicitly. Record whether the goal is exact next-occurrence replay, bounded repetition, or a fresh epoch; these are different accepted behaviors. Measure restart cost as well, since fast-forwarding a large remote stream can be correct but operationally expensive.

The modular lab demonstrates why one displayed state can omit execution history. Reset, set both modulus m=8 and step a=3, predict residues after three and eleven additions, then Step. Both positions have residue one because the orbit repeats every eight additions. This exact additive process is not a loader checkpoint or RNG. It illustrates that saving only the visible residue cannot distinguish those histories. A real checkpoint must retain enough traversal and random state for its next-output contract, rather than assume a short label or epoch counter encodes the full pending stream.