Consistent checkpoint-cut selection

In a distributed execution, events are related by local program order and communication dependencies. A consistent cut includes the causal predecessors of every included event. The classic Chandy–Lamport snapshot formulation also accounts for channel state when messages cross the chosen cut. A training system often avoids capturing arbitrary in-flight channels by selecting a coordinated update boundary where required communications and updates have completed. That is an application design choice, not evidence that arbitrary per-rank memory dumps automatically form such a snapshot.

Suppose one rank saves parameters after update 101 while another saves an indispensable optimizer shard from update 100. Both local files may be individually well formed, yet their union is not the declared post-101 state. Checkpoint generation, optimizer-update number, accumulation position, and any relevant version tags should agree according to the chosen protocol. A barrier can coordinate process arrival, but it neither invents missing state nor necessarily completes unrelated asynchronous device work. Use the documented completion dependencies for the actual streams and operations.

A practical boundary might be immediately after a successful optimizer update and the associated scheduler transition, with the next logical data cursor already defined. The precise order matters. If a scaler skips the optimizer update, an attempted-step counter may advance while the successful-update counter does not. Pipeline schedules may still hold activations for other microbatches, and an input pipeline may have read ahead. Either drain or represent that work under the contract. Calling every boundary an epoch checkpoint hides these differences.

Capture does not require all files to finish writing simultaneously. Once every mutable source has been copied into an immutable snapshot representing the agreed cut, writers may proceed at different speeds. Until that separation exists, an optimizer mutation or buffer update can race a slow copy and produce a mixture. Persistent buffers updated during forward execution deserve particular attention: model parameters being unchanged during backward does not prove every state-dictionary entry is unchanged. Identify writes to each category rather than assuming one global safe window.

Draw a short timeline showing update completion, scaler and scheduler actions, cursor advancement, staging, and the first subsequent mutation. Give each saved quantity a version from that timeline. Then inject a delay in one contributor and ask whether the version check fails or the protocol waits safely. A writer finishing late is harmless when it reads immutable captured state. A writer silently reading a newer live tensor is a correctness failure even if its resulting checksum and filename look plausible.

The modular instrument can expose version mismatch as a counting analogy. Reset, set both m=8 and a=1, predict residue two after two additions, then Step twice. Separately Reset, set both m=8 and a=1, predict residue three after three additions, then Step three times. Combining the two observations does not make them the same logical position. This deterministic counter does not model distributed messages or validate a checkpoint cut. Its wraparound also reminds you that a small repeating residue is insufficient as a unique generation identifier.