Distributed checkpoint manifests

A useful entry identifies a logical tensor or state field, global shape, datatype, stored region, object location, byte range, and integrity information under a versioned encoding. It also identifies the checkpoint generation and logical cut. Replicated state requires a declared owner or replica policy; otherwise summing every stored copy can falsely look like complete coverage. Metadata must distinguish actual tensor elements from padding, compressed payload, or a framework-specific flattened layout. A loader needs the interpretation as well as the object name.

For an ordinary partitioned tensor, stored regions should cover the required logical domain without gaps. Unexpected overlap is suspicious unless it is explicitly declared replication with a consistency rule. Some formats split regions across shared files or pack multiple tensor fragments into one object, so one-file-per-rank is not a general invariant. PyTorch DCP and Megatron use planners or sharded mappings for these relationships. Application metadata must additionally name non-tensor requirements such as data progress and scheduler state; a tensor-complete manifest can still be training-incomplete.

Writers can report object creation and checksums to a coordinator, but an acknowledgment must have a defined persistence meaning. A complete manifest should reference immutable objects whose writes have succeeded under the storage backend contract. It should not list a mutable temporary path that another writer may later overwrite. Coordinator retries need a generation identity so a late response from an abandoned attempt cannot satisfy a newer attempt accidentally. Treat an incomplete generation as an ordinary recoverable failure state, not as something the loader must optimistically repair.

Commit metadata is a protocol boundary. Megatron documentation describes final metadata publication after asynchronous shard-saving finalization; this is a concrete implementation detail, not proof that every filesystem or application has the same guarantees. Validate the selected backend and version. A checksum proves agreement with expected bytes only when the expectation itself is trusted, and structural coverage does not prove that all tensors belong to the same update. Keep those checks distinct so an incident report can name precisely which property failed.

A restore preflight should resolve one committed generation, validate its schema and logical cut, check required objects and tensor coverage, and reject conflicting identities before starting expensive computation. If an object is unavailable, use a documented replica or fall back to an older complete generation; do not silently initialize that optimizer region to zero. Recovery can stream validated pieces to bound memory, but partial success must not be reported as a usable training state. Keep error paths explicit enough to distinguish missing payload from unsupported schema.

The reliability instrument illustrates why a majority is an inadequate completeness test. Reset, set both n=5 and k=1, predict four available and majority threshold three, then Play or Step. Separately Reset, set both n=5 and k=0, predict five available, then Play or Step. If all five represent indispensable nonredundant shards, only the second case has every required piece despite both having a majority. This toy counts availability; it neither parses a manifest nor models replicas, erasure coding, persistence, or tensor coverage.