Checkpoints and storage

An inference export can contain only weights and model configuration. A training checkpoint usually needs more: optimizer and scheduler state, update count, gradient-scaling state where applicable, random-number-generator state, and data-sampler position. Include dataset release identity, code revision, and relevant environment information. Decide how to handle accumulated gradients if a save occurs between optimizer steps; checkpointing at an agreed update boundary is often easier to reason about than reconstructing a partially accumulated update.

Distributed training adds ownership and consistency requirements. Each shard must correspond to the same logical step and compatible layout. PyTorch Distributed Checkpoint supports saving and loading distributed state, including resharding supported state across different layouts. That does not eliminate the need to test your model's state mapping or version compatibility. Record an explicit format version and verify recovery across the hardware and software changes you intend to support, rather than assuming every future environment can read every old artifact.

Write checkpoint objects under a fresh immutable version. Record expected shards, sizes, hashes, and logical step in a manifest. Publish the version as eligible for restore only after all required data is durably written and validated. A crash before publication leaves an incomplete candidate that readers ignore. A crash after publication must leave every referenced object readable according to the storage system's contract. Garbage collection should retain objects referenced by live manifests and allow time for readers using an older version.

The publication primitive depends on storage semantics. A single-filesystem rename may support atomic visibility, but multi-file durability also depends on flush and directory behavior. Object stores do not generally provide a universal atomic directory rename. Use documented conditional writes or a transactional metadata store when coordinating a latest pointer, and verify the specific backend. Checksums detect corruption; they do not prove that independently saved shards share one training step. Asynchronous saves also require a stable staged snapshot so training cannot mutate buffers still being serialized.

Frequent checkpoints reduce recomputation but consume bandwidth and pause or compete with training. In a simple model, a save costing $C$ seconds every $\tau$ seconds of useful work adds overhead $C/\tau$. With independent rare failures of mean interval $M$, uniformly located failures lose roughly half an interval, adding $\tau/(2M)$. Minimizing their sum gives an approximate interval. This is a planning model, not a universal recommendation for correlated rack failures or variable save times.

Recovery time is separate from recomputed work. Measure checkpoint discovery, download, validation, process launch, communicator setup, and the first successful update. Practice restoring under missing-shard, truncated-object, and incompatible-version scenarios. Keep at least one older known-good checkpoint rather than immediately replacing every recovery option with the newest write. Storage lifecycle rules and access permissions must preserve the recovery horizon, and a verified save should still be followed by periodic end-to-end restore drills.