Checkpoint corruption and restore drills

For immutable payload objects, record a cryptographic digest with the algorithm and exact byte scope in a trusted manifest. On restore, verify lengths and recomputed digests before treating those bytes as the expected representation. A checksum mismatch can detect altered or corrupted payload, while a matching checksum does not prove correct tensor placement, a consistent logical cut, or the intended next batch. Shared containers need a defined whole-object or chunk-level verification policy so a partial range read does not pretend to validate bytes it never checked.

A malicious party able to replace both payload and its unauthenticated checksum can produce a self-consistent replacement. Authenticity requires an appropriate trust mechanism, such as authenticated storage access and a signed or keyed-authenticated manifest whose identity and policy are verified. NIST’s secure hash and HMAC standards describe different primitives; a plain hash has no secret or signer identity. Also respect the deserializer’s trust boundary. Do not load arbitrary executable object formats just to discover whether an untrusted checkpoint was well formed.

Begin with the committed manifest, dependency closure, schema compatibility, tensor coverage, lengths, and digests. Then verify logical tensor identities, dtype, shape, aliases, optimizer associations, step counters, and data progress. A strict loader should reject missing required keys instead of silently initializing them. Memory allocation can fail before useful restore progress even when final tensors fit, so test peak staging and conversion requirements. Reconstruct into an isolated recovery job and keep the original immutable artifact available for diagnosis.

Next run a controlled continuation test against a trusted uninterrupted reference. Compare the next sample or token identities, loss inputs, successful-update decisions, scheduler values, optimizer states, and parameter update. Numerical closeness and bitwise equality are different oracles; choose each only where its assumptions hold. A finite loss value is weak evidence because missing moments or repeated data can still yield plausible numbers. When exact next-step comparison is unavailable, record the weaker statistical or behavioral evidence honestly rather than upgrading it to proof of faithful continuation.

A drill matrix should include missing and truncated objects, wrong-generation metadata, modified manifests, unavailable shared dependencies, unsupported versions, interrupted staging, and changed worker counts when supported. Include an intentionally corrupted case that must be rejected; a drill that only exercises the happy path cannot demonstrate detection. Measure allocation, download, validation, resharding, state application, input recovery, and first validated update. Time to deserialization completion is not time to usable training, and overlapping stages require a measured critical path rather than blindly adding independent timers.

The probability instrument illustrates limits of sampled coverage. Reset, set both p=0.1 and n=20, predict two hits on average and zero-hit probability 0.9²⁰, then Play or Step. Separately Reset, set both p=0.1 and n=30, predict three hits on average and zero-hit probability 0.9³⁰, then Play or Step. Here a hit represents an independently sampled stipulated bad case, not a checksum collision. The exact binomial display performs no restore or random fault injection, and unknown or correlated incident types do not satisfy its assumed common probability.