Checkpoint economics: state the failure model before optimizing

Let C be checkpoint overhead time and τ the useful compute accumulated between checkpoints, both in hours. Let λ be the stationary job-level interruption rate per hour, with independent Poisson arrivals and rare interruption overhead in the approximation. R is restart time. A many-device job’s λ is not automatically one device’s hazard: independent component failures, redundancy, correlated outages, and job restart policy determine the relevant process. Record whether all allocated resources remain billed during checkpointing, lost work, restart, and provisioning delays.

Under the first-order model, an interruption loses about half the useful interval on average. This uses an approximately uniform failure phase and neglects higher-order interactions, including repeated interruption during recovery or checkpointing. Checkpoint overhead per useful hour is approximately C/τ. Restart contributes λR. These terms describe an overhead ratio relative to useful computation, not an exact fraction of wall time. The original literature develops more detailed models and interval conventions; keeping those conventions explicit avoids adding checkpoint time twice or applying one paper’s formula to another variable.

For positive checkpoint time and hazard, differentiating the interval-dependent terms balances checkpoint overhead against lost useful work. The stationary point is a minimum because the second derivative of C/τ is positive. A constant restart contribution does not move that optimum in this simple approximation, though a more complete model can couple restart and failure exposure. If hazard is zero, the stated positive-hazard optimum does not apply. Practical checkpoint schedules also obey wall-time limits, storage bandwidth, minimum durability requirements, and maximum tolerable recovery loss.

A recoverable checkpoint needs a consistent model, optimizer, random state, and data-progress boundary appropriate to the training algorithm. A completed write call or manifest update does not by itself establish atomic durable publication across all required state. Incremental checkpoints, background I/O, and overlapping compute change the cost model. Evaluate storage failures and stale checkpoints explicitly. A nominally cheap interruption-prone pool can lose its advantage through long provisioning delays or repeated progress loss that the rare independent-interruption approximation does not capture.

For a fixed useful-work requirement, multiply its modeled paid resource-time by a fictional rate only after specifying the number of allocated resources and included lifecycle. Add separate storage and transfer categories when relevant. Expected cost is not a completion guarantee or a bound on tail delay; a rare long outage can dominate a deadline even when the mean looks attractive. Check sensitivity to hazard, checkpoint size, restart time, and correlation before making an operational comparison. Preserve quality and recovery correctness rather than optimizing an interval that saves incomplete state.

The probability lab offers an independent-window risk analogy. Reset, explicitly set p=0.02 and n=20, predict, then Play or Step. Compare a separate fresh Reset with p=0.1 and n=20 explicitly set; exact mean affected windows are 0.4 and two. These are hypothetical independent Bernoulli windows, not a Poisson interruption trajectory or a checkpoint schedule. Playback highlights the distribution without generating observed failures. The lab cannot infer MTBF, represent correlated outages, calculate the optimal interval, or prove durable recovery.