advanced28 min studyLesson 550 in Engineering

Worker failure: restart a coherent training state

Distinguish fixed-group collective failure from uneven inputs and elastic restart, then budget replay and membership-dependent objective changes.

Before you begin

On this page

The question for today

What you’ll understand

Explain why fixed-group collectives do not proceed by a surviving majority.
Distinguish DDP join from process failure and torchrun restart behavior.
Calculate bounded replay work while preserving checkpoint and input-state assumptions.

If one data-parallel worker disappears, the remaining workers do not automatically become a smaller correct training group. They may be inside a collective, have different partial gradients, or have reached different optimizer states. We scope restart behavior to PyTorch 2.9 torchrun elastic launch and distinguish it from DDP’s uneven-input join API. Sharded training adds state ownership that a missing worker can remove. Recovery must establish a coherent model, optimizer, input position, and new communication group before the next accepted update.

Participation failure is not a quorum decision

A fixed process-group collective expects its specified participants and compatible operations. A surviving majority is not sufficient to finish the same all-reduce, reduce-scatter, or all-gather with the intended result. Timeouts or backend errors are failure signals, not a valid zero-gradient substitute. In sharded training, surviving ranks may also lack the failed owner’s current optimizer or parameter slice. Reconstructing a group and state requires a documented recovery path; deleting the missing rank from a local loop does not change the peer group consistently.

DDP join addresses workers exhausting uneven input streams by shadowing its supported communication pattern. It is not a general dead-worker recovery mechanism, and it cannot automatically account for arbitrary extra application collectives. A token-count reduction or synchronized model operation needs its own compatible handling, with early termination where required. Do not use “uneven input supported” as evidence that an abruptly failed process can be ignored. State whether a test covers clean iterator exhaustion, an exception, a process kill, a node loss, or a network partition.

B_{\rm global}=WKB_{\rm micro}

For balanced equal-size data-parallel inputs, W is active group size, K positive microbatches per update, and Bmicro examples per rank per microbatch. The result is examples per update, not valid-token count. A membership change requires rechecking both this batch policy and the actual token normalization.↔ Scroll wide equations horizontally.

Elastic launch restarts workers; it does not recreate unsaved state

The pinned torchrun contract stops surviving workers on failures or membership changes and restarts the group within the configured restart policy. Rank assignments are not stable across restarts, and elastic world size can change. Code must discover the new group and map logical state through a supported load or reshard procedure rather than assume that old rank two owns the same tensor forever. Recompute group-dependent normalization, input partitioning, and any global-batch policy. Keeping a nominal learning rate unchanged does not establish that the resumed optimization objective is unchanged.

Choose a coherent checkpoint boundary, commonly after a completed optimizer update, and save the corresponding model, optimizer, scheduler, scaler, random-state, and input-progress information needed by the claimed replay semantics. A failure during an update can leave ambiguous partial state in surviving processes; discard it and restore the selected coherent state rather than infer a global commit from one rank’s progress message. Saving model weights alone may permit approximate continuation, but it does not prove equivalence to uninterrupted training with the same sample sequence and optimizer state.

T_{\rm replay,plan}=(s_{\rm done}-s_{\rm checkpoint})T_{\rm step}+T_{\rm launch}+T_{\rm load}

For integer completed step counts with sdone≥scheckpoint and nonnegative durations in seconds, this illustrative plan counts replay of completed unsaved steps plus launch and load. It excludes partial-step work, variable step times, input rereads, checkpoint validation, and additional failures.↔ Scroll wide equations horizontally.

Define replay, side effects, and recovery evidence

Input progress can be ahead of a model checkpoint because workers prefetched or acknowledged data before the checkpoint became durable. Restoring a dataset offset independently can skip samples; replaying without identity records can repeat them. Define the intended sample and side-effect semantics, especially for external metrics, artifact publication, or streamed data. Durable checkpoint publication and a compatible shard manifest are prerequisites to trusting the saved boundary. Recovery tests should compare a controlled interrupted run with the appropriate reference, including the first several optimizer states after restart.

Use the reliability lab to expose a misleading majority intuition. Reset, explicitly set replicas five and unavailable one, predict four available replicas and a majority threshold of three, then Play or Step. For a separate run Reset and explicitly set replicas five and unavailable zero before predicting again. The instrument only counts majority availability; it implements no training collectives, optimizer shards, checkpoint recovery, or elastic launch. Its “majority available” result must not be interpreted as permission for a fixed five-rank training collective to continue with four participants.

Work it through

Budget replay from a completed checkpoint boundary

A coherent checkpoint records completed optimizer step 100. The job completes steps 101 through 104, then fails during step 105. Assume eight seconds per completed replayed step, twenty seconds for launch, and twelve seconds for load. Count the stated replay plan back to completed step 104.

The checkpoint omits four completed steps: 101,102,103,104. Under the fixed-duration assumption, replaying those steps costs 4×8=32 seconds. Partial work from step 105 is not a completed committed update to preserve.
Adding twenty seconds for launch and twelve for load gives a 64-second plan to reach the previous completed step 104. This is not total failure overhead; the partial step’s wasted time and other recovery work are excluded.
The new workers must restore the coherent logical state and compatible input progress before replay. If membership changes, reconsider example and token counts, shard ownership, and optimizer settings. Reusing stale in-memory partial state on one surviving rank would invalidate the calculation’s coherent-restart assumption.

The result

The stated replay plan is 64 seconds, excluding partial-step loss and other variable costs. A valid restart restores a coherent state rather than relying on surviving-rank majority.

The living notebook

Explore the model

Interactive lab

Interactive lab · reliability

A majority count is not collective participation

For independent runs use Reset → explicitly set BOTH n and k → predict → Play/Step. Compare (5,1), four up and majority threshold three, with (5,0), five up. The model has no process group, collective, shard state, restart policy, or checkpoint semantics.

AvailableUnavailableMajority threshold

Available replicas: 4 of 5
Majority needed: 3
Majority available?: Yes

Replica count n5Unavailable replicas k1

t = 0 s

A replica count and a majority threshold, without a consensus protocol. Playback highlights fixed replica states. Having a majority alone does not establish linearizability, durability or failure independence. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA fixed five-rank training group cannot infer correct four-rank progress from a majority-availability indicator.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

Training collectives require their specified participants and compatible state. Majority counting is unrelated to automatic collective reconfiguration, and uneven-input join is not arbitrary worker-failure recovery.

Your turn

Try it yourself

A checkpoint records completed step 200. Steps 201 through 203 complete before failure during 204. Assume five seconds per replayed completed step, ten seconds for launch, and five for load. Calculate the stated replay plan. If world size drops from four to two with microbatch two and accumulation four, what happens to example count per update?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count completed unsaved steps separately from the partial one, then use W×K×Bmicro under the stated balanced-input assumptions.

Compare with the solution

Three completed steps require fifteen seconds of replay. Launch and load add fifteen, giving a 30-second plan that excludes partial-step waste and other costs. The example count per update falls from 4×4×2=32 to 2×4×2=16. Preserving 32 could require accumulation eight under compatible workload assumptions, but the actual token denominator, schedule, and optimizer policy still need verification. Restoring model weights alone does not establish the intended input replay or optimizer-state equivalence.

Keep the idea

Take it with you

Fixed-group collectives require compatible participation rather than a majority.
Distinguish clean uneven-input handling from abrupt failure and coordinated restart.
Restore coherent logical state and revisit ownership, input progress, and normalization after membership changes.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.