Crash recovery: an acknowledgement must survive its promise
Audit durable terms, votes, entries, snapshot state, and storage ordering at each crash cut before counting acknowledgements.
On this page
The question for today
What you’ll understand
- Identify which acknowledged protocol state must survive a restart.
- Place persistence completion before the responses that rely on it.
- Design restart and torn-write tests around the storage contract rather than process memory.
Consensus proofs remember facts: an acceptor promised a ballot, a Raft voter chose a candidate, a replica stored a log entry. If the machine acknowledges those facts and then forgets them after a power loss, its network behavior no longer matches the proof. Replication does not erase the local storage contract. This lesson follows the path from a protocol transition through durable storage to an outgoing response and asks what a recovering node can safely claim at each interruption point.
Durable state enforces behavior after the next boot
For basic Raft, current term, the vote for that term, and log entries are persistent protocol state. The implementation must preserve the relevant changes before sending responses that rely on them, following its proven ordering rules. A vote grant sent while its record exists only in a page cache can be forgotten after a crash, allowing the same node to vote differently in the same term. An append success counted as durable replication has the analogous obligation for the acknowledged entries.
Paxos has corresponding promise and accepted-proposal records. Restarting does not create a new acceptor identity free to contradict the old one's promises. Likewise, replacing disks or cloning a replica's identity is a membership and recovery event, not ordinary transparent reboot. A node that cannot recover trustworthy durable state may need controlled removal and rejoining rather than pretending its empty directory is the old member. Preserve the identity-to-history relationship that the quorum proof assumes.
A successful write call is not necessarily a durable record
Bytes can pass through process buffers, kernel caches, storage-controller caches, and device media. The selected storage API must define what completion survives, including power loss and torn writes. Use its synchronization and recovery mechanisms rather than treating a write return as universal durability. Record framing, sequence numbers, lengths, and checksums can let recovery identify a complete prefix and reject a damaged tail. Detection does not authorize silently erasing an earlier acknowledged prefix or inventing a replacement command.
Batching several records behind one durability operation can amortize cost, but every acknowledgement still waits for the completion that covers its dependencies. Storage and network work can sometimes overlap under a proven integration contract; do not impose an inaccurate claim that every message must wait for every byte in every batch. Conversely, an optimization copied from another library may violate this one's ordered Ready/Advance or snapshot contract. Review the actual integration API, including failures, truncation, and recovery, rather than only the abstract protocol core.
Recovery joins snapshots, logs, and application coherently
On restart, reconstruct the durable protocol state and the state machine's compatible snapshot and continuation. Whether applied position is persisted separately or derived through replay depends on the implementation, but applying a command twice must not duplicate an external side effect or corrupt the local state. Persist request-deduplication metadata with the state it describes. If recovery finds corruption or an impossible index relationship, fail or repair through a supported protocol instead of allowing a locally convenient suffix to become authoritative.
The reliability lab assumes that each available replica still has valid protocol state. Reset, explicitly set n=3 and k=1, predict that two meet the majority, then Play or Step. Reset and explicitly set n=3 and k=2 for the no-majority comparison. It does not distinguish a healthy disk from a returning node that forgot its vote, and it has no fsync, WAL, power-loss event, or torn-write detector. A green availability count cannot certify that acknowledgements meet the persistent-state assumption.
Work it through
Put the vote reply after durable completion
A node receives a valid vote request at 1 ms. It writes the updated term and vote at 3 ms, but the storage operation establishing their durability completes at 7 ms. A proposed implementation sends the vote grant at 4 ms and can lose cached writes if power fails at 5 ms.
At 4 ms the term/vote change has been issued but is not yet established durable under the stated storage contract. Sending the grant now lets the candidate count a promise the voter can forget.
A power loss at 5 ms occurs before completion at 7 ms. If the node returns without the record, it can no longer enforce the acknowledged same-term vote through its recovered state, invalidating the one-vote argument.
The earliest justified grant in this trace is after the required durability completion at 7 ms, plus any separate response work. Batching the record with others is possible only if their shared completed durability operation covers this vote before its grant is sent.
Moving the response dependency behind durable completion preserves the fact that quorum intersection needs after recovery. A write timestamp at 3 ms is not the same as durability at 7 ms.
The living notebook
Explore the model
Interactive lab · reliability
Available is not the same as correctly recovered
For separate majority counts choose Reset, explicitly set both n and k, predict, then Play or Step. Compare (3,1) with Reset plus (3,2). The toy assumes valid retained state and does not simulate votes, fsync, WAL replay, disk corruption, or crash recovery.
- Available replicas
- 4 of 5
- Majority needed
- 3
- Majority available?
- Yes
A replica count and a majority threshold, without a consensus protocol. Playback highlights fixed replica states. Having a majority alone does not establish linearizability, durability or failure independence. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA quorum's safety argument requires its participating replicas to preserve acknowledged protocol facts; reachability cannot replace that durability evidence.
Pause & reason
Check your understanding
Your turn
Try it yourself
Ten fixed-size log records each carry 1 KiB of payload and share one completed durability operation. What payload size does the batch have? If the last record's positive response is sent before that operation completes, does batching justify it?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use binary KiB units and separate amortized cost from the dependency that permits a response.
Compare with the solution
The payload is 10×1,024=10,240 bytes, or 10 KiB, before framing, checksums, and alignment. Batching can spread one durability operation across those records, but cannot justify acknowledging a record before its required state is durable. The response must depend on a completed operation that covers the record and any required term or vote changes. After a crash, recovery must reconstruct a valid snapshot and log continuation consistent with previously acknowledged obligations, rather than assuming every issued write completed.
Keep the idea
Take it with you
- Persistent terms, votes, and entries are premises of the consensus proof.
- A response must wait for the storage dependency that gives it its advertised meaning.
- Recovery must preserve identity, snapshot, log, and application-state consistency.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.