Snapshots: preserve a committed prefix and its meaning
Build and restore a snapshot with an applied boundary, term, membership, and retry metadata before discarding the corresponding log prefix.
On this page
The question for today
What you’ll understand
- Choose a snapshot boundary from applied committed state.
- Include the metadata needed to interpret the remaining log and client history.
- Explain installation and compaction ordering without treating a snapshot as an arbitrary file copy.
A long-running consensus log cannot retain every command forever. A snapshot replaces an already applied prefix with the state that prefix produced, allowing storage reclamation and efficient follower catch-up. The replacement is correct only if the captured state and its metadata describe the same boundary. Copying database pages while application continues without a consistent snapshot mechanism can create a state that never existed, even if the snapshot file has a valid checksum.
Capture state at one applied index
Choose an index s that has been committed and applied. The snapshot must represent exactly the state after applying the command prefix through s, from a compatible earlier state. Concurrent snapshotting can use a database's consistent snapshot mechanism, copy-on-write state, or an explicitly serialized application boundary. The implementation must preserve a coherent cut rather than letting unrelated fields come from different applied positions. A local log entry beyond commitment cannot be made irrevocable by placing its speculative effect into a snapshot.
Record the last included index and its term so the remaining log can still perform consistency checks at the compacted boundary. Include the configuration associated with the snapshot's state according to the membership protocol, not whichever membership the operator happens to prefer at restore time. Client-session and request-deduplication information may also be required to preserve retry behavior. A restored business state without the corresponding deduplication ledger can replay an already executed request even though consensus history itself was retained.
Publish a durable replacement before deleting its source
The snapshot and its recovery metadata must become durably discoverable before the covered log records are reclaimed. Exact file and manifest operations depend on the storage engine, but the recovery invariant is general: after every crash cut there must remain a complete usable snapshot plus the log continuation it needs. Writing a snapshot file, deleting old logs, and only then making the new file recoverable leaves a dangerous gap. Checksum and format validation detect damaged inputs; they do not establish ordering or membership correctness by themselves.
A follower behind the leader's retained prefix may need snapshot installation instead of ordinary AppendEntries replay. It must install a compatible snapshot atomically with its local recovery state and then accept the correct suffix. Protocol rules determine whether an already matching local suffix can be retained or must be discarded. Installation must not roll application backward or combine an older snapshot with a newer incompatible database. An outdated snapshot request can be ignored according to the implementation rather than blindly replacing a replica's later state.
Compaction policy balances costs without changing safety
Frequent snapshots consume serialization, storage bandwidth, and transfer resources; infrequent snapshots retain a larger log and make catch-up more expensive. Keep enough useful suffix for common follower delays if the storage budget permits, and measure both snapshot duration and application interference. A snapshot can be larger than the commands it replaces when current state is large. The reason to take it is a defined recovery and retention tradeoff, not an assumption that every checkpoint compresses data.
The queue lab supplies only a retention-pressure analogy. Reset, explicitly set arrival four and service three, predict growth from four retained units, then Play or Step. For a separate draining example Reset and explicitly set arrival two and service three before predicting and playing. Rate edits retain elapsed time and recompute the fixed-rate curve from four initial units. The model does not create snapshots, delete prefixes, preserve metadata, or install follower state. Treat service as an idealized removal capacity, not a literal snapshot-completion event.
Work it through
Count the retained suffix without confusing its status
A replica has last-log index 560, commitIndex 550, and lastApplied 540. It creates a consistent snapshot at applied index 500. Assume the snapshot is safely published before compaction and it then retains exactly the log entries after 500.
Index 500 is within the applied committed prefix because 500≤540≤550. The snapshot captures state and required metadata at 500, rather than silently including effects through 540.
The retained suffix is indices 501 through 560, giving 560−500=60 entries. Of those, 501 through 550 are within the known committed prefix; 551 through 560 remain beyond it.
A follower installed at snapshot 500 must apply the next committed entries in order. Reaching the stated commit boundary 550 requires 50 additional applications, while receiving a suffix through 560 does not authorize applying the final ten speculative entries.
The snapshot replaces a 500-entry prefix while a 60-entry continuation remains. Recovery metadata and commitment knowledge still distinguish which continuation entries can execute.
The living notebook
Explore the model
Interactive lab · queue
Retention pressure is not snapshot correctness
Compare independent fixed-rate retention analogies using Reset → explicitly set both rates → predict → Play or Step. Use (λ,μ)=(4,3), then Reset and (2,3). Edits retain elapsed t and recompute from four initial units. This fluid backlog does not snapshot state, compact a log, or preserve configuration metadata.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA capacity curve can motivate compaction policy; a consistent applied boundary and crash-safe recovery protocol establish snapshot correctness.
Pause & reason
Check your understanding
Your turn
Try it yourself
A follower has applied through 790 but the leader's retained log begins at 801. A valid snapshot covers index 800, and the current known committed suffix ends at 820. After installing the snapshot correctly, how many additional commands reach 820? Which membership should the snapshot carry?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Count the inclusive continuation from 801, then tie membership metadata to the snapshot boundary rather than an operator's current display.
Compare with the solution
Indices 801 through 820 contain twenty commands to apply in order after the snapshot state at 800. The snapshot must carry the membership and other recovery metadata required by the chosen protocol for that boundary, not an unrelated latest configuration copied from a different applied position. If newer membership changes appear in the suffix, process them according to their proper log order and activation rules. Installation must first establish a coherent recoverable state so the suffix is interpreted against the correct prefix.
Keep the idea
Take it with you
- Snapshot state and metadata must describe one committed applied boundary.
- Make the replacement recoverable before reclaiming its source prefix.
- A retained suffix can contain both committed and uncommitted entries.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.