A distributed snapshot must account for records between operators
Reason about consistent cuts, aligned checkpoint barriers, captured in-flight data, and why unaligned checkpoints differ from merely skipping alignment.
On this page
The question for today
What you’ll understand
- Identify a consistent cut and the channel state needed to preserve it.
- Explain aligned barrier blocking at a multi-input operator.
- Distinguish unaligned snapshots with channel state from at-least-once alignment skipping.
Saving every worker's memory at unrelated moments does not necessarily save a possible computation state. One worker may have sent a contribution that another has not yet processed, so the contribution must live somewhere in the recovery image. Distributed snapshots solve this accounting problem without requiring all machines to share one instantaneous clock. Use conservation and event dependencies to understand the cut before discussing checkpoint speed.
A cut includes processes and channels
A consistent cut does not include receipt of a message while excluding its send. Messages sent before the cut but received after it belong to in-flight channel state. The original Chandy–Lamport algorithm records local process states and relevant channel contents under its reliable FIFO-channel model; marker rules identify the cut without stopping all computation globally. The assumptions matter. A mechanism designed for ordered channels cannot simply be copied onto arbitrary reordered or lossy communication without additional rules. A snapshot is a logical boundary, not necessarily a set of states that all existed at one wall-clock instant.
A useful conservation check tracks a transferred quantity across sender state, receiver state, and the channel. Recording only both endpoint balances can lose an in-flight transfer or count it twice. Stream operators have analogous contributions in input buffers, output buffers, and state updates. Recovery must restore a compatible source position and replay responsibility so every required contribution has one place. This does not claim external sinks are automatically atomic; the snapshot defines the managed computation boundary, while a visible side effect can require a separate output protocol.
Alignment creates a compatible boundary by waiting
In Flink's aligned checkpoint mechanism, barriers follow source progress through the dataflow. At an operator with multiple inputs, arrival of one checkpoint barrier causes that input's post-barrier records to wait while other inputs continue until their corresponding barriers arrive. The operator can then snapshot a boundary that excludes all post-barrier records and forward progress. This is not a global pause of every worker at once. A slow or backpressured input can make alignment expensive, while records behind an early barrier accumulate pressure upstream.
Unaligned checkpoints instead include relevant in-flight data in the checkpoint so barriers can overtake buffered records under the mechanism's rules. This can reduce barrier delay under backpressure but adds channel-state I/O and recovery work. It is not the same as simply skipping alignment while continuing to process all inputs and omitting channel accounting; that latter at-least-once approach can replay contributions already reflected in operator state. Name the configured mode precisely. In Flink 2.1, limitations and supported combinations belong to the versioned documentation rather than an assumption that every snapshot or savepoint behaves identically.
Measure the bottleneck moved by the mode
Observe checkpoint start delay, alignment duration, synchronous snapshot work, asynchronous upload, channel bytes, and recovery time. Reducing alignment delay can expose checkpoint-storage bandwidth as the next limit; unaligned mode cannot remove an I/O bottleneck by adding more captured data. Test failure recovery under the same backpressure that motivated the change. A completed upload alone is not proof the entire checkpoint was coordinated successfully. Restore from the completed checkpoint and verify state, source positions, and sink behavior using a trace with records deliberately crossing the cut.
The queue lab illustrates waiting inventory only. Reset, set BOTH λ=3 and μ=2, predict backlog six at t=2 seconds from the initial four, then Step four times. Reset and re-set BOTH λ=2 and μ=3; predict backlog two at t=2. Editing controls retains t and recomputes from four, not a changed-rate history. These normalized fluid backlogs are not checkpoint barriers or channel snapshots, and the separate M/M/1 value is not alignment time. Use the conservation example to reason about correctness and the queue only to discuss how much work may be waiting.
Work it through
Keep an in-flight transfer in the snapshot
Two fictional operators hold 70 and 30 tokens. A sends ten to B, debiting its balance before B receives them. The cut records A after sending and B before receiving.
A's recorded balance is 60 and B's is 30. Endpoint state alone totals 90, apparently losing ten tokens.
The cut includes the send but not the receipt, so record the ten-token message as channel state. The snapshot total becomes 60+30+10=100.
On recovery, deliver that captured message exactly once under the snapshot's replay responsibility. B then reaches 40 and the channel empties, preserving 60+40=100 rather than replaying the same transfer twice.
The consistent image needs the in-flight contribution. Saving endpoint state without its compatible channel or alignment boundary loses the computation's accounting.
The living notebook
Explore the model
Interactive lab · queue
Waiting inventory is a snapshot cost, not its proof
For EACH run Reset, set BOTH rates, predict, then Step four times to t=2 seconds. From backlog four, (λ,μ)=(3,2) yields six and (2,3) yields two. Edits retain t/recompute from four. This fluid model has no barriers, FIFO-marker protocol, operator state, snapshot, or recovery semantics.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forBackpressure can increase waiting inventory, but snapshot correctness requires explicit accounting for the logical cut and in-flight records.
Pause & reason
Check your understanding
Your turn
Try it yourself
A snapshot records A=40 tokens and B=45, with a recorded in-flight transfer of 15. The conserved initial total is 100. What total is represented, and what would happen if recovery both delivered the captured transfer and replayed its send as a new effect?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Count the channel once, then consider the duplicate responsibility introduced by a second replay path.
Compare with the solution
The image represents 40+45+15=100 tokens. Delivering the captured message accounts for its one contribution. Replaying its send as a separate new transfer can duplicate the contribution or debit again, violating the intended cut; source and channel replay responsibilities must be coordinated.
Keep the idea
Take it with you
- A consistent cut includes compatible process, source, and channel state.
- Alignment waits for a boundary; unaligned capture accounts for crossing records.
- Checkpoint speed and correct recoverability require different evidence.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.