advanced27 min studyLesson 305 in Engineering

A distributed snapshot must account for records between operators

Reason about consistent cuts, aligned checkpoint barriers, captured in-flight data, and why unaligned checkpoints differ from merely skipping alignment.

On this page

What you’ll understand

  • Identify a consistent cut and the channel state needed to preserve it.
  • Explain aligned barrier blocking at a multi-input operator.
  • Distinguish unaligned snapshots with channel state from at-least-once alignment skipping.

Saving every worker's memory at unrelated moments does not necessarily save a possible computation state. One worker may have sent a contribution that another has not yet processed, so the contribution must live somewhere in the recovery image. Distributed snapshots solve this accounting problem without requiring all machines to share one instantaneous clock. Use conservation and event dependencies to understand the cut before discussing checkpoint speed.

A cut includes processes and channels

A consistent cut does not include receipt of a message while excluding its send. Messages sent before the cut but received after it belong to in-flight channel state. The original Chandy–Lamport algorithm records local process states and relevant channel contents under its reliable FIFO-channel model; marker rules identify the cut without stopping all computation globally. The assumptions matter. A mechanism designed for ordered channels cannot simply be copied onto arbitrary reordered or lossy communication without additional rules. A snapshot is a logical boundary, not necessarily a set of states that all existed at one wall-clock instant.

A useful conservation check tracks a transferred quantity across sender state, receiver state, and the channel. Recording only both endpoint balances can lose an in-flight transfer or count it twice. Stream operators have analogous contributions in input buffers, output buffers, and state updates. Recovery must restore a compatible source position and replay responsibility so every required contribution has one place. This does not claim external sinks are automatically atomic; the snapshot defines the managed computation boundary, while a visible side effect can require a separate output protocol.

Qtotal=QA+QB+Qin flightQ_{\rm total}=Q_A+Q_B+Q_{\rm in\ flight}
This fictional conservation example counts transferable tokens in two process balances and their channel. It assumes no creation or destruction; omitting channel state is valid only when the chosen cut ensures there is nothing in flight to account for.↔ Scroll wide equations horizontally.

Alignment creates a compatible boundary by waiting

In Flink's aligned checkpoint mechanism, barriers follow source progress through the dataflow. At an operator with multiple inputs, arrival of one checkpoint barrier causes that input's post-barrier records to wait while other inputs continue until their corresponding barriers arrive. The operator can then snapshot a boundary that excludes all post-barrier records and forward progress. This is not a global pause of every worker at once. A slow or backpressured input can make alignment expensive, while records behind an early barrier accumulate pressure upstream.

Unaligned checkpoints instead include relevant in-flight data in the checkpoint so barriers can overtake buffered records under the mechanism's rules. This can reduce barrier delay under backpressure but adds channel-state I/O and recovery work. It is not the same as simply skipping alignment while continuing to process all inputs and omitting channel accounting; that latter at-least-once approach can replay contributions already reflected in operator state. Name the configured mode precisely. In Flink 2.1, limitations and supported combinations belong to the versioned documentation rather than an assumption that every snapshot or savepoint behaves identically.

Bsnapshot=Boperator+Bchannel+BmetadataB_{\rm snapshot}=B_{\rm operator}+B_{\rm channel}+B_{\rm metadata}
All terms are bytes represented by a checkpoint under a stated mode. A mode that captures channel data can increase B_channel while reducing alignment waiting; actual incremental storage references and compression affect physical bytes transferred.↔ Scroll wide equations horizontally.

Measure the bottleneck moved by the mode

Observe checkpoint start delay, alignment duration, synchronous snapshot work, asynchronous upload, channel bytes, and recovery time. Reducing alignment delay can expose checkpoint-storage bandwidth as the next limit; unaligned mode cannot remove an I/O bottleneck by adding more captured data. Test failure recovery under the same backpressure that motivated the change. A completed upload alone is not proof the entire checkpoint was coordinated successfully. Restore from the completed checkpoint and verify state, source positions, and sink behavior using a trace with records deliberately crossing the cut.

The queue lab illustrates waiting inventory only. Reset, set BOTH λ=3 and μ=2, predict backlog six at t=2 seconds from the initial four, then Step four times. Reset and re-set BOTH λ=2 and μ=3; predict backlog two at t=2. Editing controls retains t and recomputes from four, not a changed-rate history. These normalized fluid backlogs are not checkpoint barriers or channel snapshots, and the separate M/M/1 value is not alignment time. Use the conservation example to reason about correctness and the queue only to discuss how much work may be waiting.

Keep an in-flight transfer in the snapshot

Two fictional operators hold 70 and 30 tokens. A sends ten to B, debiting its balance before B receives them. The cut records A after sending and B before receiving.

  1. A's recorded balance is 60 and B's is 30. Endpoint state alone totals 90, apparently losing ten tokens.

  2. The cut includes the send but not the receipt, so record the ten-token message as channel state. The snapshot total becomes 60+30+10=100.

  3. On recovery, deliver that captured message exactly once under the snapshot's replay responsibility. B then reaches 40 and the channel empties, preserving 60+40=100 rather than replaying the same transfer twice.

The result

The consistent image needs the in-flight contribution. Saving endpoint state without its compatible channel or alignment boundary loses the computation's accounting.

Explore the model

Interactive lab

Interactive lab · queue

Waiting inventory is a snapshot cost, not its proof

For EACH run Reset, set BOTH rates, predict, then Step four times to t=2 seconds. From backlog four, (λ,μ)=(3,2) yields six and (2,3) yields two. Edits retain t/recompute from four. This fluid model has no barriers, FIFO-marker protocol, operator state, snapshot, or recovery semantics.

Waiting inventory is a snapshot cost, not its proofFluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Current readings: Fluid backlog: 4; Net rate · λ − μ: −1 /s; M/M/1 mean residence: 1 s. All parameters are adjustable with the labeled controls below.02.557.51001.534.56Elapsed time · sFluid backlogλ < μ
Fluid backlogCurrent time
Fluid backlog
4
Net rate · λ − μ
−1 /s
M/M/1 mean residence
1 s
t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forBackpressure can increase waiting inventory, but snapshot correctness requires explicit accounting for the logical cut and in-flight records.

Check your understanding

What distinguishes a correct unaligned checkpoint from merely ignoring barrier alignment and omitting channel state?

Try it yourself

A snapshot records A=40 tokens and B=45, with a recorded in-flight transfer of 15. The conserved initial total is 100. What total is represented, and what would happen if recovery both delivered the captured transfer and replayed its send as a new effect?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count the channel once, then consider the duplicate responsibility introduced by a second replay path.

Compare with the solution

The image represents 40+45+15=100 tokens. Delivering the captured message accounts for its one contribution. Replaying its send as a separate new transfer can duplicate the contribution or debit again, violating the intended cut; source and channel replay responsibilities must be coordinated.

Take it with you

  • A consistent cut includes compatible process, source, and channel state.
  • Alignment waits for a boundary; unaligned capture accounts for crossing records.
  • Checkpoint speed and correct recoverability require different evidence.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.