Exactly-once state recovery does not automatically commit the outside world
Connect checkpoint recovery to transactional output visibility, idempotent commits, consumer isolation, and explicit limits for external side effects.
On this page
The question for today
What you’ll understand
- Separate replayed execution from one committed visible contribution.
- Explain how checkpoint-coordinated sink commits require external cooperation.
- Identify what Kafka transaction isolation does and does not cover.
A recovered operator may execute a record again while still producing the intended committed result once. That distinction is central to meaningful exactly-once claims. A checkpoint can recover internal state, but it cannot erase an email already sent or a payment already requested from an unrelated service. Draw the external visibility boundary and ask which protocol makes a replay, abort, or repeated commit safe at that boundary.
Execution count and committed effect count are different
After failure, records following the restored checkpoint can run again. Correct recovery reconstructs their managed-state contributions without retaining both pre-failure and replayed versions. A sink that exposes each attempt immediately can still duplicate visible output even when internal state is restored correctly. Exactly-once output therefore needs a sink-specific mechanism, such as a compatible transaction or an idempotent identity-based update. State the unit being counted: a source log position, a business event ID, or a keyed aggregate version. Duplicate business events at different source positions are not automatically deduplicated by processing each position once.
Late corrections can legitimately produce several updates for one window, so exactly-once does not mean one output message for every conceptual result forever. It concerns the promised committed contribution or update semantics under recovery. An append-only consumer and an upsert consumer interpret repeated result keys differently. Verify that the sink contract matches the operator's output mode, including retractions and replacements. A technically transactional append of two distinct but semantically duplicate business requests still creates two requests unless another application invariant prevents it.
Preparation, completion, and recoverable commit must agree
A checkpoint-coordinated transactional sink can prepare output, retain the information needed to commit it, and make it visible through a compatible completion protocol. Flink 2.1's sink interfaces separate writing and committables from committing. The destination must support the required transaction semantics, and recovery must resolve prepared work correctly. A commit acknowledgment can be lost after the destination commits; repeating a recovery action must not create a second distinct effect. Transaction identifiers, fencing or equivalent stale-writer protection, durability, and timeout behavior belong to the connector's actual correctness argument.
Kafka 4.0 describes atomically writing output records and consumer offsets within Kafka transactions, with read_committed consumers avoiding uncommitted and aborted transactional records. That guarantee covers the coordinated Kafka path under its configuration and failure rules. It does not automatically extend to a database, HTTP service, email system, or a consumer that exposes uncommitted records. Flink connector checkpoint coordination is another implementation path that must meet its documented prerequisites. Do not mix settings from different APIs and claim the combined words 'transaction' and 'checkpoint' prove an end-to-end guarantee.
Test the ambiguous commit boundary
Inject failure after prepare, after checkpoint completion, during destination commit, and after destination success but before the committer observes it. Restore without local caches and inspect both managed state and externally visible effects. Include transaction timeout and long recovery conditions; a prepared transaction expiring before recovery can break a simplistic success story. For destinations without compatible transactions, use a documented idempotent protocol, an outbox boundary, or a narrower guarantee. Report the limitation explicitly rather than treating a replayed API call as harmless because the operator state is exactly once.
The probability lab illustrates repeated successful attempts without deduplication. Reset, set BOTH p=0.5 and n=2, predict probability 0.25 for two successes, then Play or Step. Reset and re-set BOTH p=0.5 and n=4; predict probability 11/16 for more than one success. Edits retain t; playback highlights exact counts rather than sampling attempts. There is no sink, transaction, commit, or idempotency key. The point is that successful delivery opportunities can repeat; only the actual output protocol determines whether several successes create one effect or several.
Work it through
Replay work without exposing its abandoned transaction
Assume a correct fictional transactional sink and isolated readers. A completed boundary covers 100 input records, each contributing three tokens. Twenty later records execute into an uncommitted transaction before failure; that transaction is aborted during recovery.
The completed boundary exposes 100×3=300 tokens. The later transaction contains 20×3=60 additional tokens, but its uncommitted output is not visible to these readers.
Recovery restores the boundary at 100 records and aborts the abandoned 60-token transaction. The twenty later records are replayed under a new compatible transaction.
After replay commits, total visible contribution is 300+60=360 tokens for 120 records. It is not 420, because the abandoned transaction never became a second visible contribution.
Some execution happened twice, while the defined output contribution became visible once under the stated transaction, recovery, and reader-isolation assumptions.
The living notebook
Explore the model
Interactive lab · probability
Repeated delivery needs an effect protocol
For EACH run Reset, set BOTH p and n, predict, then Play or Step. With p=0.5, n=2 gives P(two successes)=0.25; n=4 gives P(more than one)=11/16. These exact independent attempt counts contain no transaction, sink, deduplication, or sampled execution.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA repeated successful attempt is possible; the application must establish how attempts map to committed effects.
Pause & reason
Check your understanding
Your turn
Try it yourself
A destination committed transaction T, but its success response was lost before the stream committer recorded completion. What recovery property is needed, and why is simply issuing a new unrelated transaction unsafe?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The missing acknowledgment does not imply the destination failed to commit; preserve transaction identity and the external outcome boundary.
Compare with the solution
Recovery must resolve or repeat the commit for the same logical transaction safely under the destination's protocol, without creating a second effect. Starting a new unrelated transaction can expose the same output twice if T already committed. Durable transaction identity, reader visibility, and connector recovery rules establish the answer.
Keep the idea
Take it with you
- Exactly-once contributions can require replayed execution after failure.
- Transactional output needs destination cooperation and matching reader isolation.
- External side effects require their own compatible commit or idempotency boundary.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.