advanced25 min studyLesson 302 in Engineering

A watermark is a progress claim with an input policy

Compute minimum input progress, explain idle-source exclusion and late resumption, and separate event time from processing time and checkpoint barriers.

On this page

What you’ll understand

  • Distinguish event timestamps from current processing time.
  • Calculate downstream watermark progress from active inputs.
  • Explain why declaring an input idle changes completeness assumptions.

A worker can be busy processing old events while another worker has already seen much newer timestamps. The current wall clock does not tell an event-time operator whether more old data will arrive. Watermarks communicate a policy-dependent claim about progress through event time. They enable decisions despite unbounded input, but their usefulness depends on how sources estimate progress and how the application handles evidence that arrives later than expected.

Separate a record's time from the machine's clock

Event time is a timestamp attached to the event under the application's definition, such as when a sensor sampled a value. Processing time is when an operator executes work according to its local clock. Network delay, batching, retries, and replay can put those times far apart. Event-time aggregation can preserve an intended temporal grouping despite changed arrival order, but only if timestamps, progress policy, and retained state support it. A watermark is not a measurement that all producers share synchronized clocks, and it does not repair incorrect timestamps.

A bounded-out-of-orderness strategy can derive a watermark from the largest timestamp observed minus a configured tolerance, with implementation-specific discrete-time details. This is a strategy, not a proof that no older event exists. A producer outage or delayed partition can violate the expectation. Name the timestamp units and boundary convention when computing examples; a one-millisecond adjustment can matter at a window edge. Watermarks also differ from checkpoint barriers: one advances event-time decisions, while the other coordinates a recoverable computation cut. Receiving either is not itself an external output commit.

Willustrative=Tmax⁡−δW_{\rm illustrative}=T_{\max}-\delta
T_max and tolerance δ use the same event-time units. This continuous-time teaching expression omits a generator's discrete timestamp adjustment; a source strategy determines actual emitted watermarks and its assumptions about future lateness.↔ Scroll wide equations horizontally.

The slowest relevant input can hold progress back

An operator combining inputs generally advances using the minimum relevant input watermark, so a fast input cannot unilaterally certify progress for a slow one. Flink 2.1's source idleness support can mark an input idle after a configured lack of activity, excluding it from that minimum while it remains idle. Without such a policy, a silent partition can stall event-time windows indefinitely. With it, other inputs can progress, but the silent partition has not been proven complete; a later resumption can bring events behind the advanced watermark.

An idle timeout is therefore a liveness-versus-completeness policy with a defined clock and observation point. Check whether silence means no business activity, a failed reader, backpressure, or a broken source. Late resumed events need a chosen disposition: update retained state, route to an audit path, or reject according to the contract. Watermarks are monotonic progress at an operator; resuming an old input cannot simply rewind already emitted results. Watermark alignment is another mechanism that can constrain fast sources to limit buffering skew, and should not be confused with checkpoint-barrier alignment.

Wout=min⁡i∈AWiW_{\rm out}=\min_{i\in\mathcal A}W_i
A is the nonempty set of active relevant inputs and W_i their progress timestamps in matching units. Idle exclusion and monotonic forwarding rules are policy/state dependent; the formula alone does not define behavior when all inputs are idle or finished.↔ Scroll wide equations horizontally.

Observe the reason for progress, not only the number

Monitor per-input event-time progress, processing-time inactivity, source lag, idle transitions, and late-event outcomes. A high output watermark after excluding one input can make a latency dashboard look good while that input's customers wait for corrections. Report completeness policy alongside freshness. Tests should pause one partition, advance another, resume the paused partition with an old event, and inspect both initial output and late handling. This trace tests the actual contract more directly than verifying that a watermark counter eventually increases.

The Markov lab models availability labels only as an exact two-state thought experiment. Reset, set BOTH p=0.2 from active state 0 to idle state 1 and q=0.3 back, predict idle probabilities 0.2 after one transition and 0.3 after two, then Step twice. One Step advances one transition because n=floor(2t). Edits retain t and recompute from initial active state. The displayed stationary idle probability is 0.4, not a measured idle rate. There are no timestamps, timeout detector, minimum watermark, or sampled source failures; this model cannot justify excluding a real input.

Exclude silence without erasing its risk

Two active input watermarks are 100 and 70 seconds of event time. The second input later meets the configured idleness rule while the first advances to 120. It then resumes with an event timestamped 80.

  1. Initially downstream progress is limited by min(100,70)=70 seconds. The first input's higher watermark cannot cover the second input's uncertainty.

  2. After the second input is explicitly excluded as idle, the active minimum can advance to 120 under the policy. This is progress enabled by an assumption, not proof that the second input ended.

  3. The resumed event at 80 lies 40 seconds behind the advanced watermark. Apply the defined late-event policy; do not rewind the watermark or silently pretend the event never mattered.

The result

Idle exclusion improves progress but creates a concrete late-resumption case that the application's output and retention policy must handle.

Explore the model

Interactive lab

Interactive lab · markov

Idle labels are assumptions, not completeness proofs

Reset, set BOTH p=0.2 and q=0.3, predict, then Step twice: exact idle probabilities are 0.2 and 0.3, with stationary value 0.4. Each Step is one transition; edits retain t/recompute from state 0. This two-state distribution is not sampled source behavior, an idle timeout, or a watermark algorithm.

Idle labels are assumptions, not completeness proofsExact finite-step probabilities for a two-state chain starting in state 0, not sampled trajectories. Playback advances two transitions per second. When p = q = 0 every distribution is stationary; when p = q = 1 the chain alternates forever. A single zero transition creates an absorbing state. Current readings: P(state 1) at step 0: 0%; Stationary P(state 1): 60%; Chain behavior: Monotone convergence. All parameters are adjustable with the labeled controls below.Two-state Markov distribution after 0 steps: state zero probability 1, state one probability 0.Initial distribution (1, 0) · step 0p = 0.3q = 0.201100%0%1−p1−qP(0)P(1)Monotone convergence
P(state 0)P(state 1)Stationary distribution
P(state 1) at step 0
0%
Stationary P(state 1)
60%
Chain behavior
Monotone convergence
t = 0 s

Exact finite-step probabilities for a two-state chain starting in state 0, not sampled trajectories. Playback advances two transitions per second. When p = q = 0 every distribution is stationary; when p = q = 1 the chain alternates forever. A single zero transition creates an absorbing state. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forAn input's availability label and a defensible event-time completeness claim are different pieces of evidence.

Check your understanding

An input is marked idle and excluded from a watermark minimum. What has the system established?

Try it yourself

Active inputs advertise watermarks 45, 52, and 41 seconds. Calculate the minimum. If only the 41-second input becomes idle and the others remain unchanged, what active minimum is available?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Remove only the explicitly idle input from the active set; do not average progress values.

Compare with the solution

The first minimum is 41 seconds. Excluding that input leaves min(45,52)=45 seconds, subject to the operator's monotonic forwarding rules. The four-second increase says nothing about whether the idle source has undiscovered events before 45.

Take it with you

  • Event time, processing time, watermarks, and checkpoint barriers have different roles.
  • Active-input minimums preserve a stated multi-input progress assumption.
  • Idle exclusion needs late-resumption behavior and completeness visibility.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.