Engineering / advanced
Stateful computation over unbounded input
Trace state, progress, replay, and recovery through streaming operators while stating the exact boundary of processing guarantees.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Stateful computation over unbounded input
Trace state, progress, replay, and recovery through streaming operators while stating the exact boundary of processing guarantees.
- 01
Partitioning assigns work; state gives that work a history
Distinguish source partitions, keyed state, operator state, and recoverable positions before reasoning about parallel stream execution.
24 min - 02
A watermark is a progress claim with an input policy
Compute minimum input progress, explain idle-source exclusion and late resumption, and separate event time from processing time and checkpoint barriers.
25 min - 03
Window assignment is a rule over time and identity
Calculate half-open tumbling and sliding membership, merge event-time sessions, and distinguish assignment from triggering, retention, and output updates.
25 min - 04
Retaining late-data state is part of the answer's contract
Separate watermark-based window cleanup from processing-time TTL, estimate retained payload, and define how expired or late corrections are handled.
25 min - 05
A distributed snapshot must account for records between operators
Reason about consistent cuts, aligned checkpoint barriers, captured in-flight data, and why unaligned checkpoints differ from merely skipping alignment.
27 min - 06
Live state storage and durable checkpoint storage are different layers
Separate state access from checkpoint persistence, explain incremental file reuse, and estimate upload and recovery costs without equating changed keys with changed storage bytes.
25 min - 07
Exactly-once state recovery does not automatically commit the outside world
Connect checkpoint recovery to transactional output visibility, idempotent commits, consumer isolation, and explicit limits for external side effects.
27 min - 08
Rescaling moves stable key groups rather than inventing new histories
Track stable key-group identity, parallel-owner reassignment, serializer compatibility, and skew while distinguishing logical redistribution from actual restore traffic.
26 min - 09
Replaying offsets is not enough when computation changes its inputs
Control ordering, randomness, lookup versions, code changes, and external effects so recovery can reproduce a defined computation rather than merely reread records.
27 min - 10
Choose when to answer and how an answer can improve
Balance event-time completeness, provisional output latency, retained-state cost, and correction protocols using measurable assumptions rather than promising immediate final truth.
27 min