Late arrivals and feature corrections
Apply event revisions without duplicate contributions, regressing newer state, or erasing the history needed for as-known retrieval.
On this page
The question for today
What you’ll understand
- Distinguish duplicate delivery, late first arrival, and correction.
- Update retained aggregates by replacing the old contribution.
- Define replay-safe publication and preserve revision availability.
An event arriving twice is not the same problem as an event arriving late or an earlier event being corrected. Treating all three as append-only new activity can inflate model inputs. A correction path needs stable event identity, a revision policy, aggregate retraction or recomputation, and an online publication rule. It must also preserve what was known before the correction when historical serving reconstruction requires that evidence.
Classify the observation before changing state
A duplicate delivery repeats the same logical event version. A late first arrival introduces a previously unseen event whose event timestamp precedes current processing progress. A correction changes the meaning of an already known event, possibly its amount, qualification status, entity, or event time. These cases need different state transitions. Define an event identifier and monotonic or otherwise totally ordered revision identity supplied by a trusted source contract. If the same identifier and revision arrive with different payloads, flag a conflict rather than arbitrarily declaring one a harmless duplicate.
Application deduplication often needs to survive process restarts and replay. Its retained horizon must cover the replay and correction behavior the system promises. Expiring identity state too early can make an old duplicate look new after a recovery. Checkpointed stream state can help, but an external feature writer also needs compatible retry or transaction semantics. Flink’s state recovery guarantees do not automatically turn an arbitrary external append call into an exactly-once effect. Inspect the source, state, and sink boundaries together.
Retract or recompute the affected contribution
For a retained fixed window sum, replacing one known amount requires removing its old contribution and adding the corrected amount. If a correction changes membership, update the old and new entity/window destinations, not just the currently selected bucket. An event that stops qualifying can require a deletion or a tombstone. Minima, distinct counts, and quantiles need richer state or recomputation; subtracting an old scalar does not provide a general inverse for every aggregate. Preserve enough source history and policy to repair state after ordinary window cleanup.
Publication must identify the feature definition, entity, window or observation, and revision. A cumulative corrected result should replace its predecessor under that identity; a delta output needs a replay-safe delta identifier instead. A stale correction arriving after a newer revision must not move online state backward. Timestamp precedence and revision precedence must be explicit because a correction can retain an old event timestamp while carrying newer knowledge. Do not assume all online providers resolve these conflicts identically; test the configured writer and reader behavior.
Preserve temporal history and test replay schedules
An online latest-value representation can discard the earlier served value unless history is logged elsewhere. If training needs as-known joins, retain revision availability and earlier versions in an appropriate history store. The corrected value can be right for a truth query while being wrong for a decision made before its publication. Test duplicate replay, reordered revisions, restart after writing but before acknowledgment, entity reassignment, and a correction arriving after deduplication state would normally expire. Assert both final aggregate values and the permitted sequence of visible versions.
The modular instrument contrasts repeated changes with an effective no-op; it does not implement deduplication. Reset, set both m=8 and a=1, predict residue two after two additions, then Step twice. Separately Reset, set both m=8 and a=0, predict residue zero and period one, then Step twice. A nonzero addition changes state on each repetition, whereas a repeated already-applied event should have no additional effect under the contract. The zero-step case contains no event IDs, revision comparison, storage retries, or correction history; those mechanisms must be implemented in the actual pipeline.
Work it through
Replace one corrected contribution exactly once
A retained integer-valued window sum is 40, including event E at revision one with amount 7. Revision two changes E’s amount to 12 without changing its entity or window. The same revision-two message is then delivered again.
The first correction replaces the contribution: 40−7+12=45. It is not a new independent amount to append to the prior sum.
Recording E at revision two with its payload lets the replay be recognized. Applying the identical revision again leaves the sum at 45 under the stated idempotency contract.
Blindly adding twelve for the replay would produce 57, which is wrong. A later delivery of revision one must also not regress the stored contribution to seven; the revision and conflict rules decide that transition.
Correct aggregation requires both the right replacement arithmetic and durable recognition of which logical revision has already been applied.
The living notebook
Explore the model
Interactive lab · modular
A repeated operation needs an explicit no-extra-effect rule
Reset, set both m=8 and a=1, predict residue two after two additions, then Step twice. Separately Reset, set both m=8 and a=0, predict residue zero and period one, then Step twice. This additive model contrasts change with a no-op; it has no event identity, deduplication, revision ordering, or external sink.
- Residue r0
- 0
- Orbit period · m/gcd(m,a)
- 12
- Distinct residues visited
- 1 of 12
Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forIdempotency must be implemented at the identity and effect boundary; repetition alone does not provide it.
Pause & reason
Check your understanding
Your turn
Try it yourself
A fixed-window sum of 100 includes amount 20 from an event that is corrected to amount 5. Membership is unchanged. What should the sum become, and what happens on an identical correction replay?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Remove the prior contribution before adding its replacement.
Compare with the solution
The corrected sum is 100−20+5=85. An identical replay of that same event revision should leave 85 unchanged after deduplication or an equivalent idempotent replacement. Adding five again or forgetting which revision was applied violates the intended logical-event contract.
Keep the idea
Take it with you
- Separate duplicates, late first arrivals, and corrected versions.
- Update affected contributions and membership with aggregate-appropriate state.
- Make external effects replay-safe and retain availability history when required.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.