Replaying offsets is not enough when computation changes its inputs
Control ordering, randomness, lookup versions, code changes, and external effects so recovery can reproduce a defined computation rather than merely reread records.
On this page
The question for today
What you’ll understand
- Identify hidden inputs that can change a replayed result.
- Distinguish order-independent mathematics from implementation-level numerical reproducibility.
- Design replay fixtures that inspect durable effects as well as internal state.
The same source offsets can produce a different result if a replay reads a new model, calls a changed lookup service, observes a new clock value, or consumes randomness differently. Recovery semantics need a defined computation, not just a record range. List every input that influences the answer, including inputs the program never serialized, and decide which must be versioned, checkpointed, or removed from the replay-sensitive path.
Make hidden inputs explicit
A deterministic state transition maps a prior state and a defined event to one next state and output. A current wall-clock read, mutable external lookup, random draw, thread race, or changing configuration adds another input. Merely rereading the event does not recreate that value. For model enrichment, pin the model and feature version or retain the original lookup result when the contract requires replaying the original decision. An alternative contract can intentionally recompute with current information, but then label it as recomputation rather than reproduction of the original result.
A random seed helps only with additional control over generator state, algorithm, call ordering, and partitioning. Rescaling or a new code branch can consume draws in a different order despite the same initial seed. Using processing-time timers can also make event-to-output timing or expiry depend on replay speed. Event-time rules reduce some timing dependence, but watermark and late-data policies still matter. Record code, schema, serializer, configuration, and artifact versions alongside source positions so an audit can reconstruct which computation was intended.
Ordering and arithmetic can change the final bits
Interleaving across partitions can change after recovery. An associative, commutative exact aggregate can tolerate some reorderings mathematically, while a sequence-sensitive state machine cannot. Floating-point addition is not exact real addition, so even a mathematical sum can change its final bits under another reduction order. Decide whether the contract requires bitwise equality, a justified numerical tolerance, or an invariant such as a conserved integer count. Do not apply a tolerance to identifiers or financial-style integer units merely to make a replay comparison pass.
An external side effect is not a pure replay input or output unless the protocol makes repeated execution compatible. Sending an HTTP request again can read different data or commit a second effect. Managed-state checkpoints cannot roll back arbitrary remote work. Use a compatible transactional or idempotent mechanism where required, and retain evidence of its outcome. Distinguish a function being executed twice from a logical contribution becoming visible twice. A deterministic function can still duplicate a side effect if called again without a correct external boundary.
Compare a trace against the actual replay promise
Build a fixture containing input records, initial state, source positions, artifact versions, lookup responses, and the expected output invariant. Run it through a failure-free path and a restore path with deliberately changed partition timing. Include a failure after external commit but before its acknowledgment, and inspect durable effect count. If current-data recomputation is intentional, compare against that separately specified target. Version changes need an explicit state migration or compatibility decision; a successful restart that silently changes the transition function does not establish equivalent results.
The probability lab demonstrates why identical distributions are weaker than identical histories. Reset, explicitly set BOTH p=0.5 and n=10, inspect the exact binomial bars, then Play or Step. Reset and re-set BOTH identical controls for comparison: the bars remain identical because the lab computes probabilities, not sampled outcomes. Edits retain elapsed time; playback only highlights a bar. This does not show a seeded event sequence or a deterministic stream processor. Explain how two different ten-trial histories can have the same success count and distribution, then identify which actual sequence and state inputs a replay test must preserve.
Work it through
A changed lookup is a changed computation input
A stream enriches 100 events with a model coefficient. The original run uses coefficient two and sums one unit from each event; a replay consults an unversioned service now returning coefficient three. Ignore all numerical error.
The original aggregate is 100×1×2=200 units under the original lookup value.
The same 100 event positions with coefficient three produce 100×1×3=300 units. No event was lost or added; the hidden enrichment input changed.
Pin coefficient version two or retain the original enrichment if the contract requires reproducing 200. If recomputing with version three is intentional, label 300 as a new result version and update the external output consistently.
Matching source positions did not reproduce the original answer. Versioned decision inputs and an explicit recomputation contract determine correctness.
The living notebook
Explore the model
Interactive lab · probability
Same probabilities do not mean the same execution
Reset, set BOTH p=0.5 and n=10, inspect, then Play or Step. Reset and re-set BOTH identical values to compare the same exact binomial distribution. Edits retain t; playback highlights outcomes, not sampled histories. No random generator state, event ordering, checkpoint, lookup, or side effect is replayed.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forDistributional agreement and reproducible execution are different claims; replay evidence must cover the decision inputs and state that actually matter.
Pause & reason
Check your understanding
Your turn
Try it yourself
Two replay runs process integer events with values 2, 5, and 3. One uses order 2,5,3 and another 5,2,3. Compare an exact integer sum with a state machine returning the first value. What must the contract specify?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Ask whether the operation is invariant under this reordering, not merely whether the event multiset matches.
Compare with the solution
Both exact sums are ten, assuming no overflow, but the first-value outputs are two and five. A multiset-equivalent replay is enough for the first invariant but not the second. The sequence-sensitive contract needs a defined ordering or another explicit decision rule, along with any external effect semantics.
Keep the idea
Take it with you
- Code versions, clocks, random state, and lookup results are computation inputs.
- Order invariance and numerical reproducibility need explicit arguments.
- Replay tests must verify durable effects and the promised result version.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.