Minimize a reproducible failure without confusing a smaller trace with its root cause
Separate replay modes, preserve causal preconditions, use valid failure predicates, and distinguish deletion minimality from global minimality or causal explanation.
On this page
The question for today
What you’ll understand
- Define action replay, observation replay, and policy rerun boundaries.
- Explain one-deletion minimality with valid passing countertests.
- Calculate intermittent-failure detection probability under explicit iid replay assumptions.
A long failed execution contains observations, actions, timing, and state changes that may or may not be necessary for the failure. Replay and minimization can make it easier to investigate, but only when the replay preserves the relevant semantics. Removing a setup action can make a later request invalid rather than reveal that the action caused the bug. A compact failing trace is a useful counterexample; it is not automatically the smallest possible counterexample or an explanation of the policy's internal decision.
Choose the replay boundary and preserve its evidence
An action replay executes recorded actions against a reset fixture. An observation replay feeds recorded tool responses to a policy, potentially without reproducing their real effects. A fresh policy rerun can make different choices even from a similar snapshot. Name which mode is used and preserve initial state, versions, seeds, time assumptions, and external-response handling. Stubbed responses are useful for diagnosis but may remove the very commitment or delay behavior that caused the real failure. Do not treat a replayed success under a weaker simulator as proof that the original environment was correct.
Keep replay inside controlled fixtures. Reissuing a state-changing production request can create another payment, message, or file mutation instead of merely observing history. Preserve action identifiers and the intended idempotency semantics when those are part of the bug. Verify a failure predicate on authoritative evidence, such as duplicate committed effects or a specific violated invariant, rather than any exception string. Classify invalid or incomplete replays as unresolved when they cannot assess the intended failure; converting them into ordinary passes can mislead minimization and causal interpretation.
A minimal counterexample depends on the allowed transformation
Delta debugging systematically removes parts while retaining a chosen failure. Zeller and Hildebrandt's original work distinguishes minimality properties under that test contract; a one-deletion-minimal input need not be the globally smallest one. For agent traces, deletions can violate authentication, object creation, or causal ordering preconditions. Consider grouping dependent events or changing the fixture explicitly, then label that as a changed minimization space. Rewriting a tool response or amount can reveal another failure but is not the same operation as deleting an irrelevant read from an otherwise fixed action trace.
A minimized action sequence can demonstrate that two calls trigger duplicate effects without proving why the agent chose the retry or whether another independent defect also exists. Compare controlled counterfactuals and preserve the original trace alongside the reduction. The first visibly wrong output may be downstream of an earlier observation or state error, and a plausible blame explanation needs corroborating evidence. Classify environment, policy, grader, and instrumentation defects separately where the data permits it rather than assigning every failure to the last tool call.
Validate the reduction and preserve the unreduced failure
Replay the original and reduced cases under the same declared boundary, verify decisive state effects, and test the allowed single deletions before claiming the chosen minimality. For intermittent failures, preserve a deterministic trigger where possible or use a repeated statistical protocol with uncertainty and stopping rules. Do not accept an occasional pass as proof that a removed event was necessary. Record transformation history, unresolved candidates, and which environmental inputs were fixed. A good regression retains the failure mechanism while remaining small enough to run often and inspect meaningfully.
The probability lab illustrates intermittent detection under an assumed model. Reset, explicitly set BOTH p=0.2 and n=10, predict at least one failure with probability about 0.8926, then Play or Step. Reset and re-set BOTH p=0.2 and n=20; predict about 0.9885. Edits retain t; playback highlights exact iid counts. There are no traces, deletions, stateful tools, or minimization algorithms. The calculation cannot validate a replay fixture or establish independence; it only makes the stipulated sampling risk of missing an intermittent failure explicit.
Work it through
Reduce a duplicate-effect action replay
A reset sandbox starts at balance one hundred and expects one ten-unit debit. A fixed eight-event action replay contains six read-only observations and two unkeyed effective debit calls, the second a retry after a lost acknowledgment. Assume deleting reads leaves the calls valid and the failure predicate is duplicate committed debit.
The original action replay ends at 100−10−10=80 and fails the one-debit contract. Removing the six irrelevant reads leaves two valid calls and the same duplicate-effect failure.
Deleting either remaining call gives a single effective debit and balance 90, which passes this fixed predicate. Both single-deletion countertests are valid under the stated fixture.
The two-call trace is one-deletion-minimal in this action-deletion space. It does not prove a globally shortest failure under every input rewrite, explain the policy's original retry decision, or establish that a fresh policy rerun will choose those same actions.
A small replayable counterexample localizes an effect mechanism while preserving the limits of the minimization and causal claim.
The living notebook
Explore the model
Interactive lab · probability
One clean replay can miss an intermittent defect
For EACH run Reset, set BOTH p and n, predict, then Play or Step. At stipulated failure p=0.2, n=10 gives P(K≥1)≈0.8926 and n=20 gives≈0.9885. Edits retain t. This exact iid count model has no actual replay, causal state, delta debugging, fixture validation, or root-cause identification.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forReproduction and minimality claims require a valid replay contract and evidence, not only a short or occasionally failing trace.
Pause & reason
Check your understanding
Your turn
Try it yourself
Under an assumed iid replay failure probability 0.1, what is the chance of seeing no failure in ten valid replays, and of seeing at least one? Would that calculation apply automatically to ten replays sharing one corrupted fixture?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the complement event and examine whether the stipulated independence survives shared state.
Compare with the solution
No-failure probability is 0.9¹⁰≈0.3487, so at least one is 1−0.9¹⁰≈0.6513. A shared corrupted fixture can create dependent or changed conditions, so the iid calculation does not automatically apply; validate resets and the actual replay population.
Keep the idea
Take it with you
- Replay modes preserve different boundaries and must be named explicitly.
- One-deletion minimality does not establish global minimality or root cause.
- Invalid candidates and intermittent failures need their own recorded test semantics.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.