A replay tests a counterfactual only within its captured world

Specify the mechanism being changed, such as a cache-capacity rule, retry budget, or allocation lifetime, and define the outcome that would demonstrate the failure. Keep that outcome distinct from incidental logs. If the suspected problem is duplicate commitment, an HTTP success response is not a sufficient oracle; inspect the committed effects. If it is GPU memory retention, latency alone can miss a leak that has not yet exhausted the device. The replay must expose the claimed mechanism rather than simply produce a familiar error string.

Capture artifact identities, configuration, request ordering, relevant data snapshots, and initial resource state. A cold cache and a warm cache are different conditions even when requests are identical. Seeds can help make some random choices reproducible, but do not control thread scheduling, asynchronous dependencies, or hardware nondeterminism by themselves. Explicitly stub or record external services when appropriate and state what realism that removes. A simulator that assumes the desired behavior cannot independently verify that behavior in the real dependency.

Run baseline and changed configurations against the same declared case set, controlling reset and warm-up rules. Record repeated runs when nondeterminism matters and avoid treating many requests within one shared run as independent replicas. If the baseline no longer reproduces the failure, investigate whether the captured context is missing the triggering condition before claiming the patch fixed it. A positive control helps show that the oracle detects the old failure, while negative controls test that the harness does not label unrelated behavior as the mechanism.

A successful replay supports a bounded claim: under these inputs, state, and modeled dependencies, the change altered the specified outcome. Generalization to production needs evidence that the omitted conditions are not essential or tests that exercise them. Shared-resource interference, long-duration accumulation, different prompt shapes, and real cancellation timing can all lie outside a short deterministic fixture. Preserve those limits in the incident report so later readers do not turn a narrow reproduction into a universal causal proof.

The cache lab provides a small fully specified replay world. Its request sequence is address (i·s) mod 16, initial cache empty, and replacement rule LRU. That makes a capacity intervention inspectable without guessing hidden server state. It does not represent a model-serving cache's concurrency, serialization, entry sizes, or admission policy. A correct conclusion about this world can motivate a real fixture, while a mismatch in the real fixture can reveal which assumption was missing.

Reset, set both C=3 and s=4, predict zero initial hits over the sixteen requests already evaluated, then inspect and Play or Step. Reset again and set both C=4,s=4 to compare twelve initial hits with the same trace and initial-state rule. Playback extends the deterministic trace up to 4,096 requests; edits retain time, so Reset is required for equal-length comparisons. The capacity change causes the hit-count difference in this explicit model, but does not by itself establish what caused a production incident.