Delivery-system failure rehearsals

Choose points before and after an external effect, durable progress update, evidence check, or destructive state transition. A crash after traffic changed but before acknowledgment tests a different path from a crash before the request was sent. Delay or duplicate responses, return stale health for an earlier revision, remove telemetry, and interrupt an abort as separate cases. Use a bounded isolated environment or another explicitly controlled scope with an independent stop mechanism. Scope selection and cleanup matter because a misselected target can test an unrelated system rather than the intended release protocol.

Start from a known artifact, configuration, routing state, and data snapshot. Record operation identities, generation numbers, fault timing, and what each component could observe. The controller may need to relist or reobserve after disconnection; a replayed event should not be assumed fresh. A timed-out operation can already have completed externally. Rehearse restarts and compensations with that uncertainty intact rather than resetting the target to the happy path before the resumed controller can encounter the difficult state.

Define forbidden observations: an unapproved digest receives traffic, exposure exceeds its allowed cap, stale evidence promotes a new revision, compensation happens twice, or committed data violates the business invariant. Define liveness separately, including a bounded expected recovery condition when dependencies return. Use target-side routing observations, request traces, data reconciliation, and an independent workload probe where appropriate. The controller's status is one observation, but relying on that same controller to announce whether its own safety rule was violated creates a weak oracle.

Include mixed-version data cases. Let the candidate write legitimate new-format records before testing a rollback or forward repair. Keep long-lived sessions active through cutover, and distinguish retryable requests from already completed non-idempotent effects. Test missing and inconclusive metrics so that absence does not become a synthetic pass. For every case, name the evidence that would fail the test and preserve it even if the system later recovers. A temporary invariant violation still matters when a final steady-state dashboard looks normal.

Run deterministic traces first so a failure is reproducible, then add controlled timing variation or property-based exploration where it can reveal additional interleavings. Record the tested versions and assumptions. Passing a finite set does not prove correctness under every crash, arbitrary delay, Byzantine response, or concurrent operator action. Convert a discovered counterexample into a durable regression case and fix the protocol or invariant rather than merely lengthening a sleep until the trace passes. Rehearse the emergency stop and cleanup path as part of the same operational capability.

The Markov instrument illustrates a distinction between recovery and recurrence, not fault injection. Reset, set both p=0.5 and q=0, predict state-one probability 0.75 after two Steps, then Step twice. Separately Reset, set both p=0.5 and q=0.5, predict 0.5 after two Steps, then Step twice. It starts in state zero and displays exact two-state probabilities. The first state-one condition is absorbing; the second permits return. Neither setting tests a controller, data invariant, or real failure schedule. Explain why eventual recovery in a model is weaker than preventing every forbidden exposure in a rehearsal trace.