Recovery traces: bind the observed run to artifacts and fault oracles

Record logical run, incarnation, step, attempt, provider operation identity, parent-child links, and the corresponding journal sequence. OpenTelemetry supplies concrete span and context APIs for diagnostic tracing, including parent relationships and links. Sampling, exporter failure, or retention can leave those diagnostics incomplete. A missing span is not proof that a tool never ran, and a success span is not an authoritative provider receipt. Keep the durable control record and effect evidence distinct from optional observability data.

Bind each decision to tool/schema version, canonical arguments, input artifact versions, model response artifact, approval identity, and relevant budget transitions. Record result and output artifact digests with controlled access to the actual content needed for recovery. A hash identifies bytes under its stated assumptions; it does not prove the artifact was approved, truthful, or available. Avoid relying on mutable file names or model aliases as the complete identity of an executed action. Record amendments as new evidence when reconciliation changes what is known.

Inject crashes before dispatch, after provider commit but before acknowledgement, after result observation but before journal append, and during cleanup or compensation. Use an instrumented provider or controlled fixture that records authoritative logical effects independently of the worker. Test what happens when it honors keys, expires them, rejects payload reuse, or lacks idempotency support. A simulator can validate the proposed protocol under its assumptions; it cannot establish that an untested production provider has the same guarantees.

Compare effect counts, resource versions, accepted approvals, outstanding budget reservations, and ownership of every child or cleanup obligation. Replay the same accepted history with the intended compatible code and recorded nondeterministic inputs, then verify command identities and outputs. A model-quality evaluator answers another question: whether the chosen plan was useful. Runtime recovery tests should not pass merely because a final response sounds plausible or because a mocked tool returned a success-shaped JSON object.

Name each fixture's initial state, logical intent, fault location, provider behavior, expected recovery, and final authoritative observations. Preserve minimized failing histories and compatible artifacts. Include duplicate and delayed messages, concurrent cancellation, changed approvals, schema evolution, and expired deduplication records. A single invariant violation matters even when most runs recover; aggregate availability can conceal a systematic commit-gap failure. Conversely, a test that explicitly expects an unresolved obligation can pass when the runtime safely refuses an unsafe retry.

The probability lab illustrates only the chance of missing a hypothetical independent defect. Reset, set p=0.05 and n=20, predict mean one defect and zero-defect probability about 0.3585, then Play or Step. Separately Reset, set p=0.05 and n=30, predict mean 1.5 and zero-defect probability about 0.2146, then Play or Step. It performs no fault injection and assumes identical independent trials. Real crash windows are structured cases, so deterministic boundary coverage and independent oracles remain necessary even after many apparently clean runs.