Declare recovery against exercised conditions and explicit oracles
Verify user outcomes, drained state, restored capacity, and recurrence fixtures while interpreting a clean finite observation window under its sampling assumptions.
On this page
The question for today
What you’ll understand
- Separate successful mitigation commands from verified user recovery.
- Interpret a fixed-sample zero-failure upper bound without calling it certainty.
- Choose verification conditions that exercise delayed and previously hidden failure paths.
A rollback completing and an error graph turning green are useful observations, but recovery needs a defined user outcome and a tested system state. Queues can still contain poisoned work, old connections can hide a bad new handshake path, and a fast AI fallback can return incorrect content while transport metrics look healthy. State what restored means, choose a verification window that exercises the relevant conditions, and preserve the evidence. A clean interval supports a bounded conclusion, not a promise that recurrence is impossible.
Check the outcome at the user's boundary
Define the service-level indicators and semantic checks relevant to the incident. If users received empty recommendations, verify useful results as well as HTTP status and latency. If a backlog caused delayed jobs, check completion and age rather than only a falling admission rate. Inspect affected cohorts separately so a healthy majority does not hide one broken region, model version, or tenant. Keep the denominator and success definition consistent across the incident and recovery views.
Also verify the restoration mechanism: intended artifact loaded, correct routing, released resources, required redundancy, and normal dependency behavior. A mitigation can mask the symptom by dropping useful work or bypassing a feature, which may be an acceptable degraded state but should be labeled. Recovery communication should distinguish mitigated, restored under tested conditions, and corrective work still pending. That makes the claim actionable without waiting for an unrealistically absolute proof that every future condition is safe.
Zero observed failures is not a zero failure probability
Under a fixed predeclared sample of independent identically distributed Bernoulli trials with constant failure probability, a zero-failure result supports a calculable one-sided upper confidence bound. It does not estimate every unexercised workload or correct biased selection. Correlated bursts, changing load, missing telemetry, and repeated peeking until a clean run appears invalidate the simple fixed-sample interpretation. State those assumptions alongside the number rather than attaching a confidence label to an arbitrary green dashboard interval.
A confidence bound is a property of a procedure's repeated-sampling coverage under its model, not automatically a posterior probability that this system's true rate lies below the reported value. Nor does a bound on transport errors establish semantic correctness. Pair rate-based evidence with deterministic recurrence fixtures and targeted boundary checks. For rare failures, the required sample can be much larger than intuition suggests, and a production risk decision can also depend on impact and uncertainty outside the chosen model.
Exercise delayed conditions before closing the claim
Choose conditions that can reveal the original mechanism after immediate traffic settles: new connection creation, cache expiration, delayed jobs, restart, peak-shaped requests, or cancellation after a timeout. A window's duration matters only through the conditions it actually contains. Waiting longer during idle traffic does not exercise a missing high-concurrency path. Record which recurrence tests passed and which conditions remain untested, then assign follow-up verification rather than quietly expanding the meaning of the recovery statement.
The sampling lab makes a clean-run limitation visible. Reset, set both p=0.01 and n=20, predict the exact zero-failure probability, then inspect and Play or Step. Reset and set both p=0.01,n=100 for a separate comparison. Even the latter zero bar is about 0.366 under the synthetic iid model. Playback only highlights outcomes; it is not a monitor or sequential test. Use exact bars for this rare-event question rather than treating the normal approximation or its CDF bound as a guarantee about a single outcome.
Work it through
Evaluate a clean fixed sample
A fictional predeclared iid verification sample of 100 requests has zero defined failures. Use a one-sided 95% binomial upper confidence endpoint. Then ask how many zero-failure trials would make that endpoint at most 0.1% under the same model.
Set α=0.05 and n=100. The endpoint is 1−0.05^(1/100) ≈ 0.02951, or 2.951%. One hundred clean requests therefore do not support a 0.1% bound under this procedure.
Require (1−0.001)^n ≤ 0.05, giving n ≥ log(0.05)/log(0.999). The smallest integer is 2,995 trials, all with zero failures for this endpoint calculation.
This is a fixed-sample model calculation, not a recommendation to keep inspecting until a convenient run is found. Dependence, selection, different failure definitions, and untested recurrence conditions require separate evidence.
The hundred-request clean run has a roughly 2.951% upper endpoint; the stated 0.1% target needs at least 2,995 clean fixed-sample trials under the stringent assumptions.
The living notebook
Explore the model
Interactive lab · sampling
A rare failure can leave a clean observation window
Compare fresh Reset → set both p=0.01,n=20 and Reset → set both p=0.01,n=100 cases. Predict exact zero-count mass, then Play or Step. The model assumes iid trials and highlights outcomes; it does not verify recovery, stopping rules, or semantic service quality.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA clean finite window is conditional evidence whose strength depends on sampling, workload, and the definition of failure.
Pause & reason
Check your understanding
Your turn
Try it yourself
For a fixed iid sample of 1,000 requests with zero failures, calculate the one-sided 95% upper endpoint using 1−0.05^(1/1000). Explain one reason that result does not prove a corrected AI service returns useful answers.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The endpoint is close to three divided by the sample size. Distinguish the measured failure indicator from semantic correctness.
Compare with the solution
The endpoint is approximately 0.002991, or 0.2991%, under the fixed iid Bernoulli model. If the failure indicator counts only HTTP errors, fast empty or incorrect AI responses can remain outside it. Verify the semantic outcome and relevant cohorts with an appropriate oracle rather than promoting a transport-error bound into proof of useful service or universal recovery.
Keep the idea
Take it with you
- Recovery is verified against explicit user outcomes and exercised system conditions.
- Zero observed failures supports a conditional finite-sample statement, not zero future risk.
- Delayed recurrence fixtures and semantic checks complement rate-based evidence.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.