Engineering / advanced
Evidence and causal reasoning in incidents
Construct testable explanations from production evidence and verify corrective actions without reducing complex incidents to a single convenient cause.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Evidence and causal reasoning in incidents
Construct testable explanations from production evidence and verify corrective actions without reducing complex incidents to a single convenient cause.
- 01
Build a timeline that preserves what the evidence can prove
Reconstruct fictional service incidents using event identity, clock uncertainty, collection provenance, and explicit separation of observation from interpretation.
25 min - 02
A correlated release is a hypothesis, not yet a cause
Distinguish observational association, workload confounding, and a defined intervention using fictional model-serving error rates.
26 min - 03
Rank explanations while making their assumptions inspectable
Use declared priors and likelihoods in a toy diagnosis, avoid double-counted evidence, and choose observations that can contradict a preferred hypothesis.
26 min - 04
Align the measurement process before aligning the story
Distinguish raw events, rolling windows, scrape and ingestion delays, and repeated change-point searches when reconstructing an incident onset.
26 min - 05
Reconstruct how one failure can become many symptoms
Trace retries, fan-out, queues, and shared dependencies in a fictional inference service without mistaking span order or downstream symptoms for causal proof.
26 min - 06
The requests you can see may differ from the requests that failed
Analyze outcome-dependent trace retention, telemetry loss, inclusion probabilities, and known request denominators before estimating incident impact.
26 min - 07
A replay tests a counterfactual only within its captured world
Design paired failure replays with controlled inputs, initial state, dependencies, and oracles while limiting claims to the conditions actually reproduced.
27 min - 08
A corrective action needs a mechanism and an observable invariant
Turn incident findings into bounded admission, cancellation, and verification controls with explicit scope, ownership, and regression oracles.
26 min - 09
Declare recovery against exercised conditions and explicit oracles
Verify user outcomes, drained state, restored capacity, and recurrence fixtures while interpreting a clean finite observation window under its sampling assumptions.
27 min - 10
Keep the conditions that make an incident lesson usable
Preserve mechanism evidence, applicability scope, verification history, and invalidating changes so runbooks remain useful as systems evolve.
24 min