Keep the conditions that make an incident lesson usable
Preserve mechanism evidence, applicability scope, verification history, and invalidating changes so runbooks remain useful as systems evolve.
On this page
The question for today
What you’ll understand
- Separate historical evidence from current applicability of a runbook action.
- Attach assumptions, verification artifacts, and invalidating changes to operational knowledge.
- Explain why retrieval and document freshness do not guarantee present correctness.
An incident report can remain accurate history while its recommended action becomes wrong for the current system. A cache reset that once released memory may now delete important state, or a retry rule may no longer match a new protocol's commitment boundary. Preserve the mechanism and the conditions that justified the lesson, not just the command that appeared to help. Operational knowledge needs an owner, a verification history, and a way to recognize when a relevant change invalidates its assumptions.
Record a claim with its scope and evidence
A useful incident record distinguishes observed impact, the supported mechanism, unresolved alternatives, mitigation, and verified corrective behavior. Link the relevant artifact versions, configurations, workload conditions, and regression fixture. A blameless account explains how the system and available information shaped decisions rather than stopping at a person's last action. That makes the record more reusable: a future investigator can compare mechanisms and evidence instead of matching one error string and repeating an old command.
Historical uncertainty should remain visible. If a replay covered only short prompts on one accelerator type, do not rewrite the conclusion as proof for every model and shape. If an omitted telemetry interval prevented a complete impact estimate, retain that gap even after service is restored. Updating a report can add new evidence without erasing what was known at decision time. Version the interpretation when it changes so later readers can distinguish the original response from later analysis.
Expiry can be triggered by context, not only age
Attach explicit invalidators to operational instructions: changed storage semantics, a different service identity, a new retry layer, altered memory allocation, or replaced telemetry definitions. A periodic review date is useful but cannot certify applicability between reviews. Conversely, an old mathematical explanation can remain valid when its assumptions still hold. Treat age as a scheduling signal, while context checks decide whether the particular action and evidence still apply.
A runbook can include a cheap preflight that checks relevant versions and modes, plus a supported verification step after the action. If the preflight fails, direct the responder to reassess rather than silently applying the old procedure. Automation should preserve the same boundary. An AI assistant retrieving a familiar incident entry needs the current configuration and applicable evidence; similarity of text does not establish permission or correctness. Keep human decision responsibility and automated checks connected to concrete system state rather than a document's confident tone.
Measure whether learning changed future behavior
Track whether corrective actions were deployed and verified, whether recurrence fixtures still run, and whether their assumptions match current interfaces. A low count of similarly titled incidents is weak evidence if detection or classification changed. Review new failures for mechanism overlap, including cases where the original safeguard worked but another path bypassed it. Learning is observable when future decisions and system behavior improve under comparable conditions, not merely when the report is published and a meeting is held.
The cache lab illustrates retrieval without semantic validation. Reset, set both C=4 and s=4, predict twelve hits in the sixteen initial requests, then inspect and Play or Step. Reset and set both C=4,s=1 for a separate zero-hit initial case. Playback is capped at 4,096 evaluated requests, and parameter edits retain time. The cache knows only numeric identifiers, not document versions, preconditions, or expiry. A hit demonstrates retrieval of a retained identifier; it supplies no evidence that an old instruction is applicable to today's incident.
Work it through
Apply an observed invalidation before the review date
A fictional runbook is verified on day 10 with a fourteen-day review interval. A documented incompatible storage-mode change occurs on day 18. The updated runbook is then verified on day 22 with the same review interval.
The original scheduled review date is day 10 + 14 = day 24. That is an administrative review deadline, not a promise that assumptions cannot change earlier.
The observed incompatible change triggers review on day min(24,18) = 18. The old action should not be treated as applicable merely because day 24 has not arrived.
After the revised action and fixture are verified on day 22, its new scheduled review date is day 36, subject again to earlier detected invalidators. Preserve the old report as history while identifying the new applicable procedure.
Context invalidation moves reassessment to day 18, and verified revision establishes a new day-36 schedule. A date alone never replaces the precondition check.
The living notebook
Explore the model
Interactive lab · cache
A retrieval hit does not validate an old instruction
Compare fresh Reset → set both C=4,s=4 and Reset → set both C=4,s=1 cases. Predict initial sixteen-request hits, then Play or Step. The capped LRU has no semantic document versions, context checks, expiry, or correctness oracle.
- Hits / requests
- 0 / 16
- Hit rate
- 0%
- Address-stream period
- 16 addresses
A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forIncident knowledge remains useful when its mechanism, assumptions, and current verification can be checked after retrieval.
Pause & reason
Check your understanding
Your turn
Try it yourself
A retrieved incident report says increasing retries fixed a timeout, but today's service adds an outer retrying proxy. What assumption should be checked before reusing the action, and what evidence would support an updated recommendation?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The retry counting boundary and feedback path may have changed even though the error string is identical.
Compare with the solution
Inspect total attempts across the new proxy and client layers, deadlines, retryable outcomes, and the downstream capacity mechanism. Reusing the old increase can multiply load rather than restore useful completion. An updated recommendation needs a current mechanism model, a controlled fixture or measured verification under relevant load, and user-outcome checks. Preserve the earlier report's historical scope while marking the changed counting assumption and new applicable procedure.
Keep the idea
Take it with you
- An accurate historical account and a currently applicable action are different artifacts.
- Context invalidators and preconditions can expire advice before its scheduled review date.
- Retrieval, publication, and task closure do not replace current mechanism verification.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.