Build a timeline that preserves what the evidence can prove
Reconstruct fictional service incidents using event identity, clock uncertainty, collection provenance, and explicit separation of observation from interpretation.
On this page
The question for today
What you’ll understand
- Distinguish source event time from collector observation time.
- Apply stated clock-error bounds before asserting cross-host order.
- Preserve evidence and mark inference separately from direct observation.
An incident timeline is an argument assembled from observations, not merely a sort of timestamp strings. Different machines can disagree about time, collectors can delay delivery, and a log line can describe work that began earlier. A useful reconstruction records what each source actually measured and how that evidence was preserved. The examples in this module use fictional AI and infrastructure incidents; none is presented as a previously reported real outage. Keep uncertain ordering visible instead of smoothing it into a convenient story.
Give every observation a provenance
For each relevant event, retain the source service and instance, process or deployment identity, event identifier, raw timestamp and timezone, collection timestamp, and the query or export that produced it. OpenTelemetry distinguishes a record's event timestamp from the time it was observed by the collection system. That distinction matters when a telemetry buffer flushes late. A deployment event, a request error, and an operator's later recollection are different evidence types even if they appear on the same incident page.
Preserve original exports before filtering or joining them, and document transformations so another investigator can reproduce the derived view. Include missing ranges and dropped-record indicators rather than silently excluding them. Protect sensitive payloads and restrict access while retaining the identifiers needed for analysis. A digest can help show that an exported artifact has not changed since capture, but it does not prove that the original source was truthful or complete. Provenance supports scrutiny; it is not a substitute for it.
Cross-host order can remain unresolved
If two time intervals overlap, their wall-clock evidence alone may not determine which event occurred first. More printed decimal places do not reduce clock error. A message send/receive relation, a local sequence number, or a valid monotonic duration can add ordering evidence, but only under its own semantics. A monotonic clock is useful for elapsed time within its defined clock domain; values from unrelated hosts are not automatically comparable because both were called monotonic.
Record uncertainty in the timeline itself: definite before, definite after, or unresolved under the available bounds. Separate a code rollout's start, its arrival at one replica, and the first request that exercised the new path. An observed error preceding an uncertain deployment marker does not conclusively exonerate the release, nor does a nearby timestamp prove it caused the error. First establish which versions and operations were actually involved, then build and test a causal explanation.
Preserve a decision log alongside the event log
During mitigation, record the hypothesis motivating each action, its intended effect, who or what executed it, and when the effect was actually observed. Keep attempted, acknowledged, and verified changes distinct. A rollback command returning success is not evidence that every worker loaded the earlier artifact. This decision log lets later reviewers distinguish what responders knew at the time from explanations inferred after additional evidence arrived, and helps avoid blaming a decision using information unavailable to its maker.
The waves lab illustrates only how visible alignment can be changed without adding event evidence. Reset, set both phase difference to 0 degrees and amplitude ratio to 1, predict the aligned components, then Play or Step. Reset and set both phase to 90 degrees and ratio to 1 for a separate comparison. The two analytic waves are not service logs, clocks, or a causal model. Their similar shapes remind us to inspect provenance and ordering rather than treating an attractive overlay as proof of a shared incident sequence.
Work it through
Order three uncertain event times
In a fictional reconstruction, A is reported at 100 ms with a valid ±5 ms bound, B at 108 ms with ±7 ms, and C at 120 ms with ±2 ms. All refer to the same time origin.
Construct intervals: A lies in [95,105] ms, B in [101,115] ms, and C in [118,122] ms. The units and common origin are part of the premise.
A and B overlap between 101 and 105 ms, so these wall-clock bounds alone do not determine their order. A later collection timestamp would not resolve that source-time ambiguity.
A's latest possible time 105 ms and B's latest 115 ms are both earlier than C's earliest 118 ms. Therefore both precede C under the bounds, while A versus B remains unresolved.
The defensible timeline places A and B before C and leaves their mutual order uncertain. It establishes temporal constraints, not which event caused another.
The living notebook
Explore the model
Interactive lab · waves
Alignment is a presentation, not a provenance record
Compare fresh Reset → set both phase 0°,ratio 1 and Reset → set both phase 90°,ratio 1 cases. Predict component alignment, then Play or Step. These analytic waves do not model clock error, service events, or causality.
- Sum at x = 0
- 0.866
- Resultant amplitude
- 1.732
- Phase difference
- 60°
Two waves share the same frequency and wave number. Playback advances t in sin(2x − t) + a sin(2x − t + phase). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forSimilar plotted timing needs source and uncertainty evidence before it supports an incident timeline.
Pause & reason
Check your understanding
Your turn
Try it yourself
A request event is reported at 50 ms ±3 ms and a configuration application event at 58 ms ±2 ms on the same reference timeline. Can the request be ordered before configuration application, and does that prove the configuration could not affect any later errors?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Compare the latest request time with the earliest application time. Keep that one request separate from later requests and other replicas.
Compare with the solution
The request interval is [47,53] ms and application interval [56,60] ms, so this request precedes that application event under the stated valid bounds. That does not exonerate the configuration from later errors or establish what other replicas had loaded. Version-specific request evidence and a mechanism test are still needed for the broader causal claim.
Keep the idea
Take it with you
- Preserve event identity, source and observation times, transformations, and gaps.
- Justified clock intervals can constrain order while leaving some comparisons unresolved.
- Decision-time knowledge and later causal interpretation should remain distinguishable.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.