advanced25 min studyLesson 411 in Engineering

Build a timeline that preserves what the evidence can prove

Reconstruct fictional service incidents using event identity, clock uncertainty, collection provenance, and explicit separation of observation from interpretation.

On this page

What you’ll understand

  • Distinguish source event time from collector observation time.
  • Apply stated clock-error bounds before asserting cross-host order.
  • Preserve evidence and mark inference separately from direct observation.

An incident timeline is an argument assembled from observations, not merely a sort of timestamp strings. Different machines can disagree about time, collectors can delay delivery, and a log line can describe work that began earlier. A useful reconstruction records what each source actually measured and how that evidence was preserved. The examples in this module use fictional AI and infrastructure incidents; none is presented as a previously reported real outage. Keep uncertain ordering visible instead of smoothing it into a convenient story.

Give every observation a provenance

For each relevant event, retain the source service and instance, process or deployment identity, event identifier, raw timestamp and timezone, collection timestamp, and the query or export that produced it. OpenTelemetry distinguishes a record's event timestamp from the time it was observed by the collection system. That distinction matters when a telemetry buffer flushes late. A deployment event, a request error, and an operator's later recollection are different evidence types even if they appear on the same incident page.

Preserve original exports before filtering or joining them, and document transformations so another investigator can reproduce the derived view. Include missing ranges and dropped-record indicators rather than silently excluding them. Protect sensitive payloads and restrict access while retaining the identifiers needed for analysis. A digest can help show that an exported artifact has not changed since capture, but it does not prove that the original source was truthful or complete. Provenance supports scrutiny; it is not a substitute for it.

Ti∈[ci−ϵi,ci+ϵi]T_i\in[c_i-\epsilon_i,c_i+\epsilon_i]
c_i is an event's reported time and ε_i a justified absolute error bound in the same time unit. The interval claim is valid only if the bound and timestamp semantics apply to this event. Timestamp resolution or an enabled synchronization service alone does not establish such a hard bound.↔ Scroll wide equations horizontally.

Cross-host order can remain unresolved

If two time intervals overlap, their wall-clock evidence alone may not determine which event occurred first. More printed decimal places do not reduce clock error. A message send/receive relation, a local sequence number, or a valid monotonic duration can add ordering evidence, but only under its own semantics. A monotonic clock is useful for elapsed time within its defined clock domain; values from unrelated hosts are not automatically comparable because both were called monotonic.

Record uncertainty in the timeline itself: definite before, definite after, or unresolved under the available bounds. Separate a code rollout's start, its arrival at one replica, and the first request that exercised the new path. An observed error preceding an uncertain deployment marker does not conclusively exonerate the release, nor does a nearby timestamp prove it caused the error. First establish which versions and operations were actually involved, then build and test a causal explanation.

cA+ϵA<cB−ϵB ⟹ TA<TBc_A+\epsilon_A<c_B-\epsilon_B\ \Longrightarrow\ T_A<T_B
Under valid bounds for both events, nonoverlapping ordered intervals establish strict temporal order. The converse does not hold: overlapping intervals can hide a real order. Temporal order is necessary for many causal claims but does not alone establish a causal mechanism.↔ Scroll wide equations horizontally.

Preserve a decision log alongside the event log

During mitigation, record the hypothesis motivating each action, its intended effect, who or what executed it, and when the effect was actually observed. Keep attempted, acknowledged, and verified changes distinct. A rollback command returning success is not evidence that every worker loaded the earlier artifact. This decision log lets later reviewers distinguish what responders knew at the time from explanations inferred after additional evidence arrived, and helps avoid blaming a decision using information unavailable to its maker.

The waves lab illustrates only how visible alignment can be changed without adding event evidence. Reset, set both phase difference to 0 degrees and amplitude ratio to 1, predict the aligned components, then Play or Step. Reset and set both phase to 90 degrees and ratio to 1 for a separate comparison. The two analytic waves are not service logs, clocks, or a causal model. Their similar shapes remind us to inspect provenance and ordering rather than treating an attractive overlay as proof of a shared incident sequence.

Order three uncertain event times

In a fictional reconstruction, A is reported at 100 ms with a valid ±5 ms bound, B at 108 ms with ±7 ms, and C at 120 ms with ±2 ms. All refer to the same time origin.

  1. Construct intervals: A lies in [95,105] ms, B in [101,115] ms, and C in [118,122] ms. The units and common origin are part of the premise.

  2. A and B overlap between 101 and 105 ms, so these wall-clock bounds alone do not determine their order. A later collection timestamp would not resolve that source-time ambiguity.

  3. A's latest possible time 105 ms and B's latest 115 ms are both earlier than C's earliest 118 ms. Therefore both precede C under the bounds, while A versus B remains unresolved.

The result

The defensible timeline places A and B before C and leaves their mutual order uncertain. It establishes temporal constraints, not which event caused another.

Explore the model

Interactive lab

Interactive lab · waves

Alignment is a presentation, not a provenance record

Compare fresh Reset → set both phase 0°,ratio 1 and Reset → set both phase 90°,ratio 1 cases. Predict component alignment, then Play or Step. These analytic waves do not model clock error, service events, or causality.

Alignment is a presentation, not a provenance recordTwo waves share the same frequency and wave number. Playback advances t in sin(2x − t) + a sin(2x − t + phase). Current readings: Sum at x = 0: 0.866; Resultant amplitude: 1.732; Phase difference: 60°. All parameters are adjustable with the labeled controls below.−3.14−1.5701.573.14−3.4−1.701.73.4x · radiansAmplitude
SumFirst componentSecond component
Sum at x = 0
0.866
Resultant amplitude
1.732
Phase difference
60°
t = 0 s

Two waves share the same frequency and wave number. Playback advances t in sin(2x − t) + a sin(2x − t + phase). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forSimilar plotted timing needs source and uncertainty evidence before it supports an incident timeline.

Check your understanding

Two cross-host event-time intervals overlap under their justified clock bounds. What is the sound conclusion from those bounds alone?

Try it yourself

A request event is reported at 50 ms ±3 ms and a configuration application event at 58 ms ±2 ms on the same reference timeline. Can the request be ordered before configuration application, and does that prove the configuration could not affect any later errors?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Compare the latest request time with the earliest application time. Keep that one request separate from later requests and other replicas.

Compare with the solution

The request interval is [47,53] ms and application interval [56,60] ms, so this request precedes that application event under the stated valid bounds. That does not exonerate the configuration from later errors or establish what other replicas had loaded. Version-specific request evidence and a mechanism test are still needed for the broader causal claim.

Take it with you

  • Preserve event identity, source and observation times, transformations, and gaps.
  • Justified clock intervals can constrain order while leaving some comparisons unresolved.
  • Decision-time knowledge and later causal interpretation should remain distinguishable.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.