Align the measurement process before aligning the story
Distinguish raw events, rolling windows, scrape and ingestion delays, and repeated change-point searches when reconstructing an incident onset.
On this page
The question for today
What you’ll understand
- Calculate how a rolling window delays the appearance of a step change.
- Separate plotted sample times from source-event and deployment-application times.
- Explain why many retrospective searches increase false-discovery opportunities.
Two curves can appear to change at different times because their measurement pipelines differ rather than because one service caused the other to fail. A rolling error fraction, a scraped gauge, and a late log batch summarize different time intervals. Before proposing an onset or change point, identify those intervals and the missing data. Also record how many signals and candidate cut times were searched. The best-looking split found after extensive exploration is not equivalent to a prespecified hypothesis tested once.
A plotted point can summarize a preceding window
A trailing rate or fraction can smooth a sharp underlying change over its window. Query step controls where results are evaluated, while scrape cadence and record timestamps determine the available observations. Changing graph resolution does not create new raw samples. Prometheus distinguishes instant queries and range queries evaluated at successive steps; the query expression and lookback behavior remain essential context. Save the query, window, step, label filters, and data source with the incident evidence so a later graph is reproducible.
A deployment's control-plane timestamp can also precede its application at a particular worker. Telemetry ingestion can occur after the work it records. Compare event-time intervals with actual artifact and configuration evidence instead of shifting curves until they look causally convenient. When a time alignment is exploratory, label it as such. A measured lag can suggest a propagation mechanism, but the measurement process and shared causes remain alternative explanations that need separate tests.
Searching many cut points changes the evidential claim
If an analyst inspects many metrics, regions, window widths, and cut times, some apparent changes can arise under an unchanged system. Ordinary fixed-test significance levels do not automatically describe the selected strongest result. Overlapping windows create dependence, and repeatedly stopping when a convenient threshold appears changes the procedure again. Preserve the search history and use an appropriate prespecified or sequential method when making formal error-control claims. Exploratory findings are valuable inputs to mechanism tests, not reasons to conceal how the finding was selected.
A union bound offers a simple illustration for a finite family of individually valid tests: the probability of at least one false positive is at most the sum of their individual bounds. It does not require independence. For independent tests with identical false-positive probability, a different exact formula is available, but correlated telemetry usually does not meet that premise. Neither calculation is a probability that a selected incident explanation is true. Formal inference needs a procedure matching the actual search and data-generation process.
Use onset estimates to design a sharper check
After identifying an interval in which behavior changed, ask what mechanism predicts its cohort, delay, and direction. A configuration applied only to one worker pool should not automatically explain an earlier shift in an untouched pool. Conversely, shared dependencies can propagate after different buffering delays. Compare raw counts, workload mix, and instrumentation versions, then test the proposed mechanism in a supported context. An onset estimate should narrow the investigation rather than become an unquestionable timestamp around which every record is interpreted.
Use the probability lab for a deliberately independent false-positive illustration. Reset, set both p=0.05 and n=20, predict the chance of at least one success, then inspect the exact bars and Play or Step. Here success means a false positive under the toy null model. The complement of the zero bar is about 0.642. Real correlated or repeatedly inspected change-point tests need their own analysis; playback does not search telemetry or produce a valid incident p-value.
Work it through
Locate a threshold crossing created by smoothing
Assume constant request rate and an underlying error fraction that jumps at t=0 from 1% to 11%. A continuous trailing 60-second average was at its old steady state before the jump. When does it reach 5%?
For 0≤t≤60 seconds, the window contains t seconds at 11% and 60−t seconds at 1%. Its fraction is 1% + 10%×t/60.
Set that expression to 5%: the required rise is four percentage points out of the ten-point jump, so t/60=0.4 and t=24 seconds.
The plotted threshold crossing occurs twenty-four seconds after the underlying change in this model. It is not evidence that the physical failure began at t=24 or that an event near that display time caused it.
The smoothing model reaches 5% at twenty-four seconds even though the assumed raw change occurred at zero. Measurement alignment must precede causal alignment.
The living notebook
Explore the model
Interactive lab · probability
A search creates more chances for a false alarm
Reset, set both independent false-positive probability p=0.05 and n=20, predict one minus the zero bar, then Play or Step. The binomial instrument has no correlated telemetry, rolling-window search, or optional-stopping correction.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forAn exploratory change point needs its search procedure and measurement windows documented before formal claims are made.
Pause & reason
Check your understanding
Your turn
Try it yourself
For the same ideal 60-second window and 1%-to-11% jump, what fraction is displayed at t=30 seconds? Separately, what finite-family union bound applies to ten individually valid false-positive probabilities of 0.01?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The first answer weights equal halves of the window. The second adds valid individual probability bounds without assuming independence.
Compare with the solution
At thirty seconds the displayed fraction is 1% + 10%×0.5 = 6%. The union bound for at least one false positive is at most 10×0.01 = 0.10. That bound is not an exact probability for dependent tests and does not validate individual tests whose calibration was broken by an unaccounted search or stopping rule.
Keep the idea
Take it with you
- Windowing, scrape cadence, query step, and ingestion delay change displayed onset.
- Retrospective searches and repeated looks require procedure-aware interpretation.
- Temporal alignment suggests tests but does not establish a causal mechanism.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.