Rank explanations while making their assumptions inspectable
Use declared priors and likelihoods in a toy diagnosis, avoid double-counted evidence, and choose observations that can contradict a preferred hypothesis.
On this page
The question for today
What you’ll understand
- Distinguish evidence likelihood from posterior hypothesis probability.
- Update a deliberately limited hypothesis model using explicit assumptions.
- Choose a disconfirming test instead of collecting only agreeable signals.
An incident usually has more plausible explanations than the team can test immediately. Ranking them helps allocate attention, but an unexplained confidence score can turn intuition into false precision. Either state qualitative reasons clearly or declare the probabilistic model that produces a number. A useful ranking remains revisable and includes evidence that would move a favored explanation down. The model in this lesson is a fictional two-hypothesis exercise, not a probability estimate for an actual production incident.
Separate a hypothesis from its prediction
Suppose inference latency might arise from cache churn or GPU throttling. Each hypothesis should predict observations beyond the shared symptom: cache-miss behavior under a fixed request trace, device clocks under comparable work, or performance after a controlled change. Record which measurements would be surprising under each explanation. A broad statement such as the system was overloaded can accommodate almost anything; a useful hypothesis names the mechanism, affected boundary, and conditions that should reproduce the symptom.
A prior represents the model's starting distribution before the specified evidence. A likelihood is the probability of that evidence if a hypothesis were true. Neither is automatically measured merely because an experienced engineer supplied it. If values are subjective or based on another service, say so and examine sensitivity. Also allow for explanations omitted from the list. Treating two convenient hypotheses as exhaustive can force high posterior confidence even when the real mechanism is absent from the model.
Do not count one underlying signal several times
A cache-miss metric, a log derived from that metric, and an alert computed from the same series are not three independent pieces of evidence. Multiplying their likelihood ratios as if independent can greatly exaggerate confidence. Distinguish new evidence from a transformed copy of existing evidence. Sequential updates require the appropriate conditional likelihood of the next observation given what is already known, not repeated use of an unconditional number because three dashboards display it.
Sensitivity checks can expose whether a ranking is robust or merely reflects a strong prior. Recalculate using a defensible alternative prior and document which evidence changes the ordering. A ranking need not be a posterior probability to be useful: expected diagnostic value, time, disruption, and reversibility also influence the next test. During mitigation, a low-cost action that reduces impact can be reasonable before diagnosis is complete, provided its result is not later misrepresented as a controlled causal experiment.
A preferred explanation needs an unfriendly test
If cache churn is favored, define a supported replay that holds workload, cache capacity, and initial state fixed while measuring the predicted reuse behavior. A test that only checks whether latency is still high does little to distinguish cache churn from GPU throttling. Specify the expected result under both hypotheses before looking, and preserve a negative or contradictory result. Otherwise the team can keep changing the explanation until every observation appears confirmatory.
The cache lab provides an exact toy prediction to challenge. Reset, set both C=3 and s=4, predict zero hits in the sixteen initial requests, then inspect and Play or Step. Reset and set both C=4,s=4 for a separate prediction of twelve initial hits. The stream starts empty but already evaluates sixteen requests at time zero, and caps at 4,096 requests. This deterministic mechanism test is not a Bayesian incident model and contains no GPU; its value is that a claim makes a precise prediction that can be checked.
Work it through
Update an explicitly fictional diagnosis
Assume exactly two hypotheses: cache churn C with prior 0.6 and throttling G with prior 0.4. For one defined observation E, assume likelihoods P(E|C)=0.8 and P(E|G)=0.2.
Compute the weighted evidence contributions: 0.6×0.8 = 0.48 for C and 0.4×0.2 = 0.08 for G. Their sum gives model evidence probability 0.56.
Normalize: P(C|E)=0.48/0.56=6/7≈0.8571 and P(G|E)=1/7≈0.1429. Equivalently, prior odds 1.5 multiplied by likelihood ratio four produce posterior odds six.
Report the result as conditional on the assumed two-hypothesis model. Before calling C established, test a prediction that distinguishes it from G and consider whether an omitted mechanism could also generate E.
The model ranks cache churn at about 85.7%, but the number inherits every prior, likelihood, and exhaustiveness assumption. It is not a measured probability of a real outage cause.
The living notebook
Explore the model
Interactive lab · cache
Make the preferred explanation predict something exact
Compare fresh Reset → set both C=3,s=4 and Reset → set both C=4,s=4 cases. Predict initial sixteen-request hits, then Play or Step. The capped 4,096-request LRU trace contains no incident priors, GPU state, or causal-identification procedure.
- Hits / requests
- 0 / 16
- Hit rate
- 0%
- Address-stream period
- 16 addresses
A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA useful hypothesis risks contradiction by predicting an observable mechanism under stated conditions.
Pause & reason
Check your understanding
Your turn
Try it yourself
Repeat the toy update with priors P(C)=0.2 and P(G)=0.8 while keeping likelihoods 0.8 and 0.2. What ranking results, and why is the comparison useful?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Multiply each prior by its likelihood before normalizing. Changing the prior does not change the observed event itself.
Compare with the solution
Both weighted contributions are 0.16, so the posterior is 0.5 for each hypothesis. The same evidence that produced 85.7% for C under the earlier prior now gives a tie. This sensitivity does not invalidate Bayes' rule; it exposes how the starting assumptions influence the ranking and why they should be documented rather than hidden behind a confident percentage.
Keep the idea
Take it with you
- A posterior ranking is conditional on explicit hypotheses, priors, and likelihoods.
- Transformed copies of one signal are not independent evidence.
- Disconfirming predictions make explanations more useful than unfalsifiable stories.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.