A correlated release is a hypothesis, not yet a cause
Distinguish observational association, workload confounding, and a defined intervention using fictional model-serving error rates.
On this page
The question for today
What you’ll understand
- Explain how a shared workload factor can confound a release comparison.
- Calculate stratified and pooled error rates with explicit denominators.
- State the additional assumptions needed to interpret a comparison causally.
A new model release and rising errors can occur together for several reasons. The release might introduce a failure, heavier requests might reach the new replicas, or a shared infrastructure change might affect both deployment timing and reliability. Observational correlation helps generate explanations but does not choose among them. Define the intervention and outcome you care about, inspect how traffic was assigned, and look for evidence that would distinguish the competing mechanisms rather than choosing the most recent visible event.
Conditioning and intervening answer different questions
An observational comparison asks what happened among requests that received a particular release. A causal intervention asks what would happen if the release choice were changed under a defined system and population. Those groups can differ before the code executes. For example, an autoscaler may send long prompts to newly launched GPU workers, while short prompts remain on warm older workers. Prompt length then affects both release exposure and memory pressure, making a simple error-rate comparison ambiguous.
A causal graph can make assumptions explicit: request complexity may influence release assignment and errors, while release behavior may also influence errors. The graph is a proposed mechanism, not evidence by itself. Randomized assignment in an appropriate controlled experiment can address some confounding, but incidents rarely provide unlimited freedom to experiment. Stratification, matched replay, and mechanism-specific observations can narrow explanations under stated assumptions; they do not automatically eliminate unmeasured differences or interference between shared workers.
Traffic composition can reverse a pooled comparison
Always retain counts and denominators within meaningful cohorts. A release can have a lower error fraction in both easy and hard requests yet a higher pooled error fraction if it receives far more hard requests. That is not an arithmetic contradiction. It means the pooled populations answer a different question. Choose cohorts before searching for a flattering result and inspect whether their definitions or instrumentation changed across releases. Extremely sparse cohorts also produce unstable comparisons.
Reweighting both releases to a common workload mix removes that measured composition difference from the summary. It remains an adjusted association unless the relevant causal assumptions hold, including appropriate control of confounding, comparable outcomes, and a well-defined intervention. Do not adjust blindly for a variable caused by the release if doing so removes part of the mechanism you want to estimate. The causal question determines which variables belong in the adjustment, not a dashboard's ability to group by them.
A useful test predicts a mechanism-specific difference
If the release changes activation memory, compare allocation behavior on the same supported prompt shapes and hardware while recording concurrency and cache state. If traffic assignment is the suspected confounder, verify the assignment path and cohort distribution. A rollback improving errors is useful operational evidence but may also change traffic, caches, or resource pressure. Record those co-interventions. The aim is an explanation that predicts observations and survives attempts to disconfirm it, rather than a story that fits every possible outcome.
The sampling lab illustrates variability only after a sampling model has been chosen. Reset, set both p=0.1 and n=20, predict the exact sample-mean distribution, then inspect and Play or Step. Reset and set both p=0.1,n=100 for a separate comparison. The bars assume independent identically distributed Bernoulli outcomes; playback highlights possible outcomes rather than collecting data. Neither the exact bars nor their normal comparison controls confounding or proves a release effect. Population selection is a separate assumption from sample size.
Work it through
Separate a workload shift from a release comparison
Fictional old-release traffic has 800 easy requests with 8 errors and 200 hard requests with 40 errors. New-release traffic has 200 easy requests with 1 error and 800 hard requests with 120 errors.
Within cohorts, old errors are 8/800 = 1% for easy and 40/200 = 20% for hard; new errors are 1/200 = 0.5% and 120/800 = 15%. The new release has the lower observed fraction in each stratum.
Pool the actual traffic: old has 48/1,000 = 4.8% errors, while new has 121/1,000 = 12.1%. The different easy/hard mixture reverses the pooled comparison.
At a declared equal 50/50 mixture, old's summary is 10.5% and new's is 7.75%. This adjusts the measured mix but still does not prove a causal improvement without the remaining assignment and measurement assumptions.
The pooled rise from 4.8% to 12.1% cannot by itself identify the release as harmful. Cohort composition changes the question being answered.
The living notebook
Explore the model
Interactive lab · sampling
More samples do not repair a biased comparison
For each fresh run, Reset, set both p and n, predict the exact Bernoulli sample-mean bars, then Play or Step. Compare p=0.1,n=20 with separately Reset p=0.1,n=100. The instrument has no treatment assignment or confounders; normal-approximation information is not a causal guarantee.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forSampling variability and causal identification are different problems; a larger confounded sample can remain confounded.
Pause & reason
Check your understanding
Your turn
Try it yourself
Two releases have measured easy/hard error rates of 2%/10% and 1%/8%. Compute each summary at a common 75% easy and 25% hard mix, then state why the lower summary is not automatically a causal effect.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the same weights for both releases. The arithmetic standardizes the observed mixture but does not randomize the assignment.
Compare with the solution
The first summary is 0.75×2% + 0.25×10% = 4%. The second is 0.75×1% + 0.25×8% = 2.75%. The observed standardized difference is 1.25 percentage points. Unmeasured request properties, hardware, concurrent load, measurement changes, or interference can still explain some difference; a causal claim requires explicit assumptions and supporting mechanism evidence.
Keep the idea
Take it with you
- Observed release assignment and a defined intervention need not identify the same outcome.
- Pooled rates can reverse stratified comparisons when workload mixtures differ.
- Adjustment and large samples do not replace causal assumptions and mechanism tests.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.