Triage a throughput regression with matched work, uncertainty, and a falsifiable hypothesis
Reconcile latency and throughput, design repeated comparisons, and locate a regression without confusing a changed workload or noisy profile with a causal diagnosis.
On this page
The question for today
What you’ll understand
- Compute throughput change from fixed-work completed timing.
- Carry stated timing bounds into a conservative effect interval.
- Build a causal triage plan using controlled replays and corroborating profiles.
A slower dashboard number can reflect more work, a new measurement boundary, transient load, or a real execution regression. Start by reconstructing the exact comparison before tuning anything. Preserve the model, data shape distribution, valid-update semantics, hardware placement, and timing method. Then quantify the effect across appropriate repeated runs and inspect where the required dependency path changed. A profile is strongest when it tests a specific explanation for an established regression rather than supplying an attractive story for one noisy measurement.
Normalize latency and throughput to the same work
For fixed valid work W, throughput is W divided by completed time. A ten-percent increase in time produces a throughput decrease of one minus the reciprocal of 1.1, not exactly ten percent. If token counts or shapes change, report both work and time rather than treating throughput movement as a timing-only effect. Check padding conventions, accumulation, skipped updates, checkpoint intervals, and data-loader behavior. A changed step name or log cadence can move the measurement boundary without changing any device operation.
Record code and configuration versions, compiler and backend settings, hardware topology, device clocks, batch shapes, and the warmup policy. Preserve numerical-quality checks so an apparently faster configuration is not rewarded for invalid or missing work. Classify the observation window: steady training, startup, checkpointing, or an intentionally mixed operational interval. Each is useful, but comparing one with another cannot isolate a kernel regression. Validate timing synchronization at the chosen boundary before using operator-level traces to explain a supposed end-to-end effect.
Estimate an effect at the right experimental level
Repeated steps inside one warmed process can be correlated and share the same placement, compiler choices, or cache state. Thousands of such steps are not automatically thousands of independent replications of a software change. Plan repetitions at the levels where important variation occurs, such as independent runs or job placements, and keep warmup separate. Interleave or randomize baseline and candidate comparisons when appropriate to reduce confounding by time-varying load. Report an effect estimate and an uncertainty procedure whose assumptions match that design.
Distinguish deterministic measurement bounds from statistical confidence intervals. If you have guaranteed positive duration intervals, monotonicity gives a conservative ratio interval by combining the smallest numerator with the largest denominator and vice versa. A confidence interval instead has coverage properties under a repeated-sampling procedure; it does not guarantee that the unknown quantity lies inside this particular observed interval. Do not invent a confidence percentage for an arbitrary min-max range. For correlated or hierarchical measurements, use an analysis that preserves those dependence levels rather than a naive per-step standard error.
Use a change hypothesis to choose the next measurement
After confirming an effect larger than the relevant uncertainty, compare aligned dependency paths: input readiness, host launch, compute, communication, memory pressure, and synchronization. Select a minimal replay that distinguishes the leading causes, then change one factor or bisect a relevant code range. If a kernel slows only at a new shape, hold that shape fixed. If the exposed tail moves while kernels remain stable, test readiness or overlap. Recheck the final candidate in ordinary execution with quality and work equivalence before concluding that a profile-guided fix resolved the operational regression.
The sampling lab illustrates only a declared independent proportion, not timing uncertainty or a confidence interval for speedup. Reset, set both p=0.5 and n=25, predict sample-mean variance 0.01, then Play or Step. Reset and set both p=0.5 and n=100 to predict 0.0025. The bars are exact binomial masses for independent equal-probability trials; playback highlights outcomes and draws no random samples. Its normal comparison is approximate, and the Berry–Esseen number bounds standardized CDF error rather than barwise or timing-ratio error. More steps in an actual correlated training run need not reduce uncertainty according to this independent model.
Work it through
A ten-percent time increase is a different throughput percentage
Both configurations complete W=10000 valid tokens. Baseline time is nominally 100 ms and candidate time 110 ms. Separately assume valid duration bounds of [99,101] ms and [109,111] ms for these comparable quantities; these bounds are not asserted to be statistical confidence intervals.
Nominal throughputs are 10000/0.100=100000 tokens/s and 10000/0.110≈90909.09 tokens/s. Time increased by 10%, while throughput decreased by 1−100/110≈9.09%.
The candidate-to-baseline throughput ratio equals baseline-to-candidate time. The assumed bounds give [99/111,101/109]≈[0.89189189,0.92660550]. The entire interval is below one under these bound assumptions.
This supports a lower-throughput effect for the stated comparable work and bounds, but does not name its cause. Next compare the required paths and run a discriminating replay; also verify the bounds and repeat the experiment at the relevant run or placement level.
Effect size, uncertainty, and cause are separate claims. A valid triage result preserves all three and tests the final change against ordinary completed work.
The living notebook
Explore the model
Interactive lab · sampling
An independent proportion is not a training-time uncertainty model
Reset, set both p=0.5 and n=25, predict mean proportion 0.5 and variance 0.01, then Play or Step. Reset and re-enter both p=0.5 and n=100 for variance 0.0025. Exact binomial bars assume independent trials; playback highlights outcomes rather than sampling. n caps at 100. The normal overlay is approximate and the Berry–Esseen value bounds standardized CDF error, not timing ratios or each probability bar.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forChoose repetition and uncertainty methods that match the actual dependence and experimental unit.
Pause & reason
Check your understanding
Your turn
Try it yourself
For the same valid work, baseline duration is bounded by [48,52] ms and candidate by [49,53] ms. Compute a conservative candidate-to-baseline throughput ratio interval. Does it establish a direction under only these bounds?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the old-time over new-time ratio with opposite endpoints and inspect whether the interval crosses one.
Compare with the solution
The ratio lies in [48/53,52/49]≈[0.90566038,1.06122449]. It includes one and both directions, so these bounds alone do not establish which configuration has higher throughput. Obtain tighter comparable measurements or an appropriate statistical analysis, and do not convert this deterministic bound range into an unsupported confidence claim.
Keep the idea
Take it with you
- Compare the same valid work and remember that throughput is inverse completed time.
- Use repetitions and uncertainty methods matched to runs, placements, and temporal dependence.
- Establish the effect, test a causal hypothesis, and verify the final fix in ordinary execution.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.