greatitself.
Continue
research29 min studyLesson 749 in Engineering

Compare architectures with matched workloads and useful outcomes

Design controlled serving experiments that preserve resource budgets, request distributions, output semantics, and failure accounting.

Before you begin
On this page

The question for today

What you’ll understand

  • Define a request-level goodput criterion using first-token and streaming constraints.
  • Control resources, cache state, and arrival process across alternatives.
  • Separate measured evidence from paper-specific or framework-wide performance claims.

A separated-stage experiment can win by using more devices, a different output-length distribution, or a warmer prefix cache rather than by improving stage interference. Conversely, a poorly placed transfer can make a useful design look inherently inefficient. Compare complete systems under an explicitly matched workload and resource budget. DistServe and Splitwise report results for particular configurations; vLLM v0.10.2 labels its disaggregated-prefill feature experimental and does not offer a universal throughput-improvement promise. Treat each as a system to test within its own implementation scope.

Freeze the experimental object and useful-work distribution

Record model and adapter revisions, tokenizer, cache format, quantization, kernels, parallelism, hardware allocation, topology, and software versions. Match the total resource budget, including model replicas, transfer staging, and the prefill/decode split. If heterogeneous resources are compared, state their actual resource or cost accounting without substituting an unverified current price. Hold prompt lengths, output limits, sampling settings, stop conditions, and task quality criteria fixed, and preserve the distribution rather than only its average. A changed generated length can alter both service rate and cache residence.

Choose cold-cache, warmed-cache, or trace-driven reuse conditions deliberately. A global flush can test one scenario while removing reuse that production depends on; an accidentally warm cache can produce an equally misleading win. Preserve tenant authorization and cache identity during reuse. Repeat baseline and candidate runs in a paired or randomized order to reduce drift, with warmup and steady measurement intervals named. A microbenchmark of transfer bandwidth or prefill tokens/s is supporting evidence, not a replacement for client-observed complete requests.

G=#{q:successq, TTFTq≤τf, max⁡jITLq,j≤τi}TG=\frac{\#\{q:\mathrm{success}_q,\ \mathrm{TTFT}_q\le\tau_f,\ \max_j\mathrm{ITL}_{q,j}\le\tau_i\}}{T}
This illustrative request goodput uses a positive observation duration T in seconds and a fully accounted request cohort with at least two output tokens each. Thresholds are seconds. It requires every observed gap to meet the stated rule; an average-TPOT SLO is a different criterion and must not be substituted silently.↔ Scroll wide equations horizontally.

Preserve offered arrivals and missing outcomes

Use a load generator capable of sustaining the intended offered process and record scheduling delay when it cannot. A closed-loop client that waits during a stall can reduce offered pressure and hide overload. Keep offered, admitted, completed, failed, canceled, and timed-out counts; state the follow-up rule for the cohort so unfinished requests remain visible. Percentiles of successful completions alone can improve when the system drops its hardest requests. Report goodput, rejection behavior, and resource use together, along with stage traces that explain where delays moved.

Separate first-token effects, average streaming rate, and tail gaps. A transfer moved after the first visible token can improve TTFT while worsening the second-token gap, and a mean TPOT can hide that event. Compare quality and output correctness as well as latency if cache conversion or a different numerical path is used. Require the same request and generation semantics across both systems so retries do not become extra successful requests or duplicated output. Preserve the exact boundary definitions in every chart and aggregate.

ΔG=Gsplit−GcolocatedGcolocated(Gcolocated>0)\Delta_G=\frac{G_{\rm split}-G_{\rm colocated}}{G_{\rm colocated}}\qquad(G_{\rm colocated}>0)
The relative goodput change is dimensionless and meaningful only for matching cohort, constraints, resources, and workload definitions. It does not imply a universal throughput gain or explain which mechanism caused the observed difference.↔ Scroll wide equations horizontally.

Use traces and uncertainty to test a concrete hypothesis

Form a mechanism-level hypothesis such as reduced prefill blocking versus increased transfer delay, then inspect stage queues, copy completion, memory peaks, and client gaps. Vary one placement, pool ratio, or scheduling parameter while holding the intended comparison fixed. Include bursty demand, long prompts, long outputs, cancellation, and relevant failures, not just a favorable steady mix. Report run-to-run uncertainty and per-class regressions. A single best run or one paper’s maximum speedup cannot establish a reliable improvement for another deployment.

The sampling lab illustrates uncertainty only for a hypothetical independent binary pass metric. Reset, explicitly set p=0.8 and sample size twenty, predict sample-mean variance, then Play or Step. Compare a fresh Reset with p=0.8 and sample size eighty explicitly set; variances are 0.008 and 0.002. Playback highlights exact possible outcomes and the normal curve is approximate. Shared serving load can correlate requests, so the toy is not an automatic confidence interval or architecture benchmark for real goodput.

Work it through

Higher constrained goodput can coexist with fewer completions

Two matched hypothetical systems receive 1,000 requests during a 100-second reporting interval, with every request outcome resolved by its end and the same resource budget. Split execution completes 950 and has 840 successful requests meeting both stated latency constraints. Colocation completes 970 and has 800 meeting both.

  1. Offered rate is ten requests/s for both. Completion rates are 9.5/s and 9.7/s, so the split system does not win by this completion-rate metric.

  2. Constrained goodput is 840/100=8.4/s versus 800/100=8/s. The relative change is (8.4−8)/8=0.05, or five percent, under the exact stated criteria.

  3. The offered-cohort fractions meeting both constraints are 84 percent and eighty percent. Keep the fifty versus thirty noncompleted outcomes and the remaining late completions visible; they do not disappear because one aggregate favors splitting.

The result

The split system has five percent higher stated goodput but fewer completions. Both observations are needed to assess this controlled fixture.

The living notebook

Explore the model

Interactive lab

Interactive lab · sampling

An independent pass-count model is not a serving benchmark

Use independent Reset → explicitly set BOTH p and n → predict → Play/Step runs for (.8,20) and (.8,80). Exact sample-mean variances are .008 and .002; playback highlights outcomes, not measured requests. The normal comparison is approximate and real request outcomes can be dependent.

An independent pass-count model is not a serving benchmarkExact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Current readings: Mean sample proportion: 0.5; Sample-mean variance: 0.0125; Uniform CDF error bound: ≤ 0.1062. All parameters are adjustable with the labeled controls below.Exact distribution of the mean of 20 independent Bernoulli variables with success probability 0.5; playback highlights outcome 0.00.250.50.75100.0580.1150.1730.23Sample mean · k / nProbability massNormal bin masses are an approximationk = 0 · exact probability = 9.54e-7
Exact binomial massNormal interval massHighlighted outcome / mean
Mean sample proportion
0.5
Sample-mean variance
0.0125
Uniform CDF error bound
≤ 0.1062
t = 0 s

Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forPreserve workload and outcome definitions, then quantify uncertainty using the actual experiment’s dependence structure.

Pause & reason

Check your understanding

A split-stage benchmark has lower p99 among successful completions but rejects many more long prompts than the colocated baseline. What should the report do?

Your turn

Try it yourself

Matched systems run for fifty seconds. Colocation has 300 requests meeting the stated constraints and splitting has 330. Compute goodputs and relative change. What remains unproved if splitting used twice as many devices?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Calculate the defined metric, then check whether the experimental resource comparison still answers the intended architecture question.

Compare with the solution

Goodputs are six and 6.6 requests/s, a ten-percent relative increase. If splitting used twice as many devices, that result is not evidence of a matched-resource architectural gain. Report the actual resource budgets and compare a controlled alternative or an explicitly defined resource-normalized objective. Preserve model, request mix, quality, cache conditions, offered load, failures, and client-visible latency definitions before generalizing the point estimate beyond this fixture.

Keep the idea

Take it with you

  • Match complete resource and useful-work conditions before comparing architectures.
  • Count constrained successful requests against the offered cohort and preserve failures.
  • Use client-visible traces and repeated controlled runs to assess the proposed mechanism.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.