Count requests that satisfy the whole latency contract

Use a common client-side monotonic clock for the fixture: request start a, first delivered token f, subsequent token timestamps t2 through tm, and terminal completion e. Then TTFT=f−a and end-to-end latency=e−a. Server-side queue and kernel times are useful diagnostic components but are not interchangeable with those client-visible intervals. Buffering can deliver several tokens together, so specify how token timestamps are recovered and how delivery chunks affect any claimed inter-token metric.

For a request with multiple tokens, average time per output token after the first is (tm−f)/(m−1). A maximum inter-token gap is a different statistic and can expose one long stall hidden by that average. Pinned vLLM 0.10.2 benchmark code uses average TPOT in its documented goodput options; this lesson deliberately defines a stricter maximum-gap rule for its worked fixture. For zero- or one-token outputs, define applicability explicitly rather than dividing by zero or silently inventing intervals.

Select requests whose intended start times lie in a fixed interval [t0,t1), retain every offered identifier, and observe outcomes through a declared drain or timeout policy. A request qualifies only if it succeeds and satisfies all specified latency thresholds. Rejections, errors, cancellations, and missing terminal observations need explicit handling; in this module's goodput fixture they do not qualify. They remain in offered-demand and failure reports, rather than disappearing from the comparison.

Divide qualifying requests from that arrival cohort by the cohort duration for the stated cohort-goodput measure. This differs from counting completions observed in an arbitrary wall-clock window, which can mix earlier arrivals and omit late completions. DistServe motivates optimizing useful request rate under phase-specific latency requirements, but every report must preserve its chosen definitions. Raw output tokens per second can favor long generations and count tokens from requests that fail the latency contract.

Show offered requests per second, admission/rejection rates, success outcomes, output lengths, qualified request goodput, and relevant per-request latency distributions. Separate tenants or length classes when the mix changes. A system can improve raw token throughput while reducing qualified requests, or raise accepted-request percentile compliance by rejecting more work. Neither outcome is automatically a better service under a fixed demand contract. Preserve the cohort and threshold rules when comparing the baseline and changed scheduler.

The probability lab isolates a hypothetical one-second cohort of twenty requests, each independently qualifying with probability p. Reset, set both p=0.8,n=20, predict sixteen expected qualifying requests, then Play or Step. Reset and set both p=0.5,n=20 for ten. The bars are exact binomial probabilities and playback highlights outcomes rather than running a load test. Real request qualifications can be correlated through shared overload, and the lab measures no TTFT, ITL, completion time, or p99.