Replay offered arrivals without hiding overload

Retain interarrival times, prompt and output-length relationships, tenant identity, burst structure, deadlines, and cancellation or stop policies when those are relevant to the experiment. Shuffling lengths independently of arrivals can remove a difficult burst of long prompts while leaving each marginal histogram unchanged. If output generation is stochastic, distinguish a fixed-work replay from a realistic decode-policy experiment; forcing output lengths can change semantics but may be useful for a declared scheduler fixture. Record which kind of comparison is being made.

Also state initial conditions: loaded model and tokenizer, serving version, cache residency, prefix-hit policy, allocator state, warm-up, and number of outstanding clients. An arrival-rate sweep is useful, but a Poisson process is one workload choice rather than a universal description of users. Repeat with controlled seeds or preserved traces and inspect run-to-run variation. A single p99 from a small cohort is not a population guarantee, and a model's queueing mean is not a measured tail percentile.

A closed-loop client is valid when the intended question concerns a fixed number of users waiting for responses before making new requests. It is not equivalent to independent offered arrivals at a fixed rate. When responses slow, completion-gated clients submit fewer new requests, coordinating away some work that an open-loop plan would have offered. The wrk2 author's primary documentation explains this measurement problem and its planned-start accounting approach. A benchmark must choose its workload rather than silently accept the load generator's accidental behavior.

Record planned arrival, actual send, first token, and terminal time separately. Planned-start latency includes load-generator dispatch delay; actual-send latency excludes that delay. Both can be useful, but do not label the former purely server execution time. If the generator falls behind, increase its capacity or report missed dispatches and delay instead of pretending it maintained the target rate. A correction formula cannot reconstruct every interaction that would have occurred under a different actual arrival process.

Define the offered cohort, warm-up exclusion, timeout, drain period, outcome categories, and percentile estimator before the comparison. Observe late completions under the declared drain rule and report remaining incomplete or timed-out requests. Percentiles of successful completions alone can improve when the slowest requests disappear into errors. Show offered demand, actual dispatch, rejection, goodput, lengths, and resource occupancy alongside latency so a scheduler change can be judged against the same question.

The cache lab supplies a completely deterministic trace for checking replay discipline. Reset, set both C=4,s=4, predict twelve hits at sixteen initial requests, then take two Steps to reach eighteen requests and fourteen hits. Reset and set both C=4,s=4 again to reproduce the same initial and extended counts. Playback caps at 4,096; edits retain elapsed time. This model has no arrival timestamps, client capacity, inference engine, random outputs, or percentile measurements. It illustrates controlled trace and initial-state repetition without claiming to solve a real load-generation problem.