Prefill queues: service-time variation matters before saturation

Choose whether a modeled server is one device, one tensor-parallel instance, or a scheduler-owned execution lane. Its service time must match that definition and the workload being admitted. A prompt’s token length is a useful feature but not a universal linear cost because attention shape, cache reuse, batching, and kernel saturation matter. Record service-time distributions by request class and configuration. Arrival rate and service rate expressed in different units, such as requests per second versus prompt tokens per second, cannot be compared without the corresponding workload conversion.

For the illustrative M/G/1 model, utilization is arrival rate times mean service time and must be below one for the stated stationary regime. This necessary mean margin does not imply that a particular tail deadline will be met. Service-time variance changes residual work seen by arriving requests, and the second moment appears in mean queue waiting. An occasional very long prompt can therefore matter even when the mean service time is unchanged. A bursty or correlated arrival process needs another model or trace-based analysis.

A prompt can be short enough for local admission but lack a reserved compatible decode destination. Include outstanding cache bytes, destination reservations, transfer pressure, and request deadline when defining the accepted work path. Rejecting or deferring before expensive prefill can avoid stranded state, but the policy should record the affected workload classes. A finite queue or token budget is a resource control, not evidence that every accepted request completes within its first-token objective. Define cancellation and expiry so reserved capacity is released without letting stale attempts reuse it.

Chunking a long prompt can reduce one blocking interval, but it adds scheduler decisions and can change repeated cache access, transfer scheduling, and the service-time model. Priority scheduling also changes which requests wait and which fairness claim applies. Once a pool uses batching, preemption, or multiple heterogeneous servers, the simple single-server FCFS formula is no longer a literal predictor. Preserve it as a diagnostic baseline and compare with measured arrival and completion traces. Do not infer p99 delay from a mean waiting value without a justified distributional calculation.

Sweep offered load with a generator that preserves the intended arrival process, and report accepted, rejected, canceled, timed-out, and completed requests. Track prompt length, reuse, per-class queue time, cache bytes retained, and destination reservation age. A completed-only chart can improve when overloaded requests disappear from the sample. Test a decoder slowdown as well as a prefill slowdown, because downstream backpressure can retain source caches and invalidate a local capacity estimate. Verify recovery and reservation cleanup after the load returns to normal.

The queue lab is a different, smaller model. Reset, explicitly set arrival two and service three, predict its displayed M/M/1 mean residence of one second, then Play or Step. Compare a fresh Reset with arrival 2.5 and service three explicitly set, giving two seconds. Those residence values assume Poisson arrivals and exponential service; the animation itself is deterministic fluid backlog from four items. Neither is the M/G/1 variance experiment above, and the instrument has no prompt lengths, batching, destination reservation, or finite-capacity admission.