Long-sequence cost and capacity models

For a conventional uniform KV layout, count layers, concurrent sequences, retained tokens per sequence, KV heads, head dimension, and bytes per stored element. Grouped-query attention has fewer KV heads than query heads, so substituting the query-head count can overestimate this component. Quantized caches can add scales, metadata, alignment, and residual storage. Prefix sharing and model-specific layouts change accounting further. State whether a reported capacity is total device memory or the usable KV pool after weights, workspace, reserve, and other allocations.

Generation can grow the retained history beyond the prompt, so plan output tokens and simultaneous requests as well as prefill size. An allocator’s reserved capacity differs from currently live useful KV bytes. Sharding can reduce per-device storage for some layouts but requires checking actual KV partition or replication behavior; dividing by device count is not universally valid. A lower KV representation size can also introduce quality differences. Validate exact workload behavior and the serving configuration rather than turning a byte calculation into a guarantee that every model fits.

Dense causal prefill compares each query position with its permitted prefix. Counting the QK product and value accumulation with multiply and add as two operations gives a useful ideal attention-work expression. It excludes projections, feed-forward layers, softmax, normalization, and extra kernel work. IO-aware implementations can avoid materializing the entire attention matrix and improve memory traffic, while still computing the relevant pair interactions. Sparse or sliding masks change the pair set and must use their own formula rather than inherit a full-attention estimate.

For a single full-attention decode step, retained KV may need to be read through the memory hierarchy. A stated one-read traffic model gives a movement lower bound, but real kernels can reuse or reload values, and weights or other work can dominate. Effective bandwidth is not necessarily the device’s advertised peak. Time to first token, inter-token latency, and complete request latency describe different boundaries. Include workload arrival and concurrency when measuring them; a single-request lower bound cannot establish a loaded service percentile.

vLLM 0.10.1 documents chunked prefill and batching controls that trade time to first token, inter-token latency, and throughput. Those are versioned scheduler behaviors, not a theorem that one setting is optimal for every GPU and model. A long prefill can contend with existing decodes; cache pressure can trigger recomputation under a supported policy. Record preemption, cache reuse, input and output lengths, and concurrency when explaining a latency result. Also test whether a memory-saving representation changes the evidence-dependent output.

The Roofline lab illustrates only a hypothetical resource ceiling. Reset, set operational intensity to 8 and bandwidth to 80, predict the ceiling, then Step: 640 GFLOP/s. Compare a fresh Reset with both intensity at 2 and bandwidth at 80 before Step: 160 GFLOP/s. Its compute ceiling is fixed at 1000. It contains no KV layout, token sequence, attention kernel, allocator, or scheduler, so it cannot predict a real long-context service’s latency or capacity.