Inference serving and batching

Prefill processes the prompt and builds attention keys and values for its positions. Many prompt positions can be processed together. Autoregressive decoding then generates successive tokens, each depending on previous outputs. Reusing the key–value cache avoids recomputing prior keys and values, but the cache grows with retained context. Prefill often offers more parallel arithmetic, while decoding can be limited by memory traffic at modest batch sizes. The actual bottleneck depends on model architecture, kernels, and hardware.

Measure time to first token from the user's request, including queueing and prefill, and inter-token latency during generation. Also measure successful completed requests and useful output tokens per second. A high throughput result can coexist with frustrating pauses. Cancellation should release cache blocks and stop work promptly. Streaming a first token quickly while the remaining response stalls is not equivalent to meeting an end-to-end service objective; record both phases and the distribution of request lengths.

For a conventional decoder cache, each retained token stores one key and one value for every attention layer and key–value head. Multiply layer count, KV-head count, head dimension, element bytes, and retained tokens, then include both keys and values. Grouped-query attention can have fewer KV heads than query heads; using the query-head count in that case overestimates the cache. Tensor-parallel placement, quantization metadata, and implementation details can change per-device accounting.

Weights, cache, activations, temporary workspaces, and allocator fragmentation share device memory. Admission control therefore needs a token budget, not just a request-count cap. Reserve room for possible generation or explicitly support preemption and recomputation. PagedAttention demonstrates block-based cache allocation that reduces allocation waste and supports sharing. It does not make context memory free, and a serving system still needs rules for which requests to admit, suspend, or reject.

Static batching waits for a group and often lets the longest request determine the group's completion. Continuous batching can admit and retire requests between decoding iterations. Larger active batches amortize weight reads, but also increase cache occupancy and can lengthen token latency. Long prefills can delay active decodes unless the scheduler interleaves or chunks work. A maximum batch size is therefore only one part of scheduling; token budgets and phase-aware fairness matter.

Benchmark a workload distribution, including long prompts, long generations, cache misses, cancellations, and bursts. Set a queue deadline and reject work that cannot meet its deadline instead of building an unbounded queue. Consider fairness between users and between interactive and offline traffic. Repeatedly inspect quality if precision or cache policies change. Serving optimization must preserve the intended model behavior and account for completed useful work; repeated retries and abandoned responses can inflate apparent token throughput.