KV caches and inference memory

At each attention layer, the new query attends to keys and values associated with the existing prefix. Under ordinary causal decoding, earlier tokens' hidden representations do not change when future tokens are appended, so their projected keys and values remain reusable. The new token still needs its own projections and an attention operation over the retained context. Caching therefore removes repeated prefix computation; it does not make the cost of reading a growing context vanish.

Count key-value heads rather than assuming they equal query heads. Conventional multi-head attention stores a key and value head for every attention head. Multi-query attention shares one key-value head, while grouped-query attention uses an intermediate number. These architectures change cache size and attention structure. They are properties of trained model weights, not an arbitrary serving switch that can always be applied without adaptation or quality evaluation. Inspect the actual model configuration before budgeting.

A naive contiguous allocation that reserves each request's maximum length wastes unused capacity and can fragment memory as requests finish at different times. PagedAttention instead organizes cache storage in blocks with a mapping from logical positions to physical blocks. The analogy to virtual memory explains flexible allocation, not a claim that device attention is implemented by an ordinary operating-system page fault. Block size trades internal waste against bookkeeping and kernel access patterns.

Prefix sharing can save physical storage only where the serving system establishes compatible reusable state. Matching text is insufficient if model revisions, tokenizer behavior, positions, attention masks, or other inputs differ. Sharing also requires an isolation policy so one tenant cannot infer another's private prompt through timing or cache behavior. Count shared physical blocks once when measuring actual allocation, but do not subtract speculative sharing from an admission budget unless the implementation guarantees that reuse.

Total device memory includes weights, cache, temporary attention buffers, activations, communication storage, and runtime allocations. Weight quantization does not automatically reduce cache precision; those are separate choices with separate quality and kernel requirements. Reserve a safety margin rather than assigning every remaining byte to prompts. A request's output grows its cache until termination, and beam search or branching can create several continuations whose storage depends on how prefixes are shared.

Admission and scheduling need both current use and expected growth. A fixed maximum output reservation is conservative but predictable; speculative overcommit needs a deliberate response when requests all grow together. Options include controlled preemption, recomputation, or rejection, each with a latency cost. Measure peak allocated memory and out-of-memory behavior under realistic length distributions. Reporting only an average cache length conceals the long-context tail that can determine whether the service survives a burst.