Reserve memory for the work an admitted request may still create

Let a request have P prompt tokens and an enforced maximum of O generated tokens. Under a chosen conservative policy, reserve cache capacity for P+O retained tokens, rounded to the allocator's block size. This can intentionally overreserve compared with an early EOS. If the output length is only predicted rather than capped, the reservation is an estimate and needs an overflow policy; it cannot guarantee that every admitted request fits. Count speculative slots, branches, or other extra contexts when the engine can create them.

PagedAttention explains how fixed-size KV blocks and sharing can reduce waste, but paging does not remove the need to account for real resident blocks. Shared prefixes require reference-aware accounting, and a per-request upper reservation may intentionally ignore sharing to remain conservative. Use one policy consistently rather than counting a shared block once in one dashboard and once per request in another. The reserve must correspond to the actual pool and architecture, including precision, attention variant, and supported context behavior.

In vLLM 0.10.2, max_num_batched_tokens limits tokens processed in one iteration, while other controls and the KV allocator constrain sequences and memory. A decode iteration can process one new token for each of many requests while retaining thousands of earlier tokens per request. A small iteration budget therefore does not establish a small total resident cache. The admission decision should inspect both work and storage budgets, plus any separately enforced tenant or deadline policy.

Include fixed weights, bounded workspaces, runtime buffers, and a justified reserve outside the request KV pool. A conservative inequality is useful only if the categories cover the actual allocations and simultaneous peaks. Host output buffers and network queues may live in a different memory domain and need separate limits. Do not label a hand-estimated average as a hard cap. Validate capacity with maximum supported shapes and relevant in-flight overlap, and retain a safe behavior when an allocation still fails.

An implementation can reserve worst-case caps, grow allocations dynamically, preempt and recompute, or combine strategies. Those choices trade utilization, latency, and complexity. If a request increases its cap after admission, repeat the relevant checks before accepting that growth. Cancellation and completion should release reservations according to the ownership protocol, while prefix blocks still referenced by other requests remain allocated. Track reserved, physically allocated, and reclaimable bytes separately so a quota is not confused with free physical memory.

The cache lab illustrates a finite working set fitting or failing to fit a capacity, not safe KV eviction. Reset, set both C=4,s=4, predict twelve initial hits, then Play or Step. Reset and set both C=3,s=4 for zero initial hits on the same repeating identifiers. There are sixteen initial requests and a cap of 4,096. Its LRU evictions are automatic and cheap in the toy world; an inference engine cannot discard needed context without its own supported recomputation or other correctness-preserving policy.