Context-window retention and eviction

An ordinary full-attention decoder uses cached keys and values to avoid recomputing the prefix for each next token. If a serving cache releases reusable inactive blocks but retains the original input and can recompute under the same model and state contract, that is a storage and latency policy. vLLM’s prefix-cache design describes such block reuse and eviction. It does not imply that an active request may arbitrarily discard required history while preserving identical full-attention outputs. Name the cache object and its ownership when discussing an eviction.

A sliding-window attention policy instead limits which prior token states are directly accessible. Transformers 4.53.2 documents a sliding-window cache for models that support that architecture; it is not a universal plug-in replacement for every full-attention model. Removing old direct KV entries does not erase every influence those tokens had on later states, because retained states may already encode some of their effects. But those indirect representations are not a guaranteed lossless archive and do not establish exact retrieval of an evicted record.

Retaining raw text outside the device cache can permit later retrieval or recomputation, while a summary or compressed state offers a different information contract. A quantized cache may preserve the same logical token set with numerical approximation; an evicted-window policy removes direct access to some tokens. These are distinct changes and need separate baselines. Check whether an apparent memory reduction comes from fewer retained positions, fewer bits per element, shared prefixes, or simply moving state to another storage tier.

Position handling also matters when a cache rolls or its blocks move. Physical slot indices are not necessarily logical token positions, and reusing a storage slot must not silently reuse the wrong positional encoding. Compare legal cached execution with an appropriate reference for the intended attention policy. When the policy itself changes, a difference from full attention is expected and should be evaluated on task quality, not dismissed as harmless because generation stays fluent. Record the visible history at each evidence-dependent test.

Place a fresh fact before, near, and after the eviction boundary, then ask questions that require its precise value and provenance. Include a condition that retrieves the original text again and another that does not. This distinguishes an external-memory path from apparent parametric knowledge or vague continuation. Monitor cache residency, recomputation work, and task outputs together. A stable token-generation rate after hours of streaming measures operational continuity; it does not demonstrate access to every historical token.

The cache lab illustrates only reusable-entry capacity. Reset, set capacity to 4 and stride to 4, predict hits at seventeen requests, then Step once: thirteen hits. Compare a fresh Reset with both capacity at 2 and stride at 4 before Step: zero hits. The toy starts empty with sixteen requests already processed and is capped at 4,096. It has no KV tensors, logical token positions, language model, or reconstruction path, so use it only to reason about a declared working set.