Attention-sink and recency policies

Softmax distributes weight across the available keys. A token with high attention score can absorb a large part of that mass even if its value does not supply the fact the current task needs. Removing it changes the denominator and the weights on the remaining entries. StreamingLLM presents evidence that some initial positions act as such sinks in the studied models. That finding does not make attention weights a general explanation of semantic importance, nor prove that initial words are the most useful evidence for every query.

A cache policy must preserve the required representation and position handling for the selected method. StreamingLLM discusses rolling-cache positions and rotary-key treatment; merely concatenating arbitrary old and recent KV tensors is not a full implementation of its recipe. Compare the actual implementation with its documented baseline and target architecture. A policy that stabilizes language-model perplexity can still fail a question whose only supporting record lies in the omitted middle. Evaluate the task being claimed, not just continuity of plausible text.

A simple sink-plus-recency policy keeps S initial slots and R recent slots, with overlap counted only once when the stream is short. After a sufficiently long stream, its direct capacity is S plus R regardless of how many tokens have been processed. This permits bounded retained-state storage, but it leaves many historical positions without direct KV access. External source storage and retrieval can offer another access path; they are additional system components, not information guaranteed by the sink policy itself.

The appropriate counts and behavior depend on the checkpoint and workload. A model trained with an explicit dedicated sink can differ from one that happens to use several initial tokens; hybrid attention layers can have other retention rules. Test sustained generation, precise old-fact recall, task reasoning, and source attribution separately. A headline about processing millions of streaming tokens must not be rewritten as a claim that millions of tokens are simultaneously accessible or that discarded middle evidence is preserved exactly.

Compare recency-only, sink-plus-recency, and the legal full-history reference at matched conditions where feasible. Record which positions remain directly accessible and how source facts are selected for tests. Repeated references to a fact in the recent tail can make an old-fact test appear to succeed even when the original evidence was evicted. Use fresh synthetic values and checks that eliminate such shortcuts, while recognizing that a synthetic retrieval task does not cover every real long-document dependency.

The network lab illustrates a two-option logistic normalization only. Reset, set input to 1 and weight to 2.5, predict the fixed-bias logit and output, then Step: logit two gives approximately 0.8808. Compare a fresh Reset with both input at 1 and weight at 0.5 before Step: logit zero gives 0.5. This scalar unit has no cache or attention sinks and does not simulate removing a token from a trained model; it only shows sensitivity to a supplied logit difference.