Quantize activations and persistent attention state deliberately

Static activation quantization uses parameters determined ahead of inference, commonly from calibration. Dynamic schemes compute some parameters from a runtime token, tensor or group and therefore pay a range-estimation and conversion cost. Both need compatible consumer kernels and an explicit outlier policy. A tensor’s distribution can differ between prefill and decode, so one successful short prompt does not validate all phases. Accumulation, softmax and normalization may remain at higher precision even when matrix operands use fewer bits.

The KV cache stores previously computed keys and values so later tokens can attend to history without recomputing every projection. It grows with stored tokens and uses the number of KV heads; grouped-query attention can have more query heads than KV heads. Counting query heads instead would overstate this payload for such a model. Real serving allocators also reserve pages, track sequence metadata and handle sharing or fragmentation, so a raw tensor formula is a lower-level accounting boundary rather than a complete capacity forecast.

A scale chosen for an existing cache block must remain associated with those stored codes. Replacing the scale after a new outlier arrives changes the meaning of old entries unless the block is requantized or represented with a compatible scheme. Per-token, per-channel and block-grouped formats impose different metadata and layout costs. KIVI’s studied design quantizes keys per channel and values per token, with grouped blocks and a higher-precision recent residual. Those empirical choices should not become a claim that every model has the same preferred grouping or a universally safe range.

Quantized keys perturb attention scores, while quantized values perturb the weighted sum. Score changes perturb softmax probabilities; even modest resulting output changes can alter a later token choice when its competing logits are close, after which generation can diverge. The complete real-product expansion includes both query and key perturbations if both are quantized. Even if a runtime uses a higher-precision softmax, its inputs have already changed. Evaluate long-context retrieval and domain-specific generation separately rather than accepting a cache format solely because it has low elementwise error.

Report packed KV bytes together with scale metadata, allocator overhead, live temporaries and any dequantized workspace. Measure peak resident memory at declared batch and context lengths, then measure latency and output quality under those same conditions. A fourfold code-payload reduction can deliver much less total-memory reduction when weights or metadata dominate. Include prefill and decode timings, because range estimation and cache conversion can change their costs differently. A cache release also needs a format identifier so sessions cannot read old codes with new interpretation rules.

To isolate a memory-bound ceiling argument, Reset the roofline lab and set both I=4 FLOP/byte and B=80 decimal GB/s. Predict 320 GFLOP/s. In a separate run Reset and set both I=8 and B=80; predict 640 GFLOP/s. The hypothetical byte boundary must include every activation, KV and metadata transfer being compared. Play traces the unchanged roof. This synthetic instrument demonstrates a ceiling from changed traffic, not an attention kernel, a cache allocator or a promised KV speedup.