KV compression saves representation bytes under a separately tested quality contract

A uniform affine quantizer maps a range of real values to integer codes and reconstructs an approximation using a scale and zero point. Finite ranges can clip outliers, and rounding loses detail even when no clipping occurs. Quantization granularity matters: one scale per tensor, channel, token, or group exposes different dynamic-range tradeoffs and metadata costs. KIVI studies asymmetric low-bit KV quantization, reflecting that keys and values need not behave identically. Its method and evaluated results should not be replaced by the unsupported assumption that any two-bit format is interchangeable.

The vLLM 0.9.1 quantized-KV documentation describes FP8 cache formats and its then-current scalar scaling support; those version-specific features are distinct from the illustrative integer format used below. A weight-quantization setting does not automatically choose the KV representation. Count keys and values, scale storage, any retained full-precision residual, padding, block tables, and conversion buffers under the actual implementation. If prefix caches share compressed blocks, their effective format and scale interpretation must be compatible; token equality does not authorize reading one representation through another decoder.

For a specified packed format, add code bytes and group metadata rather than dividing the original footprint by the nominal bit ratio alone. Small groups can improve local range fit while making metadata significant. Padding and temporary dequantization can further reduce peak-memory savings. The toy calculation assumes tightly packed codes and a fixed metadata record per group; real kernels may impose extra alignment or keep selected values at higher precision. Report both persistent-cache bytes and transient peak bytes so a compact stored representation does not hide a large conversion allocation.

Capacity improvement is also different from latency improvement. A compressed cache can permit a larger batch yet require extra conversion, metadata reads, or kernels. Conversely, fused quantized attention may reduce traffic when supported. The cited vLLM documentation describes its particular release's performance limitations, so avoid extrapolating them into a claim about every future or alternative implementation. Benchmark the exact engine, hardware, context lengths, and concurrency with the same output and quality criteria, including host-transfer effects if compressed state is offloaded.

Compare reconstructed tensors and attention outputs against a fixed higher-precision reference, then evaluate actual tasks across long contexts, rare tokens, retrieval positions, languages, and generation lengths. A small elementwise error can interact with attention scores and normalization, and an early changed output token alters later inputs. Teacher-forced comparisons isolate numerical effects on a fixed token history; free-running generation measures the resulting closed loop. Neither exact token agreement on a few prompts nor one aggregate benchmark proves quality preservation across every workload. State the allowed degradation and sampling uncertainty.

The network lab illustrates only a nonlinear response to an altered scalar. Reset, explicitly set BOTH x=1 and w=1, predict sigmoid(0.5)≈0.6225, then Play or Step. Reset and re-set BOTH x=1 and w=0.95; predict sigmoid(0.45)≈0.6106. Edits retain t and the bias remains −0.5. This fixed unit does not quantize KV, run attention, or estimate language-model quality. Its reachable parameter change shows that a small representation perturbation can change a nonlinear output; the magnitude and acceptability of real cache errors require the actual reference tests.