Quantization and calibration

An affine uniform quantizer rounds a scaled number to an integer code and clips it into a bounded range. A scale converts between code units and real units; a zero point locates real zero. Symmetric quantizers usually use a zero-centered code range, sometimes leaving one signed code unused. State that convention explicitly. Rounding introduces local error, while clipping can introduce much larger error when the represented value falls outside the chosen range.

The granularity of a scale matters. One scale for a whole tensor is cheap but can waste resolution if one channel has outliers. Per-channel or grouped scales use extra metadata while adapting the numerical range more locally. Some methods preserve selected quantities at higher precision. These design choices mean that two checkpoints both described as four-bit can have different memory footprints, approximation properties, and kernel requirements. Count the complete stored representation rather than only packed integer payloads.

Post-training quantization uses already trained weights, often with representative inputs to estimate ranges or error-sensitive transformations. GPTQ develops weight quantization using information about layer inputs. SmoothQuant redistributes scaling between activations and weights to make low-precision computation more tractable. These are different mechanisms, not interchangeable labels for shrinking a file. Neither method's original results establish that any arbitrary model, bit width, or workload will preserve quality.

A calibration set should exercise the deployed languages, sequence lengths, formats, and activation patterns without borrowing final test answers. Freeze the transformation before evaluating on held-out tasks. Quantizer calibration concerns numerical ranges or reconstruction; probability calibration concerns whether predicted confidence matches outcome frequency. A quantized model can change both task quality and confidence behavior, so evaluate them separately. Long-context behavior and unusual inputs can expose errors absent from a small collection of short conversational prompts.

Distinguish weight-only quantization, quantized activations, and quantized key-value caches. A four-bit weight format does not imply that activations or caches use four bits. During execution, kernels may dequantize values, accumulate in a wider type, and hold temporary buffers. A hardware path without efficient support for the format may be slower despite moving fewer weight bytes. Measure latency, throughput, peak memory, and energy at representative batch sizes and lengths.

Use a full-precision or established higher-precision baseline with identical prompts, tokenizer, and decoding configuration. Report absolute quality changes on key slices, not just a relative average or file-size ratio. Include numerical failures, malformed outputs, and calibration drift. A useful deployment decision defines acceptable quality degradation and a measurable resource benefit before experimentation. Retain a compatible fallback and identify the exact quantization artifact, scales, kernels, and runtime versions so a regression can be reproduced.