Choose scale granularity with its metadata cost

Per-tensor quantization assigns one scale and zero point to the whole tensor. Per-channel is more precisely per-axis: one pair applies to each slice along a named logical dimension. For a weight matrix stored as output channels by input channels, per-output-channel scales index the first dimension. Transposing the matrix changes the physical arrangement and the axis number unless the representation carries the corresponding mapping. Never infer scale semantics from whichever dimension happens to be contiguous in one packed kernel layout.

Consider two rows [0.1,-0.1] and [10,-10]. With symmetric codes -127 through 127, a shared maximum magnitude of 10 gives scale 10/127. The small row then has roughly one code step available near its values. Independent row scales give the small row a hundred times finer spacing. This is a resolution argument, not proof of better generated answers: downstream sensitivity, correlations and clipping policy still determine how the altered errors affect a model.

Grouped quantization partitions a row or another declared axis into blocks, often so nearby weights share a scale. State the group size, direction, treatment of an incomplete final group and whether zero points are present. Smaller groups can isolate large magnitudes, but they need more metadata loads and can constrain vectorization. A group containing only a few elements may spend as many bytes on scales as on codes. Padding groups to a hardware tile can further enlarge the actual stored representation.

For a clean accounting exercise, assume N weights partition exactly into groups of G, b packed bits per code and h metadata bytes per group. Payload and metadata add; neither replaces the other. A symmetric format can omit zero points, while an affine format may need them. If scales are stored with reduced precision, their reconstruction error becomes part of the numerical model too. A theoretical byte count must be labeled as such until alignment, headers and packing have been specified.

A checkpoint with per-row scales cannot be relabeled as per-group merely because a runtime expects the latter. Conversion may require unpacking, reconstructing and requantizing, introducing another approximation. Test a small matrix whose rows have deliberately different scales and compare decoded logical elements before running a large multiply. Include partial groups in this test suite. This catches axis swaps that can leave every array shape legal while multiplying a row by the scale intended for a different channel.

The matrix lab offers a limited geometric analogy for changing one axis independently. Reset and set both diagonal a=0.5 and shear b=0. Predict determinant 0.5 and a unit square compressed horizontally while the vertical coordinate remains unchanged; Play interpolates the geometry, while readouts describe the final selected matrix. This instrument has only two parameters and no quantization groups. Use it to articulate which coordinate is scaled, then return to the explicit logical group map in the tensor contract.