Model weight-only error at the layer output

For a column activation x and weights W, define the reconstructed quantized weights as W plus E. Ignoring runtime arithmetic error, the output perturbation is exactly E times x. A norm bound follows from Cauchy–Schwarz, but it is often conservative because it ignores orientation. Two weight perturbations with identical Frobenius norms can behave differently on a given input. Measure representative layer outputs as well as weight reconstruction, while remembering that calibration inputs still cover only part of the deployment distribution.

If input coordinates have different variances or are correlated, their empirical second-moment matrix weights the reconstruction objective. This explains why independently rounding every weight to its nearest value need not minimize layer-output error. Cancellation can make several weight errors harmless on one input and harmful on another. A reconstruction method optimized for a narrow input subspace can also perform poorly outside it. Include held-out inputs with different domains and lengths before accepting the compressed artifact.

GPTQ uses approximate second-order information from calibration inputs and compensates unquantized weights as other weights are quantized. That strategy targets a layer reconstruction problem; it is not equivalent to gradient finetuning of the entire language model. AWQ instead uses activation-informed scaling and clipping choices to improve weight-only quantization, without the same Hessian-based compensation procedure. Both depend on concrete quantizer and grouping decisions. Neither algorithm name implies a universal error bound or a guarantee of identical decoded outputs.

For a calibration matrix X containing input columns, the squared Frobenius norm of EX is a useful explicit objective. Its trace form exposes XX transpose, the input second moment, rather than only E’s elementwise magnitudes. Normalizing by the number of columns changes the scale but not the minimizer when every example is equally weighted. Task-level errors can nevertheless amplify across layers and through nonlinearities, so a local reconstruction score is a diagnostic and selection signal, not a replacement for application evaluation.

A runtime can unpack and dequantize weights into registers, use a fused mixed-precision kernel, or materialize a higher-precision temporary before multiplication. These choices have different memory traffic and launch overhead. Weight-only storage savings are most relevant when parameter movement is a bottleneck, but a large batch can reuse weights enough that compute or activations dominate. Benchmark the actual shapes, packing and kernel implementation. The payload compression factor alone does not predict latency, and a fallback path can erase an expected benefit.

The network lab shows a downstream nonlinear response to a changed scalar weight, not a quantization algorithm. Reset and set both input x=1 and weight w=1; predict sigmoid(0.5), approximately 0.62246. Separately Reset and set both x=1 and w=0.5; predict sigmoid(0)=0.5, because the fixed bias is -0.5. Compare the output change with the weight change. The instrument contains one sigmoid unit and cannot estimate a transformer’s reconstruction error or select compressed weights.