Integer inference and device kernels

Affine quantization interprets an integer q as a real approximation s times q minus z, with positive scale s and integer zero point z. The zero point makes real zero exactly representable, which matters for padding and sparse values. TensorFlow’s referenced int8 convention permits asymmetric activation zero points, while its convolution weights use symmetric zero point zero and a restricted signed range. Per-output-channel weight scales are another contract dimension; changing that axis without changing the kernel’s interpretation changes the arithmetic.

Quantization calibration data should represent the actual sensor pipeline, rare values, and device preprocessing. Clipping and rounding introduce different errors: clipping saturates an out-of-range value, while rounding chooses a nearby representable value inside the range. Test them separately and inspect task-level failures rather than reporting only average tensor error. A small numerical perturbation can cross a downstream decision threshold, while a larger perturbation in an irrelevant tensor direction may have little task impact.

For symmetric weights, subtract the activation zero point before forming the integer dot product. The accumulator represents real values in units of input scale times weight scale. An integer bias must use that same scale; adding an int8 output-scale bias directly to an int32 accumulator is dimensionally wrong. After accumulation, multiply by the ratio of accumulator scale to output scale, round under the kernel’s rule, add the output zero point, and clamp to the output integer range.

Accumulator width is not implied by operand width. A conservative magnitude bound sums the maximum corrected product magnitudes and bias, then checks the required signed range. Intermediate implementation details, vector instructions, and requantization shifts need the library’s own constraints too. CMSIS-NN 7.0.0 selects pure-C, DSP, or MVE implementations based on target feature flags; its support matrix is not a promise that every int8 model or processor uses the fastest implementation. Verify the actual linked kernel and required scratch allocation.

Build reference cases for zero input, padding, extrema, negative values, bias, and saturation, then compare against the selected device kernel. Check shapes and layouts that change optimized-kernel eligibility. Record compiler architecture flags, library version, quantization metadata, and conversion settings with the artifact. A task-quality evaluation must cover the final pipeline, including any float fallback or requantization boundary. A size reduction without the intended integer kernel may still help storage but need not reduce latency or energy.

The sigmoid lab demonstrates downstream sensitivity to a weight perturbation, without implementing quantization. Reset, set x to 1 and w to 0.8, predict logit 0.3 and output approximately 0.5744, then Step. Compare a fresh Reset with both x at 1 and w at 0.75 before Step: logit 0.25 and output approximately 0.5622. The bias stays −0.5. These floating-point calculations contain no integer accumulator, zero point, clipping rule, or architecture-specific kernel.