Engineering / advanced
Reduce precision with measured quality bounds
Connect quantizer scales, calibration, outliers, kernel constraints, and evaluation so memory savings do not hide unacceptable output changes.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Reduce precision with measured quality bounds
Connect quantizer scales, calibration, outliers, kernel constraints, and evaluation so memory savings do not hide unacceptable output changes.
- 01
Affine quantizers, zero points and boundary behavior
Specify a quantizer completely enough that export and execution agree on every boundary.
24 min - 02
Choose scale granularity with its metadata cost
Relate logical axes and groups to precision, storage and kernel compatibility.
24 min - 03
Calibrate on the deployment distribution without leaking evaluation
Treat observed ranges and reconstruction objectives as estimates with explicit coverage limits.
24 min - 04
Move activation scale without changing the real linear map
Distinguish exact inverse rescaling from the approximation introduced by quantizing the rescaled tensors.
24 min - 05
Model weight-only error at the layer output
Use input-sensitive reconstruction rather than treating every weight error as equally harmful.
24 min - 06
Quantize activations and persistent attention state deliberately
Separate transient activation ranges from the lifetime, layout and errors of the KV cache.
24 min - 07
Prove accumulator bounds before shrinking operands
Track centered integer products, bias and intermediate widths instead of inferring safety from operand size.
24 min - 08
Match training-time fake quantization to the deployed operator
Distinguish PTQ reconstruction from QAT updates and make surrogate-gradient assumptions explicit.
24 min - 09
Count packed bytes and conversion traffic at a named boundary
Follow the representation consumed by the kernel instead of assuming a small checkpoint guarantees a fast multiply.
24 min - 10
Release quantization only after held-out task regression analysis
Evaluate paired changes, critical slices and uncertainty alongside memory and latency gains.
24 min