Count packed bytes and conversion traffic at a named boundary
Follow the representation consumed by the kernel instead of assuming a small checkpoint guarantees a fast multiply.
On this page
The question for today
What you’ll understand
- Compute packed payload, scale metadata and materialized conversion traffic.
- Distinguish logical quantization groups from physical kernel tiles.
- Use a roofline ceiling only with a consistent byte and operation boundary.
A compressed checkpoint can be small while its execution path moves more bytes than the original. The difference often lies in packing and dequantization: whether the kernel reads low-bit codes directly, reconstructs them in registers or writes a full higher-precision temporary to memory. Establish the actual representation and the memory boundary being measured before translating a nominal bit width into a bandwidth or latency claim.
Logical weights need a physical format
A four-bit code occupies half a byte only when two codes are actually packed into a byte. The format must specify code order, signed interpretation, scale indexing, alignment and handling of tails. A kernel can further interleave or transpose tiles for its instruction set. The same logical matrix may therefore have several incompatible physical representations. Preserve a format version and test a tiny pattern with distinct values and groups; visually uniform weights are poor tests for swapped nibbles or incorrectly indexed scale arrays.
Group boundaries can cross or align with kernel tiles, affecting the number of metadata loads and conversion instructions. Some runtimes repack weights once during loading, which adds startup time and resident copies; others require offline conversion. If a shape or format is unsupported, a fallback can change both precision and performance. Report the selected kernel and whether fallback occurred. Do not credit a checkpoint’s disk compression with a resident-memory benefit if the runtime immediately stores a second full-precision copy.
A conversion can add memory traffic
Name a boundary such as high-bandwidth device memory to the execution units. An unfused conversion may read packed weights, write a reconstructed floating-point temporary back across that boundary, and have the multiply read that temporary again. A fused path can keep reconstructed fragments in registers or shared memory without those extra global transfers. This does not make conversion free: unpacking, scale application, synchronization and register pressure still consume resources and can reduce occupancy. Measure where each temporary actually lives rather than inferring it from source-level operations.
For a synthetic roofline comparison, count the same useful mathematical operations and include all relevant bytes crossing the same boundary in every case. The corresponding intensity is useful FLOP divided by those bytes. A fixed compute ceiling is a simplification: an actual quantized kernel can have different instruction throughput or conversion limits from the baseline. A bandwidth ceiling can help reject an impossible claim, but cannot establish realized throughput when scheduling, latency, cache behavior and extra instructions have not been modeled.
Benchmark the actual converted artifact
Measure cold startup and steady-state execution separately, including repacking, graph compilation and warmup where relevant. Exercise production batch sizes, context lengths and tail dimensions; a fast aligned tile can hide a slow general path. Record device, runtime and kernel versions with correctness comparisons against the intended quantized reference. Throughput, per-request latency and peak memory answer different questions. A format can improve one while worsening another, especially when additional workspace changes how many requests fit concurrently.
In the roofline lab, Reset and set both I=0.8 FLOP/byte and B=80 decimal GB/s; predict a 64 GFLOP/s ceiling. Separately Reset and set both I=1.6 and B=80; predict 128 GFLOP/s. These are a hypothetical halving of declared traffic for unchanged useful work, distinct from the worked example’s exact byte totals. Play traces the selected roof without changing its values. The lab contains no nibble unpacker, dequantization instructions or kernel fallback, so neither ceiling is a measured quantized-kernel speed.
Work it through
An unfused temporary can erase compression gains
For one synthetic execution, 128 weights each participate in one multiply-add, counted as 256 useful FLOP. Baseline weights occupy 256 bytes. A four-bit format uses groups of 32, each with a two-byte scale and no zero point. Common activation and output traffic is 64 bytes. Count all transfers at a device-memory boundary with no cache reuse.
Packed codes occupy 64 bytes and four scales occupy eight bytes, totaling 72. The baseline moves 256 weight bytes plus 64 common bytes=320 bytes, giving useful intensity 256/320=0.8 FLOP/byte.
An unfused path reads 72 packed bytes, writes a 256-byte reconstructed temporary and reads it again for multiplication. Add 64 common bytes for 648 bytes total. Its intensity is 256/648≈0.3951 FLOP/byte, below the baseline despite the compressed stored weights.
A fused path reads the 72-byte representation and keeps reconstructed fragments inside the boundary. With the same 64 common bytes it moves 136 bytes, giving 256/136≈1.8824 FLOP/byte. Its traffic reduction is 320/136≈2.35, not the weight-payload-only ratio.
The execution path determines traffic: 320 baseline bytes, 648 unfused bytes and 136 fused bytes under the declared assumptions. Actual instruction cost and achieved memory behavior still require measurement.
The living notebook
Explore the model
Interactive lab · roofline
Keep useful work and memory boundary consistent
Reset → set I=0.8 and B=80 → predict 64 GFLOP/s. Separate Reset → set I=1.6 and B=80 → predict 128 GFLOP/s for a hypothetical halving of total boundary bytes.
- Attainable ceiling
- 640 GFLOP/s
- Ridge intensity
- 12.5 FLOP/byte
- Ceiling regime
- Memory-bound ceiling
Synthetic roofline ceiling min(1000, B·I), in GFLOP/s, with B in decimal GB/s and I in FLOP/byte. Playback traces the roof without changing the selected kernel. This upper-bound model is not a measured throughput prediction, hardware comparison or price model. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forThe fixed 1,000 GFLOP/s roof is synthetic. Playback does not run unpacking or measure a kernel, and these settings are separate from the worked example’s exact traffic totals.
Pause & reason
Check your understanding
Your turn
Try it yourself
Double the worked example to 256 weights and 512 useful FLOP, with the same four-bit groups and scales. Let common traffic be 128 bytes. Compute baseline, unfused and fused bytes, then compare intensities.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Each component doubles, including the full-precision temporary and metadata. Keep both the temporary write and read in the unfused case.
Compare with the solution
Packed storage is 128 code bytes plus 16 scale bytes=144. Baseline traffic is 512+128=640 bytes. Unfused traffic is 144+512+512+128=1,296 bytes; fused traffic is 144+128=272 bytes. Intensities remain 0.8, approximately 0.3951 and approximately 1.8824 FLOP/byte respectively, because useful work and every declared traffic component doubled.
Keep the idea
Take it with you
- Packed layouts require explicit indexing, group mapping and tail rules.
- Materialized dequantization can increase total memory traffic.
- A ceiling needs consistent useful work and a named byte boundary.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.