Occupancy: a latency-hiding tool, not the performance score

A resident warp has its context allocated on an SM. It may wait for a dependency, memory, or synchronization. If every resident warp waits on the same limiting resource, adding more can fail to improve throughput. Conversely, a smaller set of warps with sufficient instruction-level parallelism may keep the relevant execution units busy. The required amount of latency-hiding work depends on architecture, instruction mix, memory behavior, and the number of independent operations available within each warp.

Theoretical occupancy describes a resource-based upper configuration, while achieved occupancy summarizes observed active residency over a defined measurement interval. Neither directly measures useful elements completed per second. Tail effects, uneven block work, divergent paths, and memory transactions can reduce effectiveness even with attractive occupancy. Read each profiler metric’s definition and averaging scope. Do not compare a per-SM theoretical fraction with an application-wide average as if they were the same observation.

Arithmetic intensity counts operations relative to bytes moved through the selected bottleneck. Reusing data can increase intensity and reduce traffic, even if shared-memory use lowers residency. A roofline model compares a compute ceiling with a bandwidth ceiling. The model needs a defined operation type and data-movement boundary; peak tensor throughput cannot be substituted for scalar arithmetic throughput without justification. Real performance can lie far below both ceilings because the model omits dependencies, synchronization, and inefficient access.

Register pressure creates another tradeoff. Forcing extra resident warps by restricting registers can introduce spills, which consume instructions and memory bandwidth. A larger per-thread working set can sometimes lower occupancy but avoid repeated loading. Validate numerical results when changing algorithms or instruction forms, then compare kernel time and application time under the same workload. If a kernel is only a small part of end-to-end latency, a local improvement can have limited user-visible effect.

A useful tuning hypothesis names the bottleneck and expected mechanism: fewer transferred bytes, shorter dependent path, less divergent work, or improved residency without excessive spill cost. Record both the resource change and the timing change. Use representative shapes and batch sizes, include warmup where steady state matters, and report variability. A larger launch that improves one large batch can regress small workloads dominated by launch overhead. The final decision should reflect the workload the system actually serves.

The roofline lab has a fixed synthetic compute peak of 1,000 GFLOP/s. Reset, set operational intensity I=4 FLOP/byte and bandwidth B=80 GB/s, predict a 320 GFLOP/s ceiling, then Play or Step. Reset and set I=16 and B=80 to predict the 1,000 ceiling. Playback traces the roof without changing the selected kernel parameters. None of these numbers describes an NVIDIA product or measured occupancy. They isolate how a bottleneck ceiling changes when intensity improves.