Kernel fusion: remove traffic without hiding resource costs

Consider two pointwise stages producing t=f(x) and then y=g(t), with no other consumer of t. Separate kernels can read x, write t, read t, and write y. A fused kernel can compute both stages for an element and write only y. This is a simple local dependency: it does not require a grid-wide synchronization between stages. If another branch needs t, the implementation may still need to materialize it or recompute it. The full computation graph determines which traffic is actually removable.

A reduction or cross-block dependency can make fusion more complicated than placing source statements next to one another. Threads consuming a globally complete intermediate need a supported coordination design, not an ordinary block barrier pretending to cover the device. Also preserve the numerical operation the user requested. A materialized lower-precision intermediate introduces a conversion that unrestricted fused arithmetic might omit; contracting a multiply and add can change rounding. Decide whether the contract permits that difference or requires an explicit cast or other preservation of the intermediate semantics.

A fused kernel can retain more values per thread at once, increasing register demand. That may reduce the number of resident blocks or force some values into local memory. In CUDA terminology, local memory is private-addressed storage that can be backed through the device-memory hierarchy, not a promise of a fast on-chip register. Caches may satisfy some spilled accesses, but spills still add work and can undo traffic savings. Inspect compiler resource reports and profiling evidence rather than assuming the source-level number of variables equals allocated registers.

A register-only residency bound divides a multiprocessor's register capacity by registers per block, but actual residency also depends on allocation granularity, thread and warp limits, shared memory, block limits, and architecture. Restricting the register count can raise the bound while adding spills or instruction overhead. Higher occupancy is not automatically higher throughput. A kernel with fewer resident blocks may benefit from reuse or fewer instructions, while another needs more independent warps to hide latency. Compare the mechanism with measured stalls and complete workload time.

Compare separate and fused versions with the same shapes, layouts, dtypes, masks, and precision policy. Include temporary allocation and conversion only according to the production boundary, and report both kernel-only and end-to-end results when useful. Check whether fusion removes useful overlap with other work or changes scheduling for a small batch. Revalidate edge rows and nonfinite behavior after combining reductions or normalization. A faster output that omitted a required intermediate consumer or changed the function is not evidence of an optimization.

The roofline lab isolates the potential traffic benefit. Reset, set I=2 and B=100, predict ceiling 200 GFLOP/s, then Play or Step. Separately Reset, set I=4 and B=100, predict 400 GFLOP/s, then Play or Step. Its compute ceiling stays at 1,000 GFLOP/s. The model cannot change register allocation, produce spills, or fuse a graph. Use the twofold synthetic bound as a hypothesis for eliminating intermediate bytes, then explain what compiler and timing evidence would show whether register pressure or another bottleneck prevented that gain.