Compiler graphs and kernel fusion

A compiler represents operations, data dependencies, shapes, dtypes, and sometimes effects in an intermediate graph. Transformations can reorder independent work, eliminate redundant computation, specialize shapes, or lower high-level operations into device kernels. Correctness includes aliasing and mutation behavior, not merely algebraic equality on isolated values. Random-number generation, host interaction, data-dependent control flow, and unsupported operations can restrict capture or divide execution into separate graph regions.

Specialization makes assumptions that must be checked when the program runs again. A shape or stride change can require another compiled variant; too many variants can consume compilation time and memory. Distinguish first-call latency from steady-state latency and report the workload's shape distribution. A benchmark with one repeated tensor shape may conceal recompilation that dominates a real service with varied requests. Keep a reliable eager reference path for small correctness comparisons.

Consider elementwise operations y=a x+b followed by z=max(y,0), where a and b are scalar constants. An unfused path may read x, write y, read y, and write z. A fused path can read x and write z while retaining y locally. Count bytes at the relevant memory boundary and state whether caches, constant loads, and allocation overhead are excluded. Lower traffic matters most when bandwidth or launch latency limits execution.

The roofline model compares arithmetic intensity with available compute and bandwidth to give a simplified performance ceiling. Fusion can increase useful work per transferred byte, but can also increase register use, cause spills, reduce occupancy, or prevent reuse of a highly optimized library operation. Tiling and recomputation can be more valuable than blindly merging everything. FlashAttention is an important example of reorganizing attention around memory movement; its exact-attention claim concerns the mathematical operation, not bitwise identity across floating-point schedules.

Validate outputs, gradients, and relevant side effects before timing. A reassociation valid over real numbers may change floating-point rounding or exceptional-value behavior. Decide whether the application permits that relaxation and quantify its consequences. Test representative dtypes, noncontiguous layouts, edge shapes, and boundary conditions. Numerical tolerance should follow the operation and conditioning, rather than being widened until every failing result passes.

Use device-aware timing that waits for the measured asynchronous work to complete, with warm-up and enough independent repetitions. Report kernel counts, transferred bytes when measured, peak memory, compile time, and end-to-end latency or training time. An optimized kernel that saves two milliseconds may have little product impact when input parsing consumes a second. Conversely, reducing allocation can improve peak memory enough to enable a larger useful batch, an effect a fixed-batch microbenchmark would miss.