Benchmark an architecture-specific schedule within the whole workload

FlashAttention established an I/O-aware exact tiling approach; FlashAttention-2 improves parallelism and work partitioning; FlashAttention-3 studies Hopper-oriented asynchrony and low precision. These are not interchangeable product labels for one universally optimal binary. The FA3 paper’s Hopper schedule overlaps data movement, matrix operations, and softmax-related work using architectural facilities. Its low-precision variants also need a numerical-accuracy contract rather than inheriting every claim made for a higher-precision calculation.

Pin implementation support to a release. For example, the FlashAttention 2.8.3 README scopes its Hopper beta to H100/H800 with CUDA at least 12.3 and lists FP16/BF16 forward/backward plus FP8 forward. Its separate CUDA and ROCm backends have their own feature matrices. These are support statements for that version, not promises about every accelerator or future release. Verify the actual chosen backend and its accepted head dimensions, layouts, masks, and backward capabilities.

Separate forward training, forward-plus-backward, prefill, and incremental decoding. State batch size, query/key lengths, query/KV head counts, head width, dtype, accumulation mode, causal alignment, dropout, and layout. Include preprocessing, packing, compilation, and cache-update costs when they belong to the production request, while reporting isolated kernel time separately. A favorable training result does not establish the same benefit for one-query decode against a large cache.

Warm up compilation and allocations before steady-state timing, or report cold-start latency as a separate target. Use synchronized device events or another documented method that waits for the measured work, and control unrelated concurrent work. Record medians and tails across repetitions instead of selecting the fastest run. Measure memory under the same semantic features, and check that an unsupported feature did not silently trigger a different backend or remove required work.

A portability benchmark should include a precise device model, toolkit, compiler, framework, kernel revision, backend, clock/power conditions where relevant, and reproducible inputs. Validate outputs and gradients on that target before interpreting timings. Compare against a strong available baseline with the same semantics; a dense low-precision fused kernel and a higher-precision unfused reference are useful for accuracy analysis but are not automatically a fair latency contest.

Report both the mechanism and its practical boundary. If a faster kernel reduces only a small fraction of request time, improve another bottleneck or reconsider whether the integration cost is worthwhile. If data movement dominates, a schedule with better reuse may help; if synchronization or matrix utilization dominates, a bandwidth-only explanation is incomplete. End-to-end evidence should decide whether the architecture-specific optimization improves the intended workload rather than merely producing an attractive isolated throughput number.