Approve a precision change with separate quality and cost evidence

Begin with small fixtures whose exact answers can be derived or computed with a genuinely stronger method. Include near-zero cancellation, a large positive partial sum followed by small contributions, finite values near range limits, and optimizer updates below destination spacing. A wider floating-point implementation is a useful reference candidate, but verify convergence as reference precision increases or compare exact representable cases when possible. The reference can have its own unstable formula, narrow intermediate, or backend-specific reduction, so merely naming its output dtype does not certify it.

Create ablations that change input storage, multiplication, accumulation, optimizer state, and output storage separately when the framework permits. Record the operations actually used under automatic mixed precision rather than inferring them from model parameters. Distinguish a master-parameter change from an accumulator change and keep scaling policy explicit. Kernel choices, fused operations, and backend precision controls can change with shape as well as dtype. Hold data, batch shapes, mathematical hyperparameters, and the intended number of valid updates fixed before attributing a result to one precision boundary.

Absolute tolerance matters near zero because a pure relative comparison has no useful nonzero allowance at a zero reference. Relative tolerance allows errors to scale with larger reference magnitudes. Inspect componentwise failures as well as aggregate norms, because a small average can conceal a critical coordinate. Choose tolerances using problem scales, sensitivity, and downstream requirements before inspecting a candidate's favorable cases. An unexplained failure calls for diagnosis; silently increasing the threshold until it passes converts a test into a record of the observed discrepancy.

Mechanism tests should be followed by representative inference or training evaluations. Track the task metric, nonfinite events, skipped optimizer steps, and convergence trajectories, including variation across repeated runs where randomness matters. Similar final losses can conceal different valid-update counts or a different data schedule after overflow skipping. Gradient checks also need care: a finite-difference step that is too small can disappear in subtraction, while a large step introduces truncation error. Check the reference method and its step-size behavior instead of treating one finite-difference disagreement as a complete diagnosis.

Benchmark after a stated warmup and compilation policy, and synchronize asynchronous device work at the intended timing boundaries. Report whether transfer, optimizer work, communication, and setup are included. Use the same effective batch and valid work count when comparing latency or throughput, and state whether memory means peak allocated, reserved, or total device usage. Keep software, hardware, shapes, selected precision controls, and reference outputs with the result. A configuration accepted for one regime should be rechecked when a different shape or backend selects a new numerical path.

The roofline lab supports a deliberately narrower cost comparison. Reset, set both I=4 FLOP/byte and B=100 decimal GB/s, predict a ceiling of 400 GFLOP/s, then Play. Reset and set both I=16 and B=100 to predict the fixed compute ceiling of 1000 GFLOP/s. Its intensity is an input parameter rather than a precision switch; the display measures no actual kernel, latency, memory footprint, or task accuracy. Use it to reason about a possible bandwidth-to-compute transition, then approve a real precision configuration only from the separate reference, quality, and measured-cost evidence.