Reduction order and compensation solve related but different problems

A sequential sum repeatedly adds the next term into one accumulator. A balanced pairwise tree combines partial sums, so a term passes through roughly logarithmic depth rather than a chain proportional to the number of terms. Under the standard local-error model with no exceptional arithmetic, that shorter depth improves a common worst-case forward bound. But a particular pairing can still lose a small term, and changing data order changes which magnitudes meet. A tighter worst-case bound does not prove every individual input is more accurate under every tree.

The conditioning of the exact sum remains relevant. When large positive and negative contributions nearly cancel, a small absolute error can be large relative to the final sum. Compare absolute error and a meaningful scale before dividing by a near-zero reference. A wider accumulator reduces local rounding but does not undo products already rounded too coarsely. In a dot product, input conversion, multiplication, accumulation, and output casting each have their own precision. Name which stage a proposed reduction change actually modifies.

One Kahan variant subtracts a stored correction from the next input, adds that adjusted input to the sum, then estimates the lost low-order part through the difference between the new and old sums. This often greatly improves a long accumulation. Yet the adjusted input and correction calculations are still finite-precision operations, and an unfavorable magnitude change can discard the very correction intended to help. Compiler reassociation can also invalidate the carefully ordered expressions. Preserve the algorithm's arithmetic contract rather than algebraically simplifying the correction to zero.

Neumaier's variant adds a correction using a magnitude comparison between the running sum and the next term, then incorporates the accumulated correction in the final answer. That branch can recover a lost contribution in cases where the ordinary Kahan recurrence fails, including the controlled example below. It is still not an all-input exact sum, especially when correction accumulation or range assumptions fail. For stronger guarantees, consider documented wider, expansion-based, or exact accumulators with their stated cost and domain. Validate the actual implementation instead of assuming its function name proves the desired accuracy.

Test realistic gradient distributions together with adversarial mixtures, permutations, and scale changes. Verify the same represented inputs and define whether products are fused with addition. Record reduction topology, accumulator precision, correction precision, compiler settings, throughput, and memory. A compensated kernel adds arithmetic and dependencies; a balanced kernel can fit parallel hardware better. The best choice depends on the accuracy required by the later computation, not merely the smallest scalar error on one convenient fixture or the highest nominal operation count.

The roofline lab isolates a synthetic work-versus-memory ceiling, not reduction accuracy. Reset, set both intensity I=2 FLOP/byte and memory bandwidth B=100 GB/s, predict 200 GFLOP/s, then Play or Step. Reset and re-enter I=8 and B=100 to predict 800 GFLOP/s. The fixed compute roof is 1,000 GFLOP/s and playback only traces it. Extra compensation operations can raise counted intensity without increasing useful sums per second, so this diagram cannot choose a reduction algorithm or certify Kahan, Neumaier, or pairwise error.