Small positive contributions can disappear systematically in a coarse accumulator

Consider adding a fixed positive value a to an already represented accumulator s. In a region with uniform spacing h and nearest rounding, an addition smaller than h/2 returns to s, provided there is no closer boundary on that side. Repeating that operation does not accumulate the discarded remainder. Each next operation begins from the same represented s. A real-valued recurrence grows with every addition, but the stored recurrence can remain fixed indefinitely until some other operation changes its scale or preserves a remainder elsewhere.

This stagnation is different from converting the small input itself to zero. The input can be a perfectly valid normal number, and the exact sum of many such inputs can be large enough to affect s. A wider multiplication followed by a narrow sum can therefore lose contributions after forming correct products. Conversely, a wide accumulator cannot reconstruct products already rounded inaccurately or flushed to zero before they reach it. Record product precision, partial-accumulator precision, and the final conversion as separate stages of the reduction.

Grouping small terms together first can create a subtotal large enough to survive addition to a larger term. That is a concrete benefit of a different summation structure, not a proof that every balanced tree produces the exact real sum. Some inputs contain severe cancellation; others lose products before summation; still others overflow intermediate partial sums. Pairwise summation improves a general depth-based error bound under its assumptions, while compensation or a wider accumulator offers a different way of preserving information. Compare methods on the actual distribution of terms.

If a positive denominator is underestimated while its numerators are otherwise accurate, normalized values can be inflated. For an exact denominator S and stored denominator S_hat, each such quotient has multiplicative factor S/S_hat relative to the desired value. The sum of these computed quotients can exceed one. This observation does not attribute every probability-normalization error to accumulation: exponential approximations, input rounding, division, masks, and later casts have their own effects. Isolate one stage in a controlled fixture before interpreting a complete model output.

For deterministic rounding, the errors of a chosen sequence are fixed values, not independent random draws. A sequence with every small addition discarded has a directional error by construction. To discuss statistical bias, first specify randomness over inputs, operation choices, or a rounding rule and define the estimator whose expectation is being compared. Signed reductions can move either way; the positive example establishes a possible systematic mechanism rather than a universal claim that lower precision always underestimates every learning computation.

Construct fixtures that keep inputs unchanged while varying only accumulation type or reduction order. Include a positive tail below local spacing, a cancellation case, a wide dynamic range, and ordinary representative batches. Measure end-task behavior after these mechanism tests rather than replacing it with one scalar sum. In the roofline lab, Reset, set both intensity I=4 FLOP/byte and bandwidth B=50 decimal GB/s, predict 200 GFLOP/s, then Play. Reset and set both I=8 and B=50 to predict 400. Its fixed compute ceiling is 1000 GFLOP/s. These synthetic bandwidth ceilings do not emulate rounding, a wider accumulator, or the changed hardware throughput such a choice may incur.