Parallel reduction order can change a valid floating-point answer

Real addition allows parentheses to move freely. A stored floating-point sum inserts rounding at each internal node of its reduction tree. Different trees therefore expose different intermediate magnitudes, cancellation patterns, and tie cases. For a sum with large positive and negative terms, a small term may survive in one grouping and disappear or be displaced by a rounding increment in another. If products are also involved, fused multiply-add and separate multiplication-plus-addition can introduce another distinction because they round at different points.

A reduction's sensitivity depends on the inputs as well as its order. For a nonzero exact sum s, the sum of absolute input values divided by |s| gives a useful summation condition measure. It is one for nonnegative inputs but can become large when cancellation makes the final sum small. A highly sensitive sum may show a substantial relative difference from a tiny absolute perturbation. When s is zero, that relative measure is undefined and an absolute scale must be stated instead of reporting a misleading infinite percentage as an ordinary accuracy result.

A deterministic execution repeats the same result under its stated platform, software, and input conditions. It may repeat a biased or inaccurate result perfectly. An execution with several permitted orders may return slightly different values that all satisfy a justified error tolerance. These are distinct properties. Neither setting a random seed nor using a higher output dtype automatically fixes a reduction order, the precision of its internal partial sums, or the numerical effects of a backend's algorithm selection.

For distributed learning, changing rank count can change how many terms are summed locally, which chunks cross rank boundaries, and which collective algorithm combines them. Averaging gradients also introduces scaling operations whose placement can affect rounding or overflow. Compare the same mathematical contribution set before attributing a difference only to order: a changed batch partition, loss normalization, skipped nonfinite step, or random augmentation can change the mathematical problem itself. Preserve and replay a small captured set of partial gradients when isolating the communication reduction.

Start with a minimal reduction fixture and a documented higher-precision or exact reference, then vary one execution choice at a time: input order, tree, accumulator type, fused operation policy, or collective layout. Record absolute errors, a meaningful scale, exceptional values, and repeatability. A mismatch much larger than a justified error envelope remains a reason to investigate indexing, synchronization, data races, and invalid assumptions. Floating-point nonassociativity explains a class of differences; it is not a blanket exemption from checking correctness.

The matrix lab illustrates conditioning separately from parallel reductions. Reset, set both a=1 and b=0, predict condition number 1, then Play. Reset and set both a=1 and b=2 to predict condition number 3+2√2, about 5.828. Readouts describe the selected final matrix A=[[a,b],[0,1]] while playback interpolates its geometry. This invertible two-dimensional family contains neither gradient reductions nor floating-point error injection. Use it to distinguish sensitivity from an implementation's error, then test actual tree-dependent drift on captured numerical data rather than treating the animation as a backend experiment.