Cancellation can expose errors that happened before subtraction

For a nonzero difference a minus b, independent relative perturbations to a and b can produce an absolute output perturbation proportional to their magnitudes. When the true difference is tiny compared with those magnitudes, the relative output change can be large. This is a property of the problem's input-to-output relationship. It does not imply that every implementation must fail or that every subtract instruction rounds badly. At an exact zero result, use absolute error or another meaningful scale because relative error against zero is undefined.

A reformulation can avoid creating unnecessarily inaccurate large intermediate values. Centering data before accumulating squared deviations often avoids subtracting two large raw moments. Yet input information already lost to a low-precision cast cannot be restored by a stable formula afterward. Likewise, adding a small constant to a denominator changes the mathematical function and can be a deliberate model choice, but it is not a proof that the original computation became accurate. Track input representation, intermediate precision, and the intended statistical definition separately.

Population variance can be written as mean of squares minus square of the mean, or as the mean squared centered deviation. These formulas are algebraically equivalent in exact arithmetic. The raw-moment form can subtract large rounded quantities to obtain a small variance, while a centered or carefully designed online method can keep intermediate values closer to the scale of the desired answer. The chosen accumulation and mean precision still matter. A centered calculation with a badly rounded mean or mismatched normalization can remain inaccurate.

Distinguish population variance from an unbiased sample-variance estimator with another denominator. An experiment comparing formulas must compute the same mathematical quantity on the same represented inputs. Test a batch with a large common offset and small but representable deviations, then compare with a wider or exact reference. Check for negative computed variance before a square root, but do not simply clamp and declare success: clamping can hide a real error source. For normalization layers, examine the downstream output and gradient as well as the intermediate variance.

Measure absolute and scaled errors against a defined reference, inspect the sizes of intermediate operands, and change one representation choice at a time. If widening inputs before a problematic operation helps, locate where the earlier path discarded useful information. If a mathematically sensitive problem remains sensitive in wider arithmetic, that is evidence for conditioning rather than permission to blame a particular accelerator. Reference precision itself needs checking near exceptional values, so use exact small fixtures where possible and increase reference precision to see whether the answer stabilizes.

The matrix lab offers a related sensitivity example, not a variance or cancellation simulation. Reset, set both diagonal scale a=1 and shear b=0, predict condition number one, then Play or Step. Reset and re-enter both a=0.25 and b=0, predict condition number four. Its supported family is only A=[[a,b],[0,1]] with positive a, and readouts describe the selected final matrix while playback interpolates geometry. This illustrates sensitivity changing with a problem, but does not evaluate batch variance, emulate binary16, or measure rounding error.