Unit roundoff is half the spacing convention, not the whole error story

For a normalized binary format with p significand bits including the implicit leading bit, spacing in the interval from one through just below two is two to the power one minus p. Here epsilon means that gap above one. Under round-to-nearest, unit roundoff u is half that spacing, two to the power minus p. Some literature uses machine epsilon for a different convention, so always state the definition. PyTorch finfo eps reports the gap above one, while tiny describes the smallest positive normal value rather than every nonzero value.

Round-to-nearest with ties to even resolves a halfway case using the destination significand's least significant bit. Directed modes instead choose toward zero or an infinity, changing error direction and the usual bound. An ulp describes a local representational step, whose scale changes at powers of two and can differ immediately above and below a boundary. Comparing an optimizer update with the spacing near its parameter is therefore more informative than comparing every update with epsilon at one. A format with wide exponent range can still have coarse spacing at ordinary parameter magnitudes.

For a basic correctly rounded operation whose exact result remains in the normal finite range, the standard model writes the rounded result as the exact result times one plus a small relative error bounded by u. This is a local operation model, not a bound on the final network output after many operations. Near cancellation, a small absolute perturbation in inputs can dominate a tiny result. An exact zero also makes a relative error quotient undefined, even when the operation itself is perfectly meaningful and exact.

Gradual underflow extends the representation with subnormals, whose absolute spacing stays fixed near zero; their relative spacing grows as magnitude shrinks. A pure relative bound by u does not describe every subnormal result. Flush-to-zero settings can remove those values entirely. Overflow can produce infinity or another specified result depending on mode, and NaNs need deliberate handling. Fused multiply-add rounds a product-plus-sum once rather than rounding the product separately first. The actual instruction, accumulation type, compiler settings, and intermediate values therefore belong in a numerical experiment's configuration.

A half-precision input tensor can feed products or accumulations performed at a different precision, followed by an output cast. Conversely, an apparently wider computation may use a reduced-precision input path or intermediate buffer. Check documented backend behavior and use controlled probes around representable neighbors, underflow thresholds, and large intermediate values. Record exact input bit patterns where practical. A final scalar that fits the output dtype does not prove every intermediate fit, and widening only the final cast cannot undo earlier discarded information.

The bisection lab separates algorithmic refinement from representation questions; it does not emulate a chosen floating-point format. Reset, set both target c=2 and upper bound U=4, predict an initial midpoint-bound of two and a bound of one after one Step, then inspect. Each Step advances one bisection because it adds half a second; refinement is capped at fifty iterations. Playback runs the fixed quadratic root model, whose own guarded arithmetic is specified separately. Do not read its shrinking bracket as a measurement of binary16 unit roundoff or an overflow test.