A nonzero gradient can still produce no stored update

For an ordinary non-boundary value on a local uniform grid, a nearest-rounded update smaller than half the step can return the original parameter. Halfway cases depend on tie rules, and binade boundaries have asymmetric neighboring spacing, so use actual neighbors when precision matters. The mathematical update can be nonzero even when its stored result is identical. Conversely, a zero gradient or zero learning rate is a different reason for no movement. Log an appropriate changed-parameter fraction or ulp-scale diagnostic alongside gradient norms when investigating apparent stagnation.

A wider master parameter can accumulate small updates that would be lost if each step wrote only a coarse value. A low-precision model copy may remain unchanged for several steps and then move when the master crosses a rounding boundary. That is not the same as applying each discarded update to a permanently coarse accumulator. Keep the master and copied-weight roles clear when comparing checkpoints or resume behavior. Saving only the visible low-precision copy can discard optimizer progress and state needed to continue the same numerical trajectory.

An adaptive optimizer can square gradients, average moments, apply bias corrections, take square roots, and add a denominator stabilizer. Each step can require more range or precision than the gradient tensor alone suggests. A representable gradient's square can underflow, and a small stabilizer can cast to zero in a coarse format. That can change the effective update or produce a nonfinite division. Retaining moments and updates in a wider representation is one design choice, but verify the actual implementation, fused kernels, and checkpoint storage instead of assuming a universal optimizer policy.

Changing the stabilizer merely to avoid a visible NaN changes the optimization algorithm and deserves a quality check. Likewise, a larger learning rate might cross the local grid while also making the mathematical method unstable. Separate representation loss from the desired optimization dynamics. Compare one-step updates on known tensors, moment evolution, nonfinite checks, and downstream loss behavior. A finite final parameter does not prove the internal state remained accurate, and an apparently identical model copy does not prove that a wider master state failed to advance.

Construct a small reference update with exactly represented parameters and gradients, then inspect the parameter's previous/next representable values and the computed destination. Compare a coarse-only update, a wider master, and a wider-state variant while holding the mathematical hyperparameters fixed. Include save-and-resume tests that restore the state representation required by the chosen method. Report storage and bandwidth cost alongside quality. A compact checkpoint that cannot resume the promised update path is a different artifact, even if it produces similar inference outputs initially.

The optimization lab illustrates ideal quadratic updates rather than ulp loss. Reset, set both step size η=0.1 and initial x0=2, predict x1=1.8, then Step once. Playback advances two iterations per second, so each half-second Step advances one. Its model is x_n=(1−η)^n x0 on x²/2 and has no low-precision parameter cast, moments, or master copy. A stationary coarse parameter with nonzero η can therefore have a representation cause absent from this diagram. Reset and re-enter both values before a separate comparison.