Scale the backward path before tiny gradients disappear

For a constant positive finite scale S independent of the parameters, differentiating S times the loss multiplies the exact gradient by S. Dividing the resulting gradient by the same factor recovers the intended gradient in exact arithmetic. In mixed precision, that transformation can prevent some intermediate backward values from becoming zero. It does not widen the exponent range of every forward operation, fix an already invalid loss, or recover information discarded before the scaled path reaches an operation. Check where underflow actually occurs before attributing a training problem to one small final gradient.

For IEEE binary16 with gradual underflow, the smallest positive normal is 2⁻¹⁴ and the smallest positive subnormal is 2⁻²⁴. Under nearest rounding, some still smaller positive values round to zero. Flush-to-zero behavior can remove a larger region. These details depend on the actual instruction and backend, not just the tensor's storage dtype. Wider gradient accumulation and optimizer state can complement scaling, but output casting, reduction precision, and individual operation support still need inspection. Another low-precision format can have different range and precision tradeoffs.

Gradient clipping thresholds are usually defined for the unscaled gradient norm. Clipping scaled gradients at the original threshold changes the effective rule, so unscale at the documented point before inspecting or clipping them. During gradient accumulation, keep the scale and normalization consistent across the effective batch; combining gradients with incompatible scales and dividing once produces a different quantity. The framework may perform unscaling and nonfinite checks within a helper, so understand that path instead of adding a second unscale operation blindly.

A large scale can overflow one or more gradient components. Dynamic scaling detects nonfinite gradients, avoids applying the invalid optimizer step according to its contract, and adjusts scale for later work. This changes the actual step history, which can interact with scheduler or accumulated-state logic. Do not force a step merely to keep an iteration counter advancing. Record skipped updates, scale trajectory, and finite-value checks when diagnosing convergence. A finite scale value alone is not evidence that every forward intermediate, loss, or optimizer state remains valid.

Use controlled batches with known tiny and large components, compare gradients before and after unscale, and verify whether parameters actually changed. Separate forward instability from backward range loss and from optimizer-state quantization. Check the documented AMP operation policy for the deployed framework and hardware; autocasting and gradient scaling perform different jobs. Compare quality and throughput under the same effective batch and update policy. A faster loop that silently skips most meaningful updates is not an equivalent training run, even if its loss values remain finite for a while.

The quadratic lab illustrates the exact scaling identity only. Reset, set both coefficient a=1 and position x=0.5, predict tangent slope one, then Play or Step. Reset and re-enter both a=2 and x=0.5, predict slope two. The fixed function is y=ax² with derivative 2ax, so changing a scales that mathematical gradient. It does not emulate tiny gradients, binary16, dynamic scales, clipping, or skipped optimizer steps. Controls retain elapsed time on edits, so each comparison uses Reset and re-enters both values.