Training dynamics and optimization failure

For $J(w)=aw^2/2$ with $a>0$, gradient descent multiplies $w$ by $1-\eta a$ each step. Convergence requires the magnitude of this factor to be less than one. A factor between zero and one approaches the minimum without crossing it; a negative factor of magnitude below one oscillates while shrinking. At or beyond the stability boundary, loss need not decrease. Even a convex problem can therefore fail because of the update rule.

In many dimensions, different curvature directions have different preferred step sizes. A narrow valley combines steep motion across the valley with slow motion along it, so one global learning rate can be awkward. Feature scaling and preconditioning help this geometry; momentum carries a moving history of gradient directions. The one-dimensional bound is exact only for the stated quadratic. A changing nonconvex neural-network loss requires empirical monitoring and does not inherit a universal safe rate.

A minibatch gradient averages gradients for a subset of examples. With suitable independent uniform sampling it estimates the full training gradient without systematic bias, but it has sampling variance. A larger batch can reduce that variance while increasing memory and work per update. It does not automatically make training faster to a target quality. Correlated examples, variable sequence lengths, and nonuniform sampling complicate the relationship between the nominal batch size and the information in an update.

Adam maintains moving averages of gradients and squared gradients, applies bias corrections, and divides a first-moment estimate by a second-moment scale. This adapts coordinate-wise step magnitudes but is not a Hessian inverse or a guarantee of convergence to a global optimum. Decoupled weight decay, as in AdamW, shrinks parameters separately from the adaptive gradient step; simply adding an L2 term to Adam's loss is generally a different update. Record the optimizer definition rather than using these names interchangeably.

When loss explodes, inspect finite inputs, labels, activations, and gradient norms; reduce the learning rate; and confirm the loss reduction and accumulation scaling. When loss is flat, verify that parameters require gradients, the optimizer owns the intended parameters, and the update actually changes them. Check that masks do not discard every target. Log update norm relative to parameter norm, because a finite gradient can still produce a negligible or destructive step.

Separate optimization success from generalization. A descending training curve says the optimizer is fitting its objective, not that future performance is improving. Use a validation schedule and compare experiments at a common budget such as processed examples or wall-clock time. Warmup and learning-rate decay are design choices with task-dependent effects. Gradient clipping bounds a measured gradient norm but cannot repair invalid labels or prove stability. Change one suspected cause at a time and retain a small reproducible failure case.