Treat a loss spike as evidence to investigate
Preserve batch, numerical, optimizer, and infrastructure evidence before attributing a spike, and distinguish gradient clipping from a stability guarantee.
On this page
The question for today
What you’ll understand
- Separate objective-accounting errors from changed data and numerical failures.
- Calculate a scoped gradient-norm clipping example.
- Design controlled replays that compare competing spike explanations.
A sudden loss increase can reflect a harder batch, corrupted preprocessing, an incorrect denominator, numerical overflow, a stale checkpoint, or an unstable update. The visible curve alone does not identify which mechanism occurred. Preserve the information needed to distinguish those explanations before modifying the run. Every example here is fictional arithmetic for a controlled investigation, not a report of a real model incident or a universal recipe for avoiding divergence.
Validate the measurement before diagnosing the optimizer
Start with the loss numerator, valid-target count, token identities or protected reproducible references, serialization policy, and data-domain mixture. Check whether the displayed value is a current batch mean, a rolling average, or an auxiliary-loss combination. A denominator change can double a metric without changing model probabilities. A batch containing a rare domain can genuinely have higher loss while weights remain healthy. Neither observation alone proves the learning rate is wrong.
Capture the checkpoint identity, successful-update index, applied rate, gradient and parameter norms, finite-value indicators, precision mode, loss-scale state when used, and relevant worker errors. Establish which values were measured before and after unscaling or clipping. PyTorch 2.9 AMP documentation requires unscaling accumulated gradients before interpreting a clipping threshold in unscaled units. A loss-scale overflow skip also changes which attempted updates actually modified weights, so align the records using the run's declared counters.
Use clipping as a specific control with a specific scope
Global norm clipping rescales a finite gradient vector when its norm exceeds a threshold. It bounds that vector's norm under the stated arithmetic, but it does not repair incorrect labels, forbidden attention, or a corrupted optimizer state. With adaptive optimizers, the parameter update depends on stored moments and other terms, so a gradient-norm bound is not automatically an identical bound on the parameter step. Inspect the actual update if that is the property of concern.
Choose an investigation that can falsify a hypothesis. Reevaluate the preserved batch at the pre-spike checkpoint in a controlled higher-precision path; compare nearby ordinary batches; inspect whether a changed packing or tokenizer artifact is implicated. If lowering the rate removes a spike in one rerun, that is intervention evidence under those conditions, not proof that all other mechanisms are absent. Changing rate, batch, precision, and corpus simultaneously makes the outcome harder to attribute.
Separate model recurrence intuition from an actual replay
Use a short, reproducible pre-spike checkpoint window with recorded data order and state. Check a baseline that reproduces the issue before interpreting a changed run. Compare targeted observations such as first nonfinite tensor, exact mask mismatch, or a changed update norm instead of relying only on eventual final loss. If the failure does not reproduce, retain the uncertainty and investigate omitted state, nondeterministic kernels, or infrastructure rather than declaring the system healed.
The optimization lab gives a fully specified counterexample to an excessively large step on one quadratic. Reset, set both η=0.5 and nonzero x0=2, predict contraction, then take four Steps. Reset and set both η=2.5,x0=2 for alternating growth to x4=10.125. Large positions can leave the plotted range while readouts retain the value. This lab has no language data, clipping, mixed precision, Adam moments, or infrastructure faults; its divergence mechanism is known by construction and cannot diagnose a real training spike.
Work it through
Bound one finite gradient before an SGD update
A fictional unscaled gradient is g=[6,8], the global Euclidean clipping threshold is c=2, and a plain SGD update uses rate η=0.1 with no momentum, decay, or other terms. Use exact arithmetic for the clipping calculation.
The gradient norm is √(6²+8²)=√100=10. Since ten exceeds two, the clipping multiplier is 2/10=0.2.
The clipped gradient is [1.2,1.6], whose norm is √(1.44+2.56)=2. This confirms the intended finite-vector bound.
The plain SGD parameter displacement is −0.1[1.2,1.6]=[−0.12,−0.16], with norm 0.2. This last bound uses plain SGD specifically; it does not follow unchanged for an arbitrary adaptive optimizer.
The example bounds a declared finite gradient and its plain-SGD step. It neither identifies a spike's cause nor guarantees that the next loss decreases.
The living notebook
Explore the model
Interactive lab · optimization
Inspect one known divergence mechanism
Fresh Reset → set both η=0.5,x0=2 predicts contraction; Reset → set both η=2.5,x0=2 predicts alternating growth and x4=10.125 after four Steps. Nonzero x0 is explicit; large values may leave the plot. No language-model data, Adam, clipping, or mixed-precision failures are simulated.
- Position x0
- 2
- Objective x²/2
- 2
- Iteration behavior
- Stable step
Gradient descent on f(x) = x²/2: xₙ = (1 − η)ⁿx₀. Playback advances two discrete iterations per second. Values outside |x| ≤ 4 are explicitly reported. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA toy mechanism can be proved within its model, while a real spike needs discriminating evidence.
Pause & reason
Check your understanding
Your turn
Try it yourself
An unchanged loss sum of 8,000 nats is correctly associated with 4,000 valid targets, but a new logger divides it by 2,000. What does the displayed loss do, and what observation would distinguish this accounting issue from genuinely worse predictions?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Recompute the mean using the original eligible positions and inspect target probabilities or the unchanged numerator.
Compare with the solution
The correct mean is 8000/4000=2 nats per token; the new display reports 8000/2000=4. The apparent doubling can occur without any probability change. Recompute the eligible-target count from the actual shifted mask and compare the preserved numerator or per-target losses under the same model and batch. That evidence tests this specific logging explanation; it does not rule out a separate genuine prediction problem elsewhere in the run.
Keep the idea
Take it with you
- A loss spike is compatible with measurement, data, numerical, optimizer, and infrastructure mechanisms.
- Clipping has a finite-vector contract and does not establish universal training stability.
- Controlled reproductions should isolate a mechanism and preserve unresolved uncertainty.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.