Instability diagnosis: localize the first broken quantity

Save clean inputs, encoded latents if used, timesteps, Gaussian draws, condition masks, coefficients, targets, and reference outputs. Recompute a forward and loss in a wide dtype and compare with the intended mixed-precision path. Check data ranges, channel order, latent scaling, coefficient broadcasting, and whether the scheduler expects epsilon or velocity. Verify the target uses the intended noise realization. A training variant that perturbs the noisy input while retaining another target is a separate declared objective, not an accidental implementation detail to ignore.

For ordinary unweighted per-coordinate noise MSE with standard Gaussian targets, a zero predictor has expected loss one at every timestep. That is a useful scale baseline under exactly those assumptions. A score target, summed feature reduction, noise offset, changed normalization, or timestep weight changes the number. One sampled batch can exceed the expectation without a defect. Inspect loss by timestep and data slice, and preserve both raw and weighted values so an objective change is not mistaken for numerical divergence.

Trace activations, loss terms, unscaled gradients, parameter updates, optimizer moments, and skipped-step counters. Mixed-precision loss scaling changes gradient magnitude until it is undone; clip the intended unscaled gradients according to the framework's supported procedure. PyTorch 2.14 documents the ordering needed for AMP and clipping. A nonfinite gradient is not repaired by ordinary norm clipping, and repeated skipped updates can leave training stalled while logging continues. Compare a small full-precision run to isolate arithmetic range problems from a wrong target or unstable objective.

Clipping limits a finite gradient norm under the chosen rule, but does not prove that the direction is useful or that the learning rate is appropriate. Optimizers with moments or preconditioning do not turn the clipped gradient into a simple learning-rate-times-gradient update in general. Record actual update norms and effective batch reduction. If instability correlates with rare timestep weights, condition groups, or malformed examples, diagnose that association before lowering every learning rate or replacing the entire schedule.

Check autoencoder reconstruction, target conversion, scheduler configuration, timestep range, conditioning representation, guidance convention, and EMA restoration before interpreting generated artifacts. A model trained for one parameterization can look broken under a sampler reading its output as another. Use fixed examples and a tiny overfit diagnostic to verify that the implementation can fit a controlled task, while retaining a separate held-out evaluation for generalization. Restore a known checkpoint and vary one suspected setting with the same random inputs to test a causal hypothesis.

The optimization lab gives an exact quadratic stability contrast. Reset, set η=0.50 and x₀=1, predict x₂=0.25, then Step twice. Separately Reset, set η=2.50 and x₀=1, predict x₁=−1.5 and x₂=2.25, then Step twice. Each Step advances one iteration. These reachable values remain visible, and the model has no diffusion loss, neural network, AMP, or optimizer moments. Use it to distinguish a demonstrated unstable recurrence from an unsupported claim that the same scalar threshold governs every diffusion optimizer.