Backpropagation through a computation graph

Consider $a=wx+b$, $h=\sigma(a)$, and a loss $L(h,y)$. The forward pass stores values needed for derivatives. Starting with the derivative of the loss with respect to its own value equal to one, the reverse pass multiplies each arriving sensitivity by the local derivative. The derivative of $a$ with respect to $w$ is $x$, so a large input can amplify the weight gradient even when the upstream sensitivity is modest. Units and input scaling matter here too.

Reverse-mode differentiation is especially useful when many parameters influence one scalar objective. It computes vector-Jacobian products without constructing every full Jacobian. The graph is the actual executed computation, including branches that were taken. Automatic differentiation follows the implemented expression; it cannot infer a missing operation or correct a wrongly normalized loss. A perfectly computed gradient of the wrong objective is still the wrong training signal.

If one parameter feeds two downstream branches, its derivative is the sum of both contributions. For $L=w^2+3w$, one branch contributes $2w$ and the other contributes $3$. Forgetting one branch loses real sensitivity. This also explains gradient accumulation across examples: a mean loss averages their contributions, whereas a summed loss adds them. Changing reduction semantics silently changes update scale if the learning rate stays fixed.

In PyTorch, backward accumulates into parameter gradient buffers, so ordinary independent optimization steps need gradients cleared at the intended boundary. Accumulation over microbatches is deliberate only when scaling matches the desired global objective. Detaching a value or converting it into an unrelated constant breaks its differentiable path. In-place changes to saved values can invalidate a backward calculation. Watch tensor shapes, graph connectivity, and reduction rules before blaming the optimizer.

For a small smooth function, compare an automatic derivative to a central finite difference: evaluate the loss at a parameter slightly above and below its current value. A discrepancy can reveal a graph error or a wrong analytical derivative. The step must balance truncation error against floating-point cancellation. Use higher precision for a diagnostic and avoid nondifferentiable points such as exactly zero for ReLU, where different conventions or nearby slopes complicate the comparison.

Deep compositions multiply many local derivatives. Repeated small factors can shrink early-layer gradients; large factors can amplify them. Initialization, activation choice, normalization, and residual paths influence this behavior. A zero derivative through an inactive ReLU is a model property, not necessarily an autograd bug. Inspect gradient norms by layer and try to overfit a tiny clean dataset. A system that cannot fit a few examples usually needs debugging before more data or compute.