Derivative checks provide targeted evidence, not an oracle

For a smooth map at a valid input, the central directional difference [f(x+hv)−f(x−hv)]/(2h) approximates J_f(x)v. Its truncation error is typically second order in h under sufficient local smoothness and a fixed direction. Floating-point subtraction and function-evaluation error can grow in relative importance as h shrinks, so making h arbitrarily tiny is not a universal improvement. Scale perturbations to the input and direction, compare several h values, and examine both absolute and relative error where appropriate.

The familiar error sketch Ch²+Dε/h explains a possible U-shaped behavior, not an exact error law or universal optimal step. Constants depend on derivatives, evaluation scale, conditioning, and the numeric format. A symmetric quadratic can have zero central-difference truncation error in exact arithmetic, while a cubic has a visible h² term. Discontinuous branches, nonsmooth boundaries, invalid domains, or stochastic evaluations can violate the smooth deterministic assumptions entirely. Distinguish a checker failing its assumptions from a derivative implementation failing the intended contract.

At fixed real input, compatible forward and reverse derivative products satisfy uᵀ(Jv)=vᵀ(Jᵀu). This is useful for large operators whose full Jacobian is impractical to form. Choose nontrivial seeds and verify dimensions before evaluating the scalar products. A transpose mistake, missing accumulation path, or incorrectly shaped reduction often breaks the identity. However, implementing JVP and VJP from the same wrong matrix can pass it exactly. Combine adjoint consistency with independent primal perturbations and hand-derived cases rather than treating two matching numbers as a proof of the true derivative.

For a scalar loss, compare directional differences with the gradient dot v instead of perturbing every coordinate if that matches the test target. Sparse layouts, aliases, custom rules, and higher-order paths need their own supported checks. PyTorch 2.9's gradcheck documentation describes assumptions and limitations, including numerical differences near nondifferentiable points. Use a suitable precision and tolerance, restore RNG and mutable state between evaluations, and avoid an optimizer update during the check. A stochastic output mismatch can otherwise dominate the derivative signal even when the rule is correct.

Choose inputs inside each valid branch and test boundary conventions separately. Include zero and scaled seeds, repeated indices, shared subexpressions, several shapes, and supported higher-order compositions. Preserve minimized failing examples and record framework version, dtype, device, state, and perturbation scale. A passing random-direction test can miss errors orthogonal to that direction; multiple directions improve evidence without proving every input. Explain whether the checker validates a true derivative, a declared surrogate, or a runtime convention, since a finite-difference mismatch can be expected for the latter two.

The quadratic lab shows why one numerical example can be unusually easy. Reset, explicitly set BOTH a=1 and x=1, predict analytic slope two, then Play or Step. Compute the central quotient for this quadratic on paper: its exact-arithmetic value is two for every nonzero h. Reset and re-set BOTH a=2 and x=1; the value is four. Edits retain t; playback traces an analytic curve and does not simulate rounding, change h, or run gradcheck. Use the cubic worked example to see nonzero truncation rather than concluding from the quadratic that step size never matters.