Differentiating a derivative needs its own graph

For a twice-differentiable scalar f:R^n→R, its gradient has n components and its Hessian is n×n. A Hessian-vector product H_f(x)v returns n components without requiring every Hessian entry to be materialized. It can be expressed as a JVP of the gradient in a fixed direction. A reverse derivative of the scalar gradient-dot-direction can also produce an adjoint product, with symmetry requiring the appropriate smoothness conditions. State whether v is fixed; differentiating a direction that depends on x adds another term.

A vector-valued original function has a higher-rank second derivative rather than one automatically canonical Hessian. Selecting an output cotangent first defines a scalar combination and therefore a particular Hessian question. The same shape discipline used for first-order JVPs and VJPs still applies. At nonsmooth boundaries, a framework can differentiate its chosen local rules without producing a classical second derivative of the original map. An apparently symmetric numerical matrix is evidence about those calculations, not a proof that the mathematical smoothness assumptions hold everywhere.

PyTorch 2.9's create_graph option requests construction of a differentiable graph for derivative computations, enabling supported higher-order work. retain_graph controls freeing the graph resources used for that traversal. Retaining a first-order graph alone does not record the derivative operations as a new differentiable graph. The defaults and interactions belong to the documented API, but their purposes remain distinct. A repeated first-order backward, a derivative of the first gradient, and several independent loss evaluations have different schedules; choose the mechanism for the requested computation.

Custom rules can break higher-order dependencies by saving an intermediate in a form whose relationship to inputs is absent from the derivative graph or by detaching required values. PyTorch's custom-function guidance requires declaring unsupported double backward explicitly. More retention cannot repair a mathematically wrong rule or recreate a dependency deliberately removed from the graph. Memory also grows with the nested computation and live residuals, so a higher-order calculation needs its own budget. Validate a small analytic example before scaling to a large model whose failure is harder to localize.

Compare a hand-computed product with a directional difference of first derivatives at valid smooth points, using a scale range rather than one perturbation. Check the shape of the first gradient, seed, and returned product separately. If a result is unexpectedly constant or detached, inspect derivative graph connectivity before changing retention settings. For custom rules, test supported nested transformations explicitly. A first-order gradcheck does not establish every second-order path, and a passing symmetry check can miss a consistently wrong symmetric Hessian.

The function lab provides analytic curvature only for y=ax². Reset, explicitly set BOTH a=1 and x=0.5, predict slope one and paper curvature two, then Play or Step. Reset and re-set BOTH a=2 and x=0.5; predict slope two and curvature four. Edits retain t; playback traces the chosen curve and does not run a higher-order derivative graph. The display contains no create_graph or retain_graph behavior. Use the scalar formula to inspect one curvature claim while the worked two-variable calculation establishes the shaped product.