Multivariable calculus

For a function $f(x,y)$, the partial derivative with respect to $x$ varies $x$ while holding $y$ fixed. It answers a controlled question rather than describing every possible path through the input plane. Collecting the partial derivatives gives the gradient. If the function is differentiable, its first-order change is the dot product of this gradient with a small displacement. The neglected remainder becomes small relative to the displacement length.

Differentiability is stronger than merely having partial derivatives at one point. The local linear model must work consistently for approaches from all directions. Continuous partial derivatives in a neighborhood are a useful sufficient condition. In physical calculations, track units: a temperature derivative per meter differs from one per centimeter. In optimization, rescaling a feature changes gradient coordinates and can change a naive algorithm's steps.

To find the rate along a unit direction $u$, take the dot product $\nabla f\cdot u$. The Cauchy–Schwarz inequality shows that this is largest when $u$ points along the gradient, provided the gradient is nonzero and ordinary Euclidean distance defines a unit step. The opposite direction is steepest descent locally. A direction perpendicular to the gradient has no first-order change, although a curved function can still change at second order.

The matrix of second partial derivatives is the Hessian. For a twice continuously differentiable scalar function, it describes the leading change in the gradient and the curvature of the local quadratic approximation. A positive-definite Hessian at a stationary point establishes a strict local minimum. A zero gradient alone does not: a saddle can be stationary while increasing along one direction and decreasing along another.

For a vector-valued function, arrange output partial derivatives in rows and input coordinates in columns. The resulting Jacobian maps a small input displacement into a predicted output displacement. Its dimensions reveal whether a proposed calculation makes sense. A function from three inputs to two outputs has a two-by-three Jacobian; multiplying it by a three-component displacement produces the required two-component prediction.

The chain rule composes these local maps: the Jacobian of an outer function multiplies the Jacobian of the inner function. This order is the same as matrix composition. Backpropagation later exploits this structure efficiently, but automatic differentiation does not establish that a model is smooth everywhere or that optimization will succeed. Here, differentiate a concrete expression, predict a small change, and compare against the exact value to learn what the derivative is promising.