BFGS updates and curvature conditions

Let s be the displacement between two successive iterates and y the corresponding gradient difference. Integrating the Hessian along their connecting segment shows that y is an average curvature action on s for a C² objective. A new approximation B is asked to satisfy Bs=y. In more than one dimension this secant equation does not determine every matrix entry. Symmetry, preservation of positive definiteness, and a controlled change from the previous approximation guide the update choice.

Use distinct notation for a Hessian approximation B and its inverse approximation H. The step usually solves Bp=−g; an inverse implementation instead multiplies p=−Hg. Confusing their update formulas produces a method that can look plausible in one dimension and fail in larger systems. Storing a dense matrix costs quadratically in dimension, while limited-memory BFGS keeps selected displacement-gradient pairs and applies an implicit inverse action. That memory reduction changes the representation and available history, not the meaning of s and y.

For a symmetric positive-definite B and a nonzero s, the Hessian-form BFGS update subtracts the old curvature contribution along Bs and adds one along y. The denominators require positive s-transpose-B-s and positive y-transpose-s. Under those conditions the resulting matrix remains symmetric positive definite and satisfies the secant equation. Thus a nonzero gradient produces a descent direction in the next step. The algebra relies on strict positivity; silently dividing by a tiny or negative curvature product defeats the guarantee.

Suppose an accepted step has s=αp with positive α and the direction is descending. The weak Wolfe curvature condition makes the gradient change projected on p positive enough that y-transpose-s is strictly positive. This connects line search to the matrix update. On a nonconvex objective a trial chosen without that safeguard can yield negative observed curvature. Implementations may skip an update, damp it, or modify the curvature pair, and should document which safeguard was used rather than claim unmodified BFGS behavior.

Positive definiteness of the approximation is an iteration-level safeguard, not a proof that the objective is convex or that the algorithm reaches a global minimum. Standard local superlinear BFGS results apply near a solution with positive-definite true Hessian, sufficiently regular curvature, suitable initial data, and compatible line search. They are not automatic on an arbitrary nonconvex loss. A useful diagnostic distinguishes the model matrix's eigenvalues from the true objective's curvature because the two can disagree deliberately.

Gradient noise is especially damaging when subtracting nearly equal gradients. A small displacement can make y mostly evaluation error, and a positive but tiny product can produce a huge update. Scale-aware curvature tests, careful stopping tolerances, and occasional restarts are practical responses. Verify a new implementation on a low-dimensional quadratic where exact gradient changes are known, then on a nonlinear example. Passing the quadratic check validates update arithmetic but cannot establish behavior on every noisy or ill-conditioned problem.