Constrained optimization

For an unconstrained smooth minimum inside an open region, the gradient must vanish because every sufficiently small direction is allowed. Under an equality constraint $g(x)=c$, a feasible tangent direction $v$ instead satisfies $\nabla g\cdot v=0$. At a constrained stationary point, the objective has no first-order change in any such tangent direction. Its gradient can be nonzero while pointing entirely across the constraint surface.

With one smooth constraint and $\nabla g\ne0$, a vector perpendicular to all feasible tangent directions must be parallel to the constraint gradient. This gives the multiplier equation. If the constraint gradient vanishes, that reasoning breaks down, and singular points need direct examination. For multiple equalities, the corresponding gradients must be independent to use the same regularity argument cleanly.

Define a Lagrangian by subtracting a multiplier times the constraint residual. Stationarity with respect to the original variables gives the parallel-gradient equation, while stationarity with respect to the multiplier restores feasibility. The sign convention is arbitrary if used consistently. The multiplier is an auxiliary unknown, not an extra design freedom that permits violating the constraint. Solve the objective and feasibility equations together.

A multiplier can also measure sensitivity. Under suitable smoothness and an isolated smoothly varying optimum, the derivative of the optimal objective with respect to the constraint level equals the multiplier for this sign convention. This shadow-price interpretation is local and depends on the parameterization of the constraint. Multiplying the constraint equation by a constant rescales the multiplier even though it leaves the feasible set unchanged.

After solving the multiplier equations, compare candidates and inspect relevant boundaries or singular points. On a compact feasible set, a continuous objective attains a maximum and minimum, but those extrema can have different values. On an unbounded set, a stationary point need not be globally best and an optimum might not exist. Substituting the constraint into the objective is often the simplest independent check in a two-variable example.

Convexity can provide a stronger guarantee. A convex objective minimized over an affine set has no worse local minima hidden elsewhere; strict convexity gives uniqueness when a minimizer exists. Inequality constraints require additional logic, including active constraints and complementary slackness in the KKT framework. We have not derived that framework here. In machine learning, a nonconvex loss or a normalization constraint must be analyzed on its own terms rather than declared solved by writing a Lagrangian.