Hamilton–Jacobi–Bellman equations

For fixed final time T, let V(t,x) be the infimum of terminal cost plus accumulated running cost from state x at time t. Assume deterministic dynamics ẋ=f(s,x,u), known state, admissible measurable controls and trajectories existing over the full remaining interval. The principle of optimality says that splitting the interval at t+h leaves an initial short running cost plus V(t+h,x(t+h)), then takes the infimum over the first control segment. It does not assume an open-loop command chosen now remains appropriate after an unmodeled disturbance.

For a continuously differentiable V and sufficiently regular f and L, expand this identity over a short interval and divide by h. The result is V_t plus the infimum of L+∇Vᵀf equal to zero, with terminal condition V(T,x)=φ(x). Compact control sets and continuity give a minimum; for unbounded controls one must verify attainment separately, as coercive quadratic control costs do in the worked example. The equation is nonlinear through optimization over u even when the original state dynamics is linear.

Suppose a continuously differentiable candidate W satisfies this equation and the terminal condition on a domain containing every relevant admissible trajectory. Along any admissible control, the chain rule gives dW/dt+L≥0, because that control’s Hamiltonian cannot be below the infimum. Integrating yields total cost at least W at the initial pair. This proves a global lower bound, without assuming that the chosen trajectory itself satisfies Pontryagin’s necessary conditions. The differentiability and integrability hypotheses justify the chain rule and time integration rather than merely formal algebra.

If a measurable feedback attains the infimum everywhere along its resulting trajectory, and that trajectory exists to T and satisfies all constraints, the inequality becomes equality. The feedback then achieves the lower bound and W is the value there. A minimizing expression that produces a nonexistent, exploding or inadmissible trajectory is not a verification. This distinction matters near state constraints or discontinuous feedback boundaries. An algebraic minimizer of the Hamiltonian is only one ingredient; its induced closed-loop behavior must still be checked.

Optimal costs can develop corners when different strategies exchange preference or when the target is reached from opposite directions. A minimum-time integrator with bounded speed has value proportional to absolute distance, which is not differentiable at the target. Demanding a classical derivative there would wrongly reject the correct value. Viscosity solutions replace unavailable derivatives by smooth test functions touching from appropriate sides. Their comparison theory can provide uniqueness under suitable continuity, growth and boundary assumptions; no unrestricted uniqueness theorem for arbitrary nonlinear control problems is being claimed here.

Where V is smooth, its state gradient often equals the normal costate along an optimal trajectory, connecting HJB feedback with Pontryagin’s principle. But a solution of one characteristic trajectory does not determine V globally; characteristics can cross and the value can remain nonsmooth. Numerically, solving a state-space HJB equation suffers dimensional growth, whereas trajectory methods avoid a full grid but provide less global information. Approximate learned value functions must be checked for residuals, boundary behavior and policy admissibility before their attractive plots are treated as certificates.