Stochastic Hamilton–Jacobi–Bellman equations

For a fixed current action a, the diffusion generator acts on a smooth function through its gradient and Hessian. The covariance matrix is $\sigma\sigma^{\mathsf T}$, and its Hessian trace carries a factor one half. Itô's formula includes a stochastic integral as well. Under the moment and localization conditions needed to take expectations, this stochastic integral contributes no drift, while the quadratic variation remains through the second-order term.

Apply dynamic programming over a short time interval and expand the continuation value. If the value is continuously differentiable in time and twice in space, the leading-order cost balance produces the HJB equation with the terminal cost as its terminal condition. This calculation motivates a classical equation; it does not prove the value has these derivatives. Degenerate diffusion and switching minimizers can destroy smoothness, requiring the viscosity formulation developed later.

With running cost $\ell$, the classical equation is $V_t+\inf_a(\mathcal L^aV+\ell)=0$. The infimum rather than a supremum matches the convention of minimizing expected cost. Since the terminal condition is prescribed at the end of the horizon, this is a backward equation. Replacing time by remaining horizon changes the sign of the time derivative, which must be done consistently in any Riccati reduction.

For scalar dynamics with drift u, constant diffusion coefficient, and running cost $u^2/2$, minimize a quadratic expression in u. When the action set is the whole real line, completing the square gives the feedback $u=-V_x$. With bounded actions this becomes projection onto the action interval and changes the resulting equation outside the unconstrained region. Thus a formula derived under unconstrained quadratic effort cannot silently be reused for a constrained controller.

Let the terminal cost be $qx^2/2$ with positive q, and try $V(t,x)=P(t)x^2/2+c(t)$. Matching the coefficients of the square and the constant yields a Riccati equation and a scalar integral. The terminal conditions are P equal to q and c equal to zero. The finite-horizon Riccati solution stays positive and bounded, so the resulting linear feedback gives a well-posed state with finite moments.

Additive noise affects the constant c, while the feedback slope P is unchanged in this fully observed linear-quadratic problem. Omitting c gives the same action but an incorrect expected cost. This particular separation depends on the linear dynamics, quadratic objective, and uncontrolled additive diffusion; it is not a principle for arbitrary nonlinear stochastic control. The solved candidate must still be linked to the value by a verification argument with adequate integrability.