Proximal and projected methods

Euclidean projection finds the feasible point closest to a proposed vector. In finite dimensions, every point has a unique projection onto a nonempty closed convex set. Closedness prevents the nearest boundary point from being missing; convexity, together with strict convexity of squared distance, gives uniqueness. Projection onto an interval clips the proposal to its endpoints. Projection onto a complicated convex set may itself require significant computation, so it should not be treated as free.

Projected gradient descent first makes an unconstrained gradient proposal and then projects it. At a constrained optimum, this combined update can remain fixed even when the raw gradient is nonzero. Its displacement divided by step size defines a gradient mapping, a more suitable stationarity residual for this setting. Under convexity, a fixed point of the exact projected step satisfies the feasible first-order condition; merely having a small raw gradient is unnecessary at a boundary solution.

For a proper lower-semicontinuous convex function h and positive step size, the proximal operator minimizes h plus a squared distance cost. The quadratic makes the subproblem strongly convex and coercive enough for a unique solution in finite dimensions. Choosing h to be zero on a closed convex set and positive infinity outside it recovers projection. Extended values encode a hard constraint without pretending it is a differentiable finite penalty.

For $h(z)=\lambda|z|$, the scalar answer is soft thresholding: subtract the threshold from positive proposals, add it to negative proposals, and send proposals within the threshold interval exactly to zero. The kink at zero is handled by the subgradient interval, not by arbitrarily assigning one ordinary derivative. This explains why absolute-value regularization can produce exact sparse coordinates, whereas a smooth quadratic penalty generally shrinks them without setting them exactly to zero.

For an objective $f+h$, proximal gradient applies the proximal operator of h to a gradient proposal from f. When f is convex with L-Lipschitz gradient, h is proper closed convex, a minimizer exists, and the operations are exact, positive fixed steps no larger than $1/L$ support convergence guarantees. Those guarantees concern this composite iteration; they cannot be imported unchanged if the proximal subproblem is solved inaccurately or the smooth gradient is replaced by biased data.

The lab uses only h=0 and unconstrained unit-curvature f, so its proximal operation would be the identity. Compare steps 0.5 and 1, then calculate the nontrivial threshold manually in the example. No clipping or thresholding occurs in the animation. Keeping the decomposition explicit lets you determine which part is cheap, which part encodes a hard modeling choice, and which numerical residual actually measures progress toward the composite problem's solution.