Inverse problems and regularization

For a forward operator A and data y, ask whether Ax=y has a solution, whether that solution is unique, and whether it changes continuously with y in specified norms. Hadamard well-posedness requires all three. A null space obstructs uniqueness; incompatible data obstruct existence; an unbounded inverse on the relevant range obstructs stable dependence. In finite-dimensional full-rank problems, inversion is continuous, yet a large condition number can still make it practically unreliable. Calling every ill-conditioned finite matrix strictly ill-posed confuses a numerical severity with the corresponding infinite-dimensional failure.

Singular vectors expose the mechanism. A right singular direction with singular value σ produces data scaled by σ. Naive inversion multiplies observed coefficients by 1/σ, amplifying noise when σ is small. A zero singular value is worse: the data contain no coefficient for that direction at all. More observations can help only if they add relevant information. If the forward model is systematically wrong, a tiny residual may merely mean that the algorithm has fitted noise and model error. Inspect data fidelity together with stability and the assumptions needed to interpret the recovered state.

Zero-order Tikhonov regularization minimizes the squared data residual plus α times the squared Euclidean norm of x, with α>0. Differentiating gives the normal equations (AᵀA+αI)x=Aᵀy in the real matrix setting. The added positive term makes this quadratic problem uniquely solvable even if A has a null space. It does not prove that the unique minimizer equals the true state. The penalty favors smaller norms, a modeling choice that may be inappropriate when another notion of structure should be preferred.

In a singular direction, the regularized coefficient is σ/(σ²+α) times the data coefficient. Relative to exact noiseless inversion, the recovered signal is multiplied by σ²/(σ²+α). Small σ directions are suppressed, reducing noise amplification while introducing bias. Large α suppresses more signal; tiny α approaches the unstable inverse in nonzero directions. Other penalties favor smoothness, sparsity, or a learned prior, but each changes the preferred solution and mathematical guarantees. A prior that produces attractive reconstructions can still invent features not established by the measurements.

A parameter-selection rule needs a noise model or some other validation principle. A discrepancy principle compares residual size with a justified noise bound; cross-validation assesses prediction on held-out data when the sampling assumptions permit it. Neither principle is a magic constant independent of units, operator scaling, or loss normalization. Convergence theorems typically let α decrease as the noise level decreases, balancing suppression of genuine signal against amplification of remaining noise. Sending α to zero while holding noisy data fixed is not generally a convergent recovery strategy.

Separate regularization from the iterative method used to compute its minimizer. A stable algorithm can accurately solve an inappropriate penalized objective; conversely a sound estimator can be poorly approximated by an unstable update. Early stopping can itself act as regularization in specific iterative settings, but this requires analyzing its spectral filter and stopping rule. The lab runs only gradient descent on the scalar quadratic x²/2. Adjust η to observe convergence and divergence, using it as a solver-stability comparison rather than a visualization of Tikhonov filters, hidden images, or an inverse reconstruction.