Iterative regularization and Landweber iteration

For the objective ||Ax−yδ||²/2, the gradient is A*(Ax−yδ). Subtracting μ times that gradient gives the Landweber update. The step size must obey 0<μ<2/||A||² for the usual noiseless convergence theorem. At the upper endpoint the strongest mode may oscillate without damping; above it a mode can diverge. A step size that works for one normalized matrix need not work after rescaling the forward operator. Computing an adjoint consistently with the chosen inner products is also part of implementing the stated method.

With exact data y in the range and x₀=0, the iterates converge strongly to the minimum-norm solution. A nonzero initial nullspace component is never changed by A*(y−Ax), so it persists forever. The zero start is therefore a selection convention, not a harmless detail in a noninjective problem. This convergence theorem concerns exact consistent data and unlimited iterations; it does not recommend unlimited fitting of a fixed noisy observation.

Starting from zero, the singular-direction data multiplier after k steps is [1−(1−μσ²)^k]/σ. For exact object data, the resolution factor is the numerator. When μσ² is small, the initial numerator is approximately kμσ², so weak directions are delayed rather than inverted immediately. Increasing k gradually removes this suppression. This is the iterative counterpart of decreasing a Tikhonov parameter, though the two filter shapes are not identical and need not give the same reconstruction.

For the convenient stricter range 0<μ≤1/||A||², the numerator lies between zero and one and can be bounded by the smaller of one and kμσ². Dividing by σ then bounds the data-to-object gain by √(μk). Thus a deterministic perturbation of norm at most δ produces iterate error at most δ√(μk) relative to the exact-data iterate. The bound explains why increasing the iteration count can eventually overwhelm the decaying approximation bias. It is a worst-case upper bound, not an exact noise trajectory.

A sufficient a priori stopping rule takes k(δ)→∞ and δ√k(δ)→0 for fixed admissible μ. The first limit removes exact-data iteration bias; the second controls noise. A discrepancy rule can instead stop when the residual first falls below a safety multiple of a valid noise bound. These choices must be analyzed with their own hypotheses and cannot be replaced by a universal number of iterations independent of resolution and observation quality. Source assumptions can provide sharper rates and work estimates.

Semiconvergence means that iterates initially improve as useful components are recovered, then worsen as noise is fitted. A decreasing least-squares objective does not contradict worsening object error because those quantities live in different spaces and refer to different targets. In a finite injective problem, full convergence may simply recover the unique noisy inverse, which can still be far from truth. Monitoring a reconstruction's appearance or training residual alone is therefore insufficient to establish an appropriate stopping time.