Singular values and low-rank approximation

For a real m-by-n matrix, an SVD writes A as UΣVᵀ with orthogonal factors and nonnegative singular values in descending order. Using the r positive singular values, A is a sum of rank-one terms σⱼuⱼvⱼᵀ. Each right singular vector describes an input direction, each left singular vector its output direction, and the singular value its gain. This is not generally an eigenvalue decomposition: input and output spaces may differ, and singular values remain nonnegative even when eigenvalues are negative or complex.

Small singular values indicate directions that have a weak forward effect and a potentially strong inverse response. A singular value of exactly zero records an exact null direction, whereas a small nonzero value may be indistinguishable from zero at the scale of measurement or arithmetic uncertainty. Numerical rank therefore requires a declared threshold. A gap between successive singular values can make a rank decision robust; a cluster near the threshold makes it sensitive. Individual singular vectors in a repeated singular-value subspace are not uniquely determined, although the relevant subspace can be meaningful.

The truncated SVD minimizes both spectral-norm error and Frobenius-norm error among matrices of rank at most k. The spectral error is the largest discarded singular value, because one omitted orthogonal direction realizes that gain. The Frobenius error is the square root of the sum of squared discarded singular values, accumulating all omitted gains. These are distinct objectives. A matrix with many modest discarded values can have a small worst-direction error yet a substantial total energy error. Rank ties may yield multiple optimal approximations.

The spectral lower bound has a useful geometric argument. Any candidate rank-k approximation must annihilate some nonzero vector in the span of the first k+1 right singular vectors, by a dimension count. On a unit vector in that intersection, the original matrix has norm at least σk+1, so no candidate can have smaller operator error. The truncated SVD reaches this bound. The Frobenius version uses orthogonal invariance and the corresponding squared-energy minimization; this explanation is a theorem outline, not a replacement for every detail of its full proof.

A full-rank least-squares inverse weights the observed component uⱼᵀb by 1/σⱼ in each retained right singular direction. Noise in a weak direction can therefore be amplified substantially. A truncated inverse sets coefficients associated with discarded singular values to zero. This can reduce noise amplification, but it also discards any real signal in those directions. The resulting estimate generally is not the full-rank least-squares minimizer. Its extra residual is a deliberate regularization cost, which should be evaluated against the noise model and desired output.

A stable SVD computation normally uses orthogonal reductions and a specialized bidiagonal problem rather than explicitly diagonalizing AᵀA, which squares singular values and can lose relative information in small ones. Even then, tiny singular values can be difficult to interpret relative to input uncertainty. Validate a selected rank by checking reconstruction error, sensitivity to the tolerance, and task-level performance. If a matrix is compressed for storage, a small norm error may be enough; if it is used for inversion, the discarded directions and subsequent inverse gains need separate analysis.