Singular value decomposition

For a real $m$ by $n$ matrix, a full SVD writes the map as $U\Sigma V^T$. The columns of $V$ are orthonormal input directions and the columns of $U$ are orthonormal output directions. The rectangular diagonal matrix $\Sigma$ contains nonnegative singular values. Reading from right to left, first express the input in the $V$ basis, then stretch each coordinate, and finally express the output in the original coordinates using $U$.

Orthogonal factors can include reflections as well as rotations. Their role is to preserve Euclidean lengths while the singular values carry the stretches. This is why a negative matrix eigenvalue does not become a negative singular value: its sign can be carried by an orthogonal direction instead. The decomposition exists for every finite real matrix, although signs and bases inside repeated singular-value subspaces are not unique.

The matrix $A^TA$ is symmetric and positive semidefinite. Its orthonormal eigenvectors provide the right singular directions, and its eigenvalues are the squared singular values. Positive semidefinite means that $x^TA^TAx=\|Ax\|^2$ cannot be negative. This argument explains both the appearance of squares and the absence of negative singular values without relying on a decomposition recipe alone.

A very small singular value signals an input direction that barely affects the output. Attempting to invert that stretch magnifies measurement noise. The ratio of largest to smallest positive singular values describes conditioning on the identifiable subspace; for an invertible square matrix it gives the usual spectral condition number. Actual numerical software generally avoids explicitly forming $A^TA$ when accuracy matters, since doing so squares the condition number.

Order singular values from largest to smallest. Keeping the first $k$ terms gives a rank-at-most-$k$ matrix that minimizes Frobenius error among matrices of that rank bound. Its squared error is the sum of the omitted squared singular values. This is a precise guarantee about a chosen norm, not a guarantee that every task-relevant detail survives. A faint feature can be important even if it contributes little numerical energy.

For centered data, principal component analysis uses right singular vectors as directions of variation. Centering matters: otherwise a large average offset can dominate the decomposition. SVD also appears in quantum bipartite-state decompositions, but interpreting those coefficients requires normalized complex amplitudes and tensor-product structure. Here the achievement is narrower and concrete: we can name how much a finite real matrix stretches each orthogonal input direction and what a low-rank approximation discards.