Least squares as orthogonal projection

Let $X$ be a fixed real matrix with $n$ rows and $p$ columns, and let $y$ contain the $n$ observed responses. A coefficient vector $b$ proposes the fitted vector $Xb$. Thus all possible fits form the column space of $X$, a closed subspace of finite-dimensional Euclidean space. Linear here means linear in coefficients: a column containing squared predictor values is allowed, even though the resulting curve is not a straight line in the predictor.

Minimizing the squared Euclidean distance to that column space gives a unique fitted vector. To see why the error must be perpendicular, perturb a proposed fit along any column combination. If the residual has a component in that direction, a sufficiently small movement reduces its length. At the minimizing fit every such directional derivative vanishes. This argument also explains why a residual plot can show structure even though residuals are already orthogonal to the columns used in fitting.

If the columns are linearly independent, the Gram matrix is invertible and the coefficients are unique. Define the hat matrix $H$ to map responses to fitted values. It is symmetric and idempotent: applying the projection twice changes nothing. The complementary projection $I-H$ extracts residuals. Consequently the fitted vector and residual are perpendicular, and their squared lengths add to the squared length of the response. Centered sums of squares require an intercept column or a separate centering argument.

Dependent columns change the coordinate description, not the existence of the nearest point. Several coefficient vectors can represent the same fitted vector, and writing an ordinary inverse then becomes invalid. A pseudoinverse supplies one convention for coefficients, while an orthonormal basis for the column space supplies the projection directly. Numerically, QR or singular-value decompositions are usually preferable to explicitly forming the inverse of the Gram matrix, whose conditioning squares that of a full-rank design.

After finding the projection, a statistical model may posit $Y=X\beta+\varepsilon$ with conditional mean-zero errors. That extra assumption makes full-rank least-squares coefficients unbiased. A covariance assumption is needed to calculate their variance, and Gaussian errors supply exact small-sample reference distributions. Neither a successful minimization nor a visually small residual establishes any of these assumptions. Predictor measurement error and omitted causes can change the scientific meaning of the fitted coefficients.

Orthogonality also has a useful invariance: replacing the columns by another basis of the same column space changes coefficient coordinates but leaves every fitted response and residual unchanged. By contrast, adding a genuinely new direction can improve the training fit without improving prediction. Keep three questions separate when interpreting software output: which space was permitted, which point was selected, and which probability assumptions justify uncertainty statements about it.