Gaussian regression and exact inference

Suppose the fixed design has rank p with $n>p$, and $\varepsilon$ is multivariate normal with mean zero and covariance $\sigma^2 I$, where σ² is positive. The errors are then independent Gaussian coordinates with common variance. Linear transformations preserve Gaussianity, so the least-squares coefficient vector is multivariate normal with its familiar mean and covariance. The residual vector is also Gaussian, but its coordinates generally are correlated and have unequal marginal variances because fitting removes model directions.

The fitted-error component is $H\varepsilon$ and the residual is $(I-H)\varepsilon$. Their covariance is zero because the projection matrices multiply to zero. Joint Gaussianity upgrades this zero covariance to independence. Choose an orthonormal basis for the residual subspace: there are n−p independent Gaussian coordinates of variance σ². The residual sum of squares divided by σ² is therefore a chi-squared variable with n−p degrees of freedom. This argument identifies both the distribution and its dimensional origin.

Define $s^2$ as residual sum of squares divided by n−p. It is unbiased for σ² under the correct spherical mean model, even without Gaussianity. Under the Gaussian model it is also independent of the coefficient estimator. For a nonzero coefficient contrast c, standardizing its estimation error by the true standard deviation gives a standard normal variable. Replacing σ with s divides that normal variable by the square root of an independent scaled chi-squared variable, producing a t distribution.

A confidence interval follows by retaining coefficient values whose t statistic lies between the chosen quantiles. Its confidence level refers to repeated intervals under the assumed model, not a posterior probability attached to a fixed coefficient. With very few residual degrees of freedom, estimating the noise scale is uncertain and t quantiles become large. Multiplying a normal critical value by a standard error can therefore substantially understate uncertainty in a small regression, even if the coefficient estimate itself looks stable.

At a fixed new row $x_*^{\mathsf T}$, the fitted conditional mean has variance $\sigma^2 x_*^{\mathsf T}(X^{\mathsf T}X)^{-1}x_*$. A new response also includes fresh observation noise. If that error is independent of the training errors and follows the same Gaussian variance model, prediction-error variance adds another σ². The resulting interval is wider than a confidence interval for the mean. Merely reusing the mean standard error would omit irreducible uncertainty in the new measurement.

Exactness also presumes that the design, target contrast, and model were specified independently of response-driven selection. Choosing predictors after inspecting many fits and then quoting an ordinary single-model t interval ignores selection. Residual checks can reveal contradictions with assumptions, but a quiet plot cannot prove them. For heteroskedasticity, dependence, non-Gaussian small samples, or a changing deployment population, alternative variance estimates or inferential arguments must be justified rather than retaining the exact label automatically.