Ridge regression and bias–variance tradeoffs

Use the objective squared residual norm plus $\lambda\|b\|^2$, with λ positive. Its gradient gives the modified normal equations. The added multiple of identity makes the matrix positive definite even if the design is rank deficient, so the minimizer is unique. This convention uses an unnormalized residual sum; software minimizing an average loss may report a penalty parameter differing by a factor of n. Compare objectives before comparing numerical tuning parameters.

The penalty treats the chosen coefficient coordinates as meaningful. Rescaling a predictor changes the cost of using its coefficient and generally changes a ridge fit unless the penalty is transformed too. Centering and standardizing predictors are therefore modeling choices that should be learned from training data and carried into validation and deployment. Many implementations leave the intercept unpenalized. The formulas here penalize every displayed coordinate; an intercept exemption requires replacing identity by an appropriate penalty matrix or centering it out first.

Write a compact singular-value decomposition $X=U\operatorname{diag}(d_j)V^{\mathsf T}$ with positive singular values. In the corresponding coefficient directions, ordinary least squares divides the response coordinate by dⱼ. Ridge replaces that multiplier with $d_j/(d_j^2+\lambda)$, shrinking the identifiable least-squares coordinate by $d_j^2/(d_j^2+\lambda)$. Small singular directions are shrunk most heavily. Nullspace directions receive zero under this penalty convention, a chosen extrapolation rule rather than newly recovered information.

Under a correct mean model with spherical error covariance, that same multiplier makes the bias and variance transparent. If θⱼ is the true coefficient in a singular-vector direction, the expected estimate is the shrinkage factor times θⱼ. Its variance is $\sigma^2d_j^2/(d_j^2+\lambda)^2$. Squared bias and variance add to mean squared error in that direction. Reducing the variance cannot by itself establish improvement, because the squared bias also depends on the unknown signal.

Unpenalized least squares already minimizes training residual sum of squares, so ridge cannot improve that particular score on the same fixed dataset. Its motivation is better risk on a specified future target or more stable interpretation under a justified penalty. Coefficient mean squared error and prediction error weight directions differently: predicting Xβ downweights weak singular directions, while extrapolating to new predictor rows can emphasize them. A statement about one risk does not automatically establish another.

Validation estimates performance without knowing the true coefficients, provided splits respect the deployment sampling structure. All preprocessing and penalty selection belong inside the training portion of each split; time dependence or grouped measurements may require structured rather than random splits. Reusing the final test set to choose λ makes it part of training. Ridge also cannot restore a missing conditional-mean assumption or identify causal effects from correlated predictors. Its uniqueness resolves an optimization ambiguity, not every scientific ambiguity in the data.