Smoothing splines and penalized functions

Take n≥2 pairwise distinct fixed design points in a bounded interval [a,b], with arbitrary real responses. Minimize mean squared residual plus $\lambda\int_a^b(f''(t))^2dt$ over the Sobolev space $W^{2,2}[a,b]$, where the second derivative is weak and square integrable. In one dimension this space has a continuously differentiable representative, so point evaluation is well defined. For lambda strictly positive, the distinct design points identify the affine nullspace and the minimizer is unique.

The variational solution is a natural cubic spline with knots at the distinct observed inputs, not one global cubic. It is piecewise cubic with matching first and second derivatives; natural boundary conditions remove unnecessary bending outside the data range. The finite representation is a theorem about this particular quadratic penalty and function space. Changing the penalty to absolute curvature or imposing inequality constraints can produce a different class of solutions, so the cubic conclusion should not be carried over unchanged.

A function has zero curvature penalty exactly when it is affine, up to the usual Sobolev representative. Increasing lambda therefore suppresses nonlinear components rather than shrinking the entire function toward zero. As lambda tends to infinity, the fitted values approach the least-squares affine fit. As lambda decreases to zero, the positive-lambda solutions approach the natural cubic interpolant at distinct inputs, the minimum-curvature member among interpolating functions. The unpenalized optimization at lambda zero itself has many function-valued minimizers.

In a suitable spline basis, the penalty is a positive semidefinite quadratic form. The coefficient normal equations involve the design Gram matrix plus n lambda times the penalty matrix under the average-loss convention. A full-rank evaluation basis makes this sum invertible for positive lambda. Predictions are linear in the responses only when lambda is treated as fixed. Selecting lambda from the responses makes the overall fitting rule adaptive and generally nonlinear, even though every candidate smoother has a matrix representation.

The trace of the fixed smoothing matrix is often called its effective degrees of freedom. It interpolates between highly flexible fitting and the two affine degrees of freedom, under the standard distinct-input cubic construction. It is not simply the count of basis coefficients, since shrinkage can leave many basis coordinates present with small influence. Leave-one-out cross-validation can exploit the diagonal smoother entries when the candidate smoothing parameter and fitting rule are held fixed for each calculation.

Under independent homoskedastic noise the covariance of fitted values is sigma squared times the smoothing matrix times its transpose. It is generally not sigma squared times the smoothing matrix, because a penalized smoother is not usually an orthogonal projection. Confidence intervals based only on this variance target the smoothed mean unless bias is negligible or separately controlled. A small cross-validation error therefore does not by itself certify a confidence band for the unsmoothed regression curve.