Orthogonal series estimation

Let $\{\phi_j\}_{j\ge1}$ be a complete orthonormal basis of a specified real Hilbert space, such as $L^2[0,1]$ with Lebesgue measure. A square-integrable target f has coefficients $\theta_j=\langle f,\phi_j\rangle$, and Parseval identifies its squared norm with the sum of squared coefficients. Completeness matters: orthogonality alone does not guarantee that the expansion represents every target. The measure also matters, since changing from Lebesgue measure to a predictor distribution changes the meaning of the projection and loss.

In the Gaussian sequence model, observe each coefficient plus independent centered Gaussian noise of known variance sigma squared divided by n. Here n is an information-level parameter, not the number of coefficient coordinates retained. This experiment is directly connected to Gaussian white-noise observation of a function. Moving from ordinary regression to that experiment requires a design argument or an asymptotic equivalence theorem with its own smoothness and noise conditions; a visually similar basis expansion does not establish equivalence.

For a fixed cutoff m, retained coordinates contribute noise variance and omitted coordinates contribute squared approximation error. Orthogonality removes cross terms in the function norm, giving an exact risk of $m\sigma^2/n+\sum_{j>m}\theta_j^2$. The cutoff estimator is unbiased for the projected function, but generally biased for the full function. Even a perfectly calibrated confidence set for the retained coordinates can miss f if it ignores the omitted tail.

Smoothness becomes a bound on coefficient decay or on a weighted sum of coefficient squares. Such a bound controls the tail, but it does not identify every individual coefficient. Hard cutoff is only one strategy: shrinkage, thresholding and block procedures use different amounts of information about weak coordinates. If the cutoff is chosen from the same observations, the fixed-cutoff risk identity cannot simply be evaluated at the random cutoff and declared an unbiased risk formula. Selection couples the retained noise with the chosen model size.

For iid density observations, the average of a basis function evaluated at the observations is an unbiased estimate of its density coefficient. Its variance is the variance of that basis function under the unknown density divided by sample size, and distinct estimated coefficients need not be independent. A finite series can integrate correctly yet become negative. Taking a positive part and renormalizing produces a density when its positive integral is finite and nonzero, but changes both coefficients and the exact risk calculation.

For regression with irregular predictors, the empirical basis Gram matrix can differ substantially from the population identity. Weighted least squares or a basis orthogonal with respect to the design may be useful, but the design still determines what is well observed. Statistical regularization and deterministic Fourier convergence are related through approximation error, yet one does not replace the other. State the observation experiment, coefficient covariance and loss before importing a coefficient-risk formula into a function-fitting algorithm.