Local polynomial regression

Write $Y_i=m(X_i)+\varepsilon_i$ with conditional mean-zero errors. At a target x, form polynomial coordinates in $X_i-x$ and minimize weighted squared residuals using nonnegative kernel weights. The fitted intercept estimates m(x); the fitted slope in a degree-one model estimates its local derivative only under additional smoothness and shrinking-bandwidth conditions. Scaling polynomial coordinates by h can improve numerical conditioning, but changes the interpretation of the slope coefficient by the corresponding factor.

Condition on the observed predictors when calculating a finite-sample fit. The weighted Gram matrix must be invertible on the polynomial columns used; a local-linear fit needs at least two distinct predictor values with positive weights. A zero denominator or nearly singular matrix is a local-data problem, not proof that the underlying mean is undefined. Random-design asymptotics typically require a positive design density near the target and enough observations inside the shrinking window, in addition to assumptions on the errors and regression function.

For an invertible local-linear design, the estimate is a weighted sum $\sum_i\ell_i(x)Y_i$, where the prediction weights sum to one and their first centered moment is zero. Therefore constant and linear mean functions are reproduced exactly conditional on the predictors. These identities hold even near a boundary. In contrast, a local-constant kernel average generally has a nonzero first moment in a one-sided window and can inherit a first-order boundary bias from a sloping mean.

The original least-squares weights are nonnegative, but the derived prediction weights need not be. Negative prediction weights are how a boundary fit can extrapolate a line back to the target. Thus a local-linear prediction need not lie between the observed responses. With independent homoskedastic errors conditional on the predictors, its variance is sigma squared times the sum of squared prediction weights. Large positive and negative weights can increase that variance substantially; polynomial reproduction does not by itself guarantee a stable estimate.

A Taylor expansion explains why reproducing a line helps when the true regression curve has a continuous second derivative. With standard regular kernels, a positive regular design density and a nonsingular limiting local moment matrix, local-linear bias is of order h squared, including at an ordinary one-sided support boundary; variance has order one over nh in one dimension. At a boundary the constants differ from the interior constants. These are local asymptotic statements, not guarantees for an empty window or a discontinuous mean.

Higher-degree fits reproduce more polynomial terms but require more information and can be unstable if the local geometry is poor. Choosing degree and bandwidth by visual flexibility alone can turn a sparse neighborhood into an extrapolation problem. The target is the conditional mean, so residual prediction error also includes irreducible response noise. A confidence interval for the mean is consequently narrower than a prediction interval for a new individual response, and both must account for any smoothing bias that remains relevant at the selected bandwidth.