Confounding, nonstationarity and forecast validation

Let tau be the origin and h the horizon. The forecast for y at tau plus h must be a function of information available by tau. Training outcomes, transformations, estimated means, feature selection and hyperparameter decisions all belong inside that boundary. A centered moving average evaluated near the origin can use future observations. Scaling on the complete dataset can also leak later distributional information, even when the prediction equation itself only uses lagged features.

Future calendar variables may be known in advance, while future weather or prices generally require their own forecasts. Evaluating a regression with subsequently realized covariates measures an ex-post scenario rather than the ex-ante forecast that could actually have been issued. State smoothing within an already available training history is legitimate. Smoothing an origin's feature using observations after that origin is not. Availability timestamps, rather than column names, determine whether the evaluation reproduces the intended task.

Rolling-origin validation repeatedly trains on the past and evaluates a later target. An expanding window retains all available history; a fixed-length rolling window discards older observations. Either can be reasonable, but they define different procedures under regime changes. Choose the window, refitting frequency and tuning policy using a chronological validation stage, then preserve a final test period for evaluating the selected procedure. Repeatedly changing a model after inspecting that test turns the test into additional training information.

Nonstationarity means old errors may poorly represent the next regime. Common trends can create apparent predictive association, while an omitted variable can drive both series; forecast accuracy does not identify a causal intervention. Even with a stable model, losses from overlapping horizons share outcomes or shocks and are generally dependent. Thus an iid standard-error formula based only on the number of error rows can be misleading. Any confidence assessment needs assumptions and methods suitable for its temporal dependence and evaluation design.

Squared error targets the conditional mean when the relevant second moment is finite; absolute error targets a conditional median under a finite first moment. Compare methods on the same origins, horizons and outcome scale, and report a simple benchmark. Pooling different horizons without stating the weighting can hide poor long-range performance. Percentage losses have additional difficulties near zero, so a familiar metric is not automatically appropriate for every outcome.

Prediction intervals need more than nominal coverage. An extremely wide interval can cover nearly everything while providing little useful information. Evaluate coverage together with width or a proper interval score, and state which sources of uncertainty the interval includes. A conditional known-parameter model interval can miss parameter uncertainty or structural change. No retrospective validation guarantees future performance under arbitrary distribution shifts, but a documented chronological design makes its evidence and limitations inspectable.