Evaluate the complete forecasting procedure through time

At each origin, assemble the data and labels that would have been usable for that fit, apply transformations learned from its legal training window, and issue the intended horizons. An expanding window retains all earlier eligible examples; a sliding window retains a fixed recent span. Expanding history offers more observations but may mix obsolete regimes. Sliding history adapts its population at the cost of smaller samples. Choose between them using the expected process and historical evidence rather than treating either as universally correct.

The retraining schedule matters independently of the forecast schedule. A service may issue hourly forecasts with parameters refitted each night and state updated after every accepted observation. A backtest that refits the model at every hourly origin gives it a different computational budget and potentially fresher parameters. Reproduce the actual cadence, including training runtime if it delays artifact availability. A fit started at midnight cannot serve its new weights at midnight if it finishes forty minutes later.

A gap between training origins and test origins can enforce outcome maturity when labels extend into the future. Suppose a training example at i predicts y at i+h and that label is usable d periods later. At fit cutoff C, its origin must satisfy i+h+d≤C. Derive that inequality from the target and finalization definition. A library’s integer gap is a row exclusion mechanism; it does not inspect your target intervals, revision times, feature availability, or training completion schedule.

Do not automatically forbid a test lag from reading a measurement that also appears in training. Real forecasts normally consume recent observed history. Reusing that legal history is different from fitting parameters on future labels. Conversely, overlapping multi-step outcomes can make errors across origins strongly dependent even when no leakage occurs. Document the overlap and use an uncertainty procedure that respects temporal dependence. A split can be causally valid without producing independent evaluation cases.

Use earlier rolling folds to choose hyperparameters, feature policies, and retraining settings, then assess the chosen procedure on a later untouched period. Repeatedly modifying the system after inspecting final-period scores turns that period into another validation set. For a global model, split on actual time across entities unless the deployment question specifically concerns unseen entities. Otherwise contemporaneous future outcomes from one entity may influence a shared model used to score earlier decisions for another.

Save predictions before joining outcomes. Their records should include origin, target interval, horizon, model artifact, feature snapshot, and any fallback status. This allows the same issued forecast to be scored under provisional and final truth without regenerating predictions using newer data. Audit several folds by hand, especially the first row after a boundary and the first row after a retrain. These small checks often expose off-by-one errors that a large aggregate accuracy table hides.