Separate lead time from the wait for truth

Define whether the target is a value at an instant, a sum over an interval, an average rate, or a maximum. Predicting jobs arriving during 16:00–17:00 is different from predicting the queue size at 17:00. Set a convention for horizon, such as target-interval start minus forecast issue time, and carry interval endpoints explicitly. Comparing systems that use different endpoint conventions can make one appear more accurate simply because it sees an extra hour of information.

Separate the operational lead from the distance to the most recent usable measurement. If a forecast is issued at noon for an interval starting at 14:00, its lead is two hours even when the latest usable observation ended at 10:00. The model must bridge four hours from that measurement to the target start. Store the observed input age alongside the forecast lead so delayed pipelines are visible as a separate condition rather than silently reclassified as a harder horizon.

A training row represents a prediction that could have been issued at its historical origin. Its target may occur later, but the fitting procedure at cutoff C may use that row only if the target label is usable by C. This is different from constructing its feature snapshot: those features must respect the historical origin, not C. One cutoff governs whether the label is available for fitting; the earlier origin governs what the hypothetical predictor was allowed to observe.

For a vector of horizons, decide whether an example waits for every label or uses an explicit per-horizon availability mask. Waiting for all horizons produces a clean rectangular target matrix but discards recently matured short-horizon examples. Masking retains them while requiring a correct loss denominator and careful tracking of which horizons are represented. Neither choice permits treating an unobserved target as zero. Record these masks in training evidence because changing them can change the effective horizon weights.

A dashboard showing only labels that have arrived can preferentially report easy cases. For example, healthy jobs may finalize promptly while failed jobs require manual reconciliation. Measure label coverage by issue cohort, horizon, and relevant regime before interpreting accuracy. A score of ten percent error on forty percent of the intended cases is an incomplete result. Keep provisional and final scoring streams separate if the business needs fast feedback while also requiring a stable final benchmark.

Feedback delay also determines realistic adaptation speed. Retraining every fifteen minutes does not create fresh supervision when labels take a day to settle. New input statistics can still be monitored, but prediction error cannot be known before its outcome is observed and accepted. Design alerts for feature freshness and serving failures independently from delayed accuracy alerts. This avoids promising instantaneous detection of a forecasting failure whose consequences have not yet occurred or whose labels have not yet become usable.