Measure where and when a forecaster fails

Join each immutable issued prediction with the target version selected by the scoring policy. Define signed error consistently; using actual minus forecast makes positive bias indicate underprediction. Group by horizon before averaging because one-hour and twelve-hour forecasts face different information gaps. For interval targets, retain duration and aggregation type. A request-count error and an average-rate error cannot be combined merely because their numeric values happen to be similar.

Choose regimes that explain the deployment question: load band, weekday, outage recovery, input freshness, entity age, or a known policy transition. Operational routing requires regime features knowable at issue time; diagnostic analysis may additionally condition on realized load, but must label that hindsight distinction. Publish case counts and label-coverage fractions in every cell. A missing high-load label should reduce reported coverage rather than quietly disappear and make the remaining high-load forecasts look unusually accurate.

For a nominal central interval, record empirical coverage and interval width by horizon and regime. Coverage alone rewards arbitrarily wide intervals, while narrowness alone rewards overconfidence. An interval score combines width with a penalty for observations falling outside its bounds. Quantile or distributional scores add other useful views, but the chosen score must correspond to the output type actually served. Compare systems on the same cases and probability levels rather than matching only a label such as confidence interval.

Marginal coverage does not imply reliable coverage in every regime, nor does a finite observed coverage percentage prove a future guarantee. If consecutive forecasts share targets or operate during the same incident, their interval-hit indicators are dependent. An iid binomial standard error can then understate uncertainty. Use a justified time-block resampling design, an appropriate dependence model, or clearly limited descriptive reporting. Select blocks using the process and horizon overlap, and show sensitivity when the dependence scale is uncertain.

Report both the observed workload mixture and any fixed reference mixture used for comparison. A weighted regime average makes a release-to-release comparison less sensitive to a changing population, but only for the regimes represented by that weighting scheme. Retain unweighted cell results so the summary can be audited. Separate macro averages across entities from micro averages across forecast cases; a busy entity can dominate the latter while a fragile small entity disappears from view.

Set alerts with delayed truth in mind. Input-age or forecast-failure alerts can fire immediately, while accuracy alerts wait for mature outcomes. For model comparisons, pair predictions on identical issue cases before estimating score differences; treating two scores from the same weather front as independent measurements wastes useful pairing and can misstate uncertainty. Keep the final later-period evaluation protected from repeated tuning, and tie any operational promotion to both forecast quality and the actual reservation or staffing decision the forecast supports.