Evaluate mature prediction cohorts with knowledge-time cutoffs

Record prediction time, the end of the outcome-observation horizon, and the time a label became available to the monitoring system. A seven-day conversion target is unresolved before seven days even when no conversion has yet been logged. After the horizon, ingestion and reconciliation may add delay. Evaluation time is a separate cutoff that determines what the report could have known. A retrospective report using repaired labels can answer a useful question, but it must not be presented as the evidence available to an earlier operational decision.

Join outcomes using a stable prediction or decision identifier plus an explicit target definition. User identifiers alone can attach one later event to several unrelated predictions. Specify attribution rules for multiple exposures, duplicate labels, cancellations, and revised outcomes. Preserve the original model score and version instead of recomputing it with today's model. A monitor needs both cohort assignment by prediction time and outcome availability by knowledge time; grouping only by the label-arrival date can turn an upstream backlog release into a misleading performance trend.

An age buffer is a scheduling choice, not proof that every outcome arrived. Report the eligible prediction count, resolved count, unresolved count, and label-delay distribution by relevant slice. When delay bounds are absent, call a cohort provisionally mature under the chosen rule and show observed coverage. Estimate recent performance only with a justified missing-outcome model or a clearly conditional estimand. Simply counting every unresolved case as negative can bias the target; discarding unresolved cases can bias it differently when difficult or positive outcomes take longer to resolve.

Keep a report revision identifier and allow late arrivals to backfill the affected prediction cohort. Preserve both the historical as-of report and the later resolved report so incident investigators can reconstruct decisions. A completeness alarm should be distinct from a model-quality alarm: the immediate corrective action may be repairing a join or ingestion job. If an outcome genuinely has no finite resolution time, define a censored-data analysis or a fixed-horizon target rather than claiming that an arbitrarily old window is complete.

Fast input, score, schema, and service-health monitors can detect suspicious changes while outcomes mature. They should carry their own timestamps rather than borrowing the apparent freshness of a live dashboard for a week-old accuracy estimate. Present the latest evaluated prediction interval beside the computation time. For a retraining decision, compare equally aged cohorts or fully resolved cohorts and inspect delay changes before interpreting a quality trend. If an intervention changes outcome collection, even a stable model can appear to improve because the evaluated subset changed.

The queue instrument illustrates an ingestion-capacity mechanism without modeling outcome maturity. Reset, set arrival λ=3 and service μ=2, predict a backlog of six after two seconds, then take four Steps. Separately Reset, set λ=2 and μ=3, predict backlog two after two seconds, then take four Steps. Its fluid backlog starts at four and uses one fixed rate pair for the entire elapsed time. Changing a control pauses but retains time. Neither the backlog nor its separate M/M/1 readout measures label completeness or supplies a bound on real ingestion delay.