Wait for comparable outcome windows or model censoring explicitly

Choose a time origin such as first eligible assignment or a valid intervention trigger, consistent with the estimand. Specify an H-day horizon and a deterministic rule for which events qualify. Repeated recommendations or exposures can compete to claim the same purchase, so record an attribution rule and deduplicate the underlying outcome. Changing from first-touch to last-touch attribution after seeing results can move credit without changing user behavior, creating a misleading model effect.

An outcome observed within the horizon can establish a positive binary label early. The absence of an observed event before H days does not yet establish a negative. A fully mature observation also needs the relevant telemetry to have arrived and passed processing checks; biological or behavioral delay and ingestion lag are different mechanisms. Track event timestamps and ingestion timestamps so the system can distinguish late behavior from late reporting.

One simple design stops enrolling before the analysis cutoff and waits until every enrolled unit has the full horizon, with additional allowance for known logging lag. Another analysis restricts both arms to units enrolled before a common maturity boundary. That preserves comparable follow-up but targets the older enrollment cohort. If traffic composition or the treatment effect changes over time, it should not automatically be presented as the result for all recently enrolled users.

More advanced methods use survival or delayed-feedback models to incorporate censored observations. They require assumptions about censoring, delay distributions, and how treatment affects event timing. A model trained on control delays may be wrong for a treatment that accelerates purchases without changing eventual conversion. Inverse censoring weights require adequate support and a correct observation model; a guessed correction factor cannot turn incomplete labels into complete experimental evidence.

A durable outcome table should identify the assignment or valid trigger, original event, model version, horizon, attribution rule, and data cutoff. Late-arriving events can revise preliminary figures, so publish maturity and completeness indicators rather than silently overwriting an apparently final result. If a metric counts multiple events instead of a binary outcome, define the aggregation and repeat-event handling explicitly; the same censoring concern applies to the unobserved tail.

Test the pipeline with an event just before the horizon, one just after it, a duplicate event, and an event ingested after the reporting cutoff. Compare treatment and control follow-up-age distributions and missing-event rates. A short-term gain can reflect shifted timing rather than greater eventual value, so measure the horizon relevant to the decision. Operational rollback does not erase outcomes that will arrive later from units already exposed, and their attribution needs to remain intact.