Monitor the population that the decision policy lets you observe

Let S indicate whether a target label is observed. Accuracy conditional on S=1 need not equal accuracy over all eligible requests. If reviewers choose difficult cases using information absent from the stored features, observed labels can remain selective even after conditioning on those features. Logging a prediction score does not automatically record every factor that influenced review. Track the policy version, review assignment, observation probability when known, and unresolved cases. Compare label coverage across score bands and groups before interpreting any change in the labeled population's quality.

Some decisions do more than reveal a preexisting label. A loan's repayment under issuance is not an observed outcome for a rejected applicant, and blocking an action can prevent the event being predicted. Such questions require a specified potential outcome or policy-value estimand, consistency, relevant exchangeability assumptions, and action support. Inverse observation weights for passive missing labels do not manufacture counterfactual outcomes. A safe audit or randomized observation program may improve identification, but it must measure the intended target without silently changing its meaning or introducing a different intervention.

For a passive label-observation setting, suppose the target outcome exists for every eligible request and observing it does not alter it. If observation is conditionally independent of the loss given recorded X, and every relevant X has a known positive observation probability e(X), an inverse-probability total divided by the full eligible population size is unbiased for that population's mean loss under the observation design. The denominator is the number of eligible requests, not the number of observed labels. Unobserved outcomes contribute zero through S, without being declared correct or negative.

Small observation probabilities produce large weights and potentially large variance. If a policy never labels a region, its zero probability violates support and prevents this estimator from recovering that region's risk. Estimated probabilities introduce further modeling and estimation issues; clipping weights changes the estimator's bias properties. Even with exact probabilities, an unbiased finite-sample estimate can fall outside the loss's usual range because a rare observed case receives a large expansion weight. Report uncertainty and coverage, and avoid describing a single weighted estimate as the known full-population truth.

A deployed recommendation or pricing policy can change future user behavior, creating a distribution that depends on the policy. A decline in observed error after a rollout may reflect changed exposure, easier selected cases, or an actual improvement. Monitor exposure and policy assignment as well as labels, and retain a defensible comparison design where the deployment question needs causal evidence. Repeatedly retraining on policy-shaped data does not universally converge to a desirable system; convergence and quality results in performative prediction require particular structural assumptions.

The Markov instrument illustrates how transition rules change future state occupancy. Reset, set p=0.20 and q=0.20, predict state-one probability 0.32 after two transitions, then take two Steps. Separately Reset, set p=0.20 and q=0.80, predict probability 0.20 after two transitions, then take two Steps. Each Step adds half a second and one transition; the initial distribution is entirely state zero. These are exact probabilities, not sampled people or policy outcomes. The illustration contains no causal identification, review selection, or inverse weighting, so it only motivates tracking the mechanism generating future observations.