Selective prediction threshold validation

Policy selection includes more than the numeric threshold. The model checkpoint, score transformation, calibration fit, tie rule, allowed input types, subgroup routing, and overload fallback all determine which cases are accepted. Choose these on development and designated selection data. Keep final evaluation labels inaccessible until the policy and release criterion are fixed. If evaluation feedback leads to a change, that feedback has become development evidence; a revised policy needs fresh final evidence or a prespecified adaptive-valid analysis.

Create splits at the independent target unit, such as customer or source document, rather than randomly dividing near-duplicate records. Match the intended workload or explicitly weight the evaluation to it with appropriate uncertainty accounting. Report accepted support by important slices because an overall selective error estimate may conceal near-zero coverage for a group. Human review outcomes also need an observation plan that avoids measuring only the cases for which labels were easiest to obtain.

Under a fixed deterministic selector with positive population coverage and iid labeled test pairs from a fixed population, conditional on a positive accepted count, accepted errors follow a binomial law with the population’s accepted-case error probability. This requires acceptance to depend on the fixed policy and input, not on inspecting test labels. For zero observed accepted errors, invert the probability of that event to obtain a one-sided exact binomial upper confidence limit. It can be surprisingly large when the accepted sample is small.

The confidence statement concerns repeated samples generated by the assumed experiment. It is not a posterior probability that this particular unknown risk lies below the computed number. Its zero-error expression applies to the observed-zero branch; a full binomial confidence procedure supplies the other branches. Correlation, label error, selection across many tested policies, and optional stopping can invalidate the simple iid fixed-sample interpretation. Use a matching analysis or redesign collection rather than treating additional decimal places as additional assurance.

Predeclare the acceptable risk, minimum coverage, review capacity, and slice-support requirements. A policy with zero errors among ten accepted cases may fail a 5% risk gate because its upper confidence limit is much larger. Searching many thresholds and reporting only the best unadjusted limit is not the same experiment as validating one frozen threshold. Separate selection data are straightforward; simultaneous bounds, sequential tests, or learn-then-test methods can support other workflows only when their actual assumptions and multiplicity handling are followed.

The probability lab shows why zero errors can occur at a nonzero rate. Reset, set p to 0.05 and n to 10, predict zero-error probability approximately 0.5987, then Step. Compare a fresh Reset with both p at 0.05 and n at 30 before Step: approximately 0.2146. It displays exact independent-binomial masses rather than sampled evidence. It does not choose a threshold, compute confidence bounds, model optional stopping, or establish that a real test population has the supplied error rate.