Risk-coverage curves and abstention

Let the selection rule g return one when the model acts and zero when it abstains. Coverage is the probability of acceptance. Selective risk is expected loss on accepted cases divided by that coverage, provided it is positive. For zero-one loss, empirical selective risk counts mistakes among accepted cases only. Dividing accepted mistakes by all arrivals measures a different quantity: accepted errors per arrival. Both can be useful, but exchanging them can make a low-coverage model look deceptively reliable.

When no cases are accepted, selective risk is undefined, not an observed zero-error performance result. Report zero coverage and the absence of accepted-case evidence. When few cases are accepted, quantify the uncertainty in their error rate rather than presenting a tiny numerator alone. Correlated records and selective label availability can reduce evidence further. Keep rejected-case labels where the observation process permits, because errors among rejected cases reveal whether the score ranks difficulty and whether the fallback improves outcomes.

Sweeping a confidence threshold creates operating points with different accepted populations. A useful score tends to reject errors before correct cases, but empirical risk need not decrease at every step. Tied scores require a fixed tie rule; random acceptance within a tie must use a declared independent mechanism and be evaluated as that policy. Compare curves under the same labels and workload mixture, and include uncertainty or paired analyses appropriate to the data rather than comparing isolated rounded percentages.

Choose a threshold with the application’s action costs, workload constraints, and acceptable risk stated in advance. In an idealized multiclass setting with exact conditional class probabilities, constant cost for any wrong accepted label, and a constant rejection cost, compare expected error cost with rejection cost. Real review can have variable delay and residual mistakes, so a rejection cost cannot silently stand in for a perfect human outcome. Evaluate the final chain of model, gate, fallback, and reviewer when reporting total system performance.

A curve built after exploring many methods on one labeled set is development evidence. Reserve separate data to validate the chosen score, threshold, tie policy, and fallback. Record the accepted count, error count, rejected count, and important slices, so the curve can be reconstructed. Thresholds can behave differently after a prevalence or workload change even if the score implementation is unchanged. A calibrated average confidence also does not ensure an optimal difficulty ranking for every subgroup or cost structure.

The probability lab illustrates accepted-case error counting only. Reset, set p to 0.02 and n to 10, predict expected accepted errors 0.2, then Step. Compare a fresh Reset with both p at 0.02 and n at 30 before Step: mean 0.6. The model assumes independent accepted cases with a known common error probability and displays exact masses. It does not choose a gate, estimate coverage, simulate abstention costs, or turn a small observed count into a risk guarantee.