Calibration by class and subpopulation

For top-label calibration, confidence is the largest predicted class probability and the outcome is whether the chosen label is correct. Classwise calibration instead compares the probability assigned to class j with the indicator that the true label is j, including cases where j was not predicted. A model can satisfy one condition without satisfying the other. Reliability plots must state which outcome they use; a plot titled calibration with no target cannot establish the property needed by a downstream decision.

Adding group membership to the conditioning event is a stronger requirement than matching an aggregate curve. Groups may be languages, acquisition devices, workload types, or overlapping intersections selected for a task-specific reason. Membership information and labels must have clear provenance. Missing labels, delayed outcomes, or noisy annotations can distort apparent group gaps, especially if the observation process differs between accepted and rejected cases. Measure support and label quality before interpreting the gap as solely a model probability error.

Suppose every case has the same reported confidence but two equal-sized groups have different correctness rates. Their aggregate correctness can equal the common confidence exactly, while both groups have substantial gaps of opposite signs. Computing the absolute aggregate gap after pooling differs from computing an average of absolute group gaps. This is an arithmetic cancellation, not a paradox, and it persists even if the grouped labels are perfectly reliable and the sampling noise is small.

Binned expected calibration error also changes with the binning scheme, sample allocation, and definition of confidence. Report bin edges, counts, mean scores, empirical frequencies, and uncertainty alongside the scalar summary. Adaptive bins and many searched intersections create additional selection effects. Use development data to discover candidate failure slices and separate data to confirm them, or apply an analysis that accounts for the search. Multicalibration studies richer collections of subpopulations; its guarantees are specific to the collection, tolerances, and assumptions actually used.

Do not interpret calibration as a complete fairness or utility criterion. Two calibrated groups can still have different error rates or encounter different consequences from the same action. Conversely, changing thresholds to meet a policy objective changes which cases receive decisions and can alter the operational distribution. Evaluate the policy’s relevant losses, coverage, and delay by group, retaining the original probability evaluation. When data are scarce, report uncertainty and unresolved support rather than silently treating an empty slice as calibrated.

The sampling lab illustrates count uncertainty under a declared common rate, not group calibration fitting. Reset, set p to 0.8 and n to 20, predict sample-mean variance 0.008, then Step. Compare a fresh Reset with both p at 0.8 and n at 100 before Step: variance 0.0016. The exact Bernoulli sampling distribution becomes narrower under independence. It does not identify real groups, discover biased labels, or prove that the supplied probability equals any deployment group’s correctness rate.