Conformal score construction and quantiles

Start with a predictor and score function fixed without the calibration labels. Larger scores mean worse agreement between an input and a candidate outcome. For scalar regression, absolute residual is a simple score in outcome units; for finite-class classification, one minus the candidate-class probability is a dimensionless alternative. The classification probabilities need not themselves be calibrated for the rank argument. Their quality matters for whether the resulting sets are informative rather than merely valid and excessively large.

Use the same score calculation on calibration pairs and on every candidate outcome at prediction time. Fitting a score or choosing among many score definitions based on those calibration labels introduces adaptation not covered by the basic split procedure. Deduplicate and split at the unit that supports the intended exchangeability assumption before fitting. A set-valued prediction also needs an interface: an empty or very large set is possible for some constructions and should trigger a specified action rather than an undocumented conversion to a confident single label.

For n calibration scores and target error level alpha strictly between zero and one, compute k as the ceiling of (n+1)(1−alpha). If k is at most n, take the kth smallest score using one-based indexing. Otherwise set the threshold to positive infinity. This convention is equivalent to appending an infinite score before selecting the order statistic. At small alpha, a finite sample cannot support the requested nontrivial rank resolution, and the full-set answer expresses that limitation honestly.

Thresholding with less-than-or-equal includes ties and preserves the conservative lower coverage bound under exchangeability. With distinct exchangeable scores and k at most n, coverage is k/(n+1), averaged over calibration and the future point. Ties can make coverage larger, so do not claim a matching near-nominal upper bound without the needed assumptions or a specified randomized tie-breaking construction. Randomness used for ties must be independent and applied symmetrically; adding arbitrary noise after inspecting inconvenient cases is not an analyzed randomization scheme.

A constant or uninformative score can produce valid but uselessly large sets. Improve score design on separate development data, then recalibrate using untouched labels under the declared sampling assumptions. Record the score version, alpha, calibration IDs, index, threshold, and tie rule so the prediction set can be reproduced. Compare set sizes and error locations as well as overall coverage. A marginal coverage statement does not ensure that every input receives a useful set, nor that any particular realized calibration sample has exactly the target coverage.

The probability lab illustrates a different calculation: repeated errors at a supplied fixed rate. Reset, set p to 0.1 and n to 10, predict expected count one and zero-error probability 0.9¹⁰ ≈ 0.3487, then Step. Compare a fresh Reset with both p at 0.1 and n at 30 before Step: mean three and zero-error probability approximately 0.0424. It assumes independent Bernoulli errors and neither computes conformal scores nor proves coverage for a shared calibration set.