Exchangeability and coverage boundaries

A finite collection is exchangeable when its joint distribution is invariant to permutation of its members. Independent identically distributed observations satisfy this property, but exchangeability is broader than independence. For split conformal prediction, require calibration and test pairs to be exchangeable conditional on the information used to fix the score. Applying one fixed score to every pair preserves the needed symmetry. Training on the calibration labels, or treating one member differently after inspecting its outcome, can destroy that argument.

Randomly shuffling a dependent time series does not by itself make the future exchangeable with the past. A repeated customer, overlapping document window, device batch, or adaptive collection policy can alter the sampling unit. Specify whether the target is a new record, a new customer, or a future period, and construct the split accordingly. Clustered or temporal procedures can offer other guarantees under other assumptions, but they are not obtained by relabeling the ordinary independent-data split conformal theorem.

A population can meet a 90% average coverage target while failing badly on a small group. The average is a weighted sum of group coverages, so an easy majority can conceal a difficult minority. Conditioning on a realized feature vector is stronger still. Without restrictions on the distribution, nontrivial exact input-conditional coverage is generally unavailable in continuous regression. Restricted groups, smoothness, or other structure can support more specific procedures, but the procedure and its guarantee must be named.

The finite-sample bound also does not imply independent coverage events for many future points sharing one calibrated threshold. Conditional on that threshold, test behavior can vary, and averaging over calibration introduces dependence between events. A repeated-test binomial confidence analysis therefore needs its own independent sampling assumptions and conditioning statement. If the product requires a high-probability bound on the coverage of the realized calibration procedure, use an appropriate calibration-conditional analysis rather than reading that stronger claim into marginal validity.

A shift in class prevalence, language, acquisition process, or labeling policy can invalidate calibration-test exchangeability. Coverage monitoring should therefore record both group mix and group-specific outcomes, together with missing-label and delay mechanisms. Weighted conformal methods under covariate shift require additional information such as appropriate density ratios and a stable conditional label law. They do not license arbitrary unknown concept shift. Recalibration with fresh representative labels is an intervention whose costs and validation must be included in the release plan.

The Markov instrument illustrates temporal dependence rather than conformal calibration. Reset, set both transition probabilities p and q to 0.1, predict state-one mass 0.1 after one Step and 0.18 after two Steps. Compare a fresh Reset with both p and q at 0.5 before two Steps: state-one mass is 0.5. These are exact distributions from initial state zero, not sampled trajectories. They demonstrate changing state probabilities without testing whether any real calibration or future data are exchangeable.