Confidence sets and test inversion

A confidence set $C(X)$ for $\theta$ has coverage at least $1-\alpha$ when $P_\theta\{\theta\in C(X)\}\ge1-\alpha$ for every parameter value in the model. The randomness is in the data and therefore the set. After observing a particular set, its inclusion of the fixed true parameter is already determined, though unknown to us. A Bayesian credible region answers a different conditional probability question under a prior and posterior. Equal numerical endpoints in a special model do not make the two guarantees identical.

Coverage is conditional on the sampling model, including dependence, stopping, missingness, and any selection rules used in the analysis. A nominal confidence level printed by software does not establish those assumptions. A procedure with valid marginal coverage can also cover poorly within some data-defined subgroups; the guarantee does not automatically hold after arbitrary conditioning on the observed width or a selection event. Report what model and repetitions the guarantee refers to, rather than treating a confidence percentage as a universal measure of truth.

Suppose for each candidate $\theta_0$ we have a measurable nonrandomized test $\phi_{\theta_0}(X)$ that rejects its point null with probability at most $\alpha$ when that value is true. Define the confidence set as all candidates not rejected. The event that the set misses the true parameter is exactly the event that the test indexed by the true parameter rejects. Its probability is at most $\alpha$, proving coverage. No correction for testing every candidate is needed for this specific coverage statement: only the test indexed by the true value controls the miss event.

Conversely, rejecting a candidate exactly when it lies outside a valid confidence set produces a level test. For a target function with nuisance parameters, the candidate-null test must control rejection for all parameters sharing that target value. Randomized tests require explicitly incorporating the auxiliary random variable if an exact inversion is intended; simply retaining partially rejected values need not reproduce the same exact size. Set measurability is also assumed so that the coverage events have defined probabilities.

For iid normal observations with known variance, the standardized sample mean has an exact standard-normal distribution. Inverting its two-sided test yields an exact-coverage interval using the corresponding normal quantile. If the normal sampling law is replaced by a general finite-variance iid law, a central limit theorem may justify an asymptotic interval, but not exact finite-sample coverage. Replacing a known standard deviation with an estimate also changes the argument; the exact Student interval needs the iid normal model and its independent mean-variance structure.

For binomial data, exact tail inversion generally gives conservative coverage because count probabilities are discrete. A one-sided upper bound after observing zero successes solves $(1-p_U)^n=\alpha$, with endpoint conventions for other observations. It stays positive even though the observed proportion is zero. By contrast, the naive Wald interval using the estimated variance can collapse to a point at an endpoint and badly undercover. Confidence construction is therefore a calibration problem, not merely attaching two standard errors to every estimate.