greatitself.
Continue
advanced29 min studyLesson 965 in Physics

Confidence coverage and interval construction

Construct repeated-sampling intervals and explain why boundaries and discreteness invalidate automatic symmetric approximations.

Before you begin
On this page

The question for today

What you’ll understand

  • Define coverage as a property of a complete interval procedure.
  • Invert sampling statements with explicit tails and confidence levels.
  • Recognize sparse-count and boundary failures of asymptotic intervals.

An interval is more than two numbers around a fitted value. Its interpretation depends on the rule that would produce it for every possible data set. In frequentist inference a physical parameter is held fixed while the data, and therefore the interval, vary across repeated experiments. Coverage describes that collection of possible intervals. We will begin with an exactly Gaussian measurement and then consider a zero-count observation, where a familiar square-root error estimate would misleadingly suggest certainty.

Define the ensemble before interpreting confidence

Let C(D) be the interval returned by an analysis applied to random data D. At a fixed true parameter theta, coverage is the probability that theta lies in C(D), computed using the experiment's sampling law. An exact nominal procedure has the target coverage throughout its specified parameter domain; a conservative one has at least that coverage. A particular realized interval either includes the fixed true value or does not. The frequentist definition does not assign posterior probability to that fixed value after observing the interval.

The relevant ensemble includes every repeated operation in the protocol. If calibration data are remeasured, they fluctuate too. If the analysis chooses cuts or decides between an upper limit and a two-sided interval based on the observed data, that choice is part of the procedure whose coverage must be assessed. Quoting the nominal level of a different fixed rule does not establish the adaptive procedure's coverage. Coverage can also depend on unknown nuisance parameters and should be checked over their relevant range.

Cov⁡(θ)=Pθ{θ∈C(D)},X∼N(μ,σ2),σ known:C(X)=[X−z1−α/2σ, X+z1−α/2σ]\operatorname{Cov}(\theta)=P_\theta\{\theta\in C(D)\},\qquad X\sim\mathcal N(\mu,\sigma^2),\quad \sigma\ \text{known}:\quad C(X)=[X-z_{1-\alpha/2}\sigma,\ X+z_{1-\alpha/2}\sigma]
The Gaussian interval has coverage 1-alpha for every unrestricted real mu under its exact known-sigma model. Here z is a standard-normal quantile.↔ Scroll wide equations horizontally.

Invert acceptance regions rather than guessing errors

A Neyman construction specifies, for each candidate parameter, an acceptance region of data values with probability at least the desired confidence level. Inverting these regions gives all candidate parameters whose regions contain the observed data. The ordering of outcomes and allocation of tail probabilities affect the resulting interval. They must be declared to make the procedure reproducible. There is generally more than one valid interval construction for the same sampling distribution, especially with discrete counts.

For a Poisson observation without background, a one-sided upper limit can be defined by the cumulative probability of observing no more than the actual count. At zero observed events this equation is particularly simple: the probability of zero is an exponential. The upper limit is positive, even though the plug-in variance estimate based on the observed count is zero. This limit concerns the mean of the original count model, so translating it into a physical rate requires the exposure and any known efficiency.

∑k=0nμUke−μUk!=α,n=0:μU=−ln⁡α,rU=μUTϵ(b=0)\sum_{k=0}^{n}\frac{\mu_U^k e^{-\mu_U}}{k!}=\alpha,\qquad n=0:\quad \mu_U=-\ln\alpha,\qquad r_U=\frac{\mu_U}{T\epsilon}\quad(b=0)
This defines the conventional one-sided Poisson upper-limit procedure with at least 1-alpha coverage because of discreteness. The zero-count closed form assumes no background and known exposure and efficiency.↔ Scroll wide equations horizontally.

Approximate intervals require diagnostic checks

A local quadratic log likelihood or an approximately normal estimator often motivates a symmetric interval or a likelihood-ratio threshold. Such asymptotics need identifiable parameters, appropriate interior true values, a regular sampling model and sufficient information. A nonnegative signal strength at zero is on a boundary; an unidentified peak location under a background-only model is another failure mode. Blindly using an interior chi-square calibration can then give incorrect tail probabilities or coverage. Sparse counts can also remain strongly asymmetric.

Exact discrete procedures often overcover because an acceptance probability cannot vary continuously when whole outcomes are added or removed. This is not the same as assigning a probability that the parameter is physically allowed. Conversely, a Bayesian interval can have useful coverage without being defined by coverage. Report the interval type, level, ordering, nuisance treatment and any simulated coverage checks. A standard deviation alone cannot substitute for these choices, and agreement between two interval methods in one example does not establish general equivalence.

Work it through

An exact known-resolution Gaussian interval

A single hypothetical energy measurement is 10 keV, with exactly known Gaussian standard deviation 2 keV and unrestricted mean mu. Use z0.975=1.96z_{0.975}=1.96 to construct a two-sided 95% confidence interval and state its interpretation.

  1. The pivot (X−μ)/σ(X-\mu)/\sigma is exactly standard normal under the specified model. Its probability of lying between minus 1.96 and plus 1.96 is approximately 0.95.

  2. Invert the pivot inequality to give μ∈[X−1.96σ,X+1.96σ]\mu\in[X-1.96\sigma,X+1.96\sigma]. The observed half-width is 1.96×2=3.92 keV1.96\times2=3.92\,\mathrm{keV}.

  3. The realized interval is [6.08,13.92] keV[6.08,13.92]\,\mathrm{keV}. Repeating this fixed procedure at any fixed mu gives approximately 95% inclusion; this statement does not assign a posterior probability to mu in this particular interval.

The result

The known-sigma Gaussian 95% interval is [6.08, 13.92] keV, with repeated-sampling coverage under the stated model.

The living notebook

Explore the model

Interactive lab

Interactive lab · sampling

Sampling variation shrinks with independent information

Reset and set BOTH Success probability p = 0.5 and Independent samples n = 10; predict sample-mean variance 0.025. For a fresh comparison, Reset and set p = 0.5 and n = 40; predict variance 0.00625.

Sampling variation shrinks with independent informationExact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Current readings: Mean sample proportion: 0.5; Sample-mean variance: 0.0125; Uniform CDF error bound: ≤ 0.1062. All parameters are adjustable with the labeled controls below.Exact distribution of the mean of 20 independent Bernoulli variables with success probability 0.5; playback highlights outcome 0.00.250.50.75100.0580.1150.1730.23Sample mean · k / nProbability massNormal bin masses are an approximationk = 0 · exact probability = 9.54e-7
Exact binomial massNormal interval massHighlighted outcome / mean
Mean sample proportion
0.5
Sample-mean variance
0.0125
Uniform CDF error bound
≤ 0.1062
t = 0 s

Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forThe instrument shows exact binomial sample-mean probabilities, a normal comparison and a CDF-error bound. It does not construct confidence intervals, measure coverage or resample experiments.

Pause & reason

Check your understanding

Which statement defines 95% coverage?

Your turn

Try it yourself

Observe zero Poisson events during 2 s with no background, unit efficiency and exactly known exposure. Find the conventional one-sided 95% upper limit on the rate. Use −ln⁡0.05≈2.996-\ln0.05\approx2.996.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Set the probability of zero events at the upper mean equal to 0.05, then convert that mean to a rate.

Compare with the solution

At the upper endpoint e−μU=0.05e^{-\mu_U}=0.05, hence μU≈2.996\mu_U\approx2.996. Dividing by two seconds gives rU≈1.498 s−1r_U\approx1.498\,\mathrm{s^{-1}}. The reported interval is from zero to this upper endpoint for this observation. It is a one-sided construction, not the upper endpoint of a two-sided equal-tail 95% interval. Unknown background, efficiency uncertainty or an adaptive stopping rule would require a revised construction and coverage assessment.

Keep the idea

Take it with you

  • Confidence refers to the repeated behavior of a specified procedure.
  • Tail allocation and discrete ordering affect interval endpoints.
  • Zero counts and physical boundaries require more than a symmetric plug-in error.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.