Sensitivity analysis and partial identification
Report what observed data determine under bounded outcomes and show how explicit extra assumptions narrow the answer.
On this page
The question for today
What you’ll understand
- Derive worst-case bounds for a binary-outcome ATE without exchangeability.
- Distinguish an identified set from a confidence interval.
- Translate a specified untreated-selection bias into an ATT sensitivity range.
A causal question can remain partly answerable when point-identifying assumptions are not credible. Partial identification describes the set of effects compatible with the observed distribution and stated restrictions. Sensitivity analysis instead varies particular untestable assumptions and records how conclusions change. Both approaches make the missing information explicit. A narrow sampling interval around a confounded contrast does not replace either task, because increasing sample size does not reveal counterfactual outcomes that the observed law never identifies.
Bound the missing potential-outcome means
Consider binary treatment and potential outcomes constrained to the interval zero through one. Assume consistency and no interference, with both observed treatment groups having positive probability, but do not assume exchangeability. Let p be the treatment prevalence and μ₁,μ₀ the observed arm means. The treated potential-outcome mean averages its known treated-group component and its unknown untreated-group component. Bound that unknown component between zero and one. Do the corresponding calculation for the untreated potential-outcome mean, whose treated-group component is missing.
Subtract the largest possible untreated mean from the smallest possible treated mean for the lower ATE bound; reverse those choices for the upper bound. The resulting interval has width one, lies within the logical effect range, and contains zero. It is sharp under these minimal restrictions because the unobserved potential outcomes can be chosen to attain the endpoints without changing the observed distribution. Its width represents ignorance about the counterfactual law, not imprecision from a small observed sample.
Additional assumptions narrow sets but must be named
The bounds are often called worst-case or no-assumptions bounds, but the latter phrase is shorthand: intervention definitions, consistency, outcome support, and the observed population are still specified. Mean independence of potential outcomes from treatment collapses the bounds to the observed mean difference. Monotone treatment response, requiring every unit’s treated outcome to be at least its untreated outcome, instead rules out negative effects while leaving room for partial identification. It is different from monotone selection, which compares baseline prospects across treatment groups.
An identified set presumes knowledge of the observed distribution. With sampled data, its endpoints must be estimated and their uncertainty handled. A plug-in interval between estimated endpoints is not automatically a confidence region for the effect or a simultaneous confidence region for the entire identified set. Those are distinct coverage goals. Statistical uncertainty can shrink as sampling improves while the population identified set remains wide, so a report should distinguish the two rather than calling both an error bar.
A targeted sensitivity parameter must match the estimand
For ATT define δ₀ as the difference between the treated group’s untreated potential-outcome mean and the observed untreated group’s mean. Add and subtract the first quantity in the observed mean contrast to obtain ATT equal to that contrast minus δ₀. Exchangeability sets this particular bias parameter to zero, while a scientifically chosen range expresses plausible departures. A restriction on δ₀ alone concerns ATT; it does not generally identify ATE, whose treated counterfactual outcome among controls also matters.
Choose the range before inspecting which values preserve a preferred conclusion, and check that it is compatible with the outcome bounds. If the observed control mean is μ₀, the treated group’s untreated mean must remain in [0,1], restricting δ₀ to [−μ₀,1−μ₀]. A sensitivity interval is conditional on that maintained range, rather than a posterior distribution or a frequentist confidence interval by itself. Similar trend-deviation parameters can be used for DID, but the parameter and target must be redefined for that design.
Work it through
Compute a sharp ATE set
Suppose treatment prevalence is one half, the treated observed outcome mean is 0.8, and the control mean is 0.4. Outcomes are binary and only consistency and no interference are maintained. Derive the ATE bounds.
The treated intervention mean has known component 0.5·0.8=0.4 and missing component between zero and 0.5, yielding [0.4,0.9].
The control intervention mean has known component 0.5·0.4=0.2 and missing component between zero and 0.5, yielding [0.2,0.7].
Subtract endpoints in the appropriate directions: the lower effect is 0.4−0.7=−0.3 and the upper is 0.9−0.2=0.7. Both can be attained by opposite assignments of the missing outcomes.
The observed mean difference 0.4 does not identify the ATE sign under these minimal conditions; the compatible set includes zero and negative values.
The living notebook
Explore the model
Interactive lab · sampling
More precision does not identify missing counterfactuals
Reset, set success probability 0.4 and sample size 10; predict sample-mean variance 0.4·0.6/10=0.024. Reset, set probability 0.4 and size 100; predict variance 0.0024. Inspect the exact binomial sample-mean distribution, whose playback highlights outcomes rather than drawing data. This illustrates sampling precision for an observed group only; the lab does not shrink a causal identified set. Changes retain time; Reset restores both controls and zero time.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forObserved mean uncertainty can shrink while the missing counterfactual component, and thus the identification interval, remains unresolved.
Pause & reason
Check your understanding
Your turn
Try it yourself
Observed treated and control means are 0.7 and 0.4. Define δ₀=E[Y(0)|A=1]−E[Y(0)|A=0] and assume δ₀ lies between 0.05 and 0.15. Compute the implied ATT range and check compatibility with binary outcome support.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Subtract the larger bias for the lower effect and the smaller bias for the upper effect; check the implied untreated mean among treated units.
Compare with the solution
The observed contrast is 0.3. ATT=0.3−δ₀ therefore ranges from 0.15 to 0.25. The treated group’s untreated mean would be 0.4+δ₀, ranging from 0.45 to 0.55, fully inside [0,1]. The sensitivity restriction is therefore support-compatible. The resulting interval is conditional on the chosen untestable bias range and does not include sampling uncertainty or automatically describe the ATE.
Keep the idea
Take it with you
- Bounded outcomes yield sharp ATE sets even without exchangeability.
- Identification uncertainty and sampling uncertainty are different and require different reporting.
- Sensitivity parameters must be defined for the exact estimand and checked against logical support.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.