Multiple testing and false discovery rate

Among a fixed family of $m$ null hypotheses, let $R$ be the number rejected and $V$ the number of rejected true nulls. The false-discovery proportion divides the latter by the former, taking value zero if there are no rejections. The false-discovery rate is its expectation over the data-generating experiment. It is not generally the ratio of two expectations, and it is not the probability that a particular selected hypothesis is false. Those are different quantities requiring different models or conditioning.

If every hypothesis is null, every rejection is false. In that special case the false-discovery proportion is the indicator of at least one rejection, so false-discovery rate equals familywise error probability. Outside the global null the criteria differ: allowing a small expected fraction of false discoveries can permit many more true discoveries than controlling the chance of any false rejection. A stated scientific decision target should determine which criterion is appropriate, rather than treating every multiplicity correction as interchangeable.

Sort the p-values in nondecreasing order and fix a target level $q\in(0,1)$. Find the largest rank whose p-value is no larger than its rank times the target divided by the family size. Reject all hypotheses through that rank; if there is no qualifying rank, reject none. This is a step-up rule, so one must not stop at the first failed inequality. A later rank can qualify even if an earlier p-value exceeded its smaller individual threshold.

Suppose the p-values are mutually independent and each true-null p-value is valid, meaning its probability of falling below any threshold is at most that threshold. Then the rule controls FDR at most $q m_0/m$, hence at most $q$, where $m_0$ is the number of true nulls. Exact uniform null p-values are a familiar sufficient case; conservative valid p-values also give the inequality. Merely observing that p-values look small does not establish calibration. Approximate regression pivots require an additional argument before inheriting an exact finite-sample testing theorem.

Certain positive regression dependence conditions also support the unmodified rule, but arbitrary correlation is not the same condition. Under completely unrestricted dependence, valid marginal p-values suffice for the Benjamini–Yekutieli correction: replace the target by $q/H_m$, where $H_m=\sum_{i=1}^m1/i$. The harmonic factor can reduce power substantially. One should state the actual dependence theorem rather than assume that any positive-looking correlation or any two-sided construction qualifies for a more favorable result.

Filtering hypotheses using the same outcomes can change both the tested family and the conditional p-value law. An independence-preserving filter or a dedicated selection-aware method needs its own justification. Likewise, an FDR guarantee does not promise that a particular realized list contains at most the target fraction of false discoveries. That stronger realized-proportion or exceedance target requires different procedures and assumptions. Reporting the family, rule, target level, calibration, and dependence conditions makes the statistical claim assessable without implying knowledge of which individual discoveries are false.