Neyman–Pearson testing and power

For simple hypotheses, both $P_0$ and $P_1$ are completely specified probability laws. A test function $\phi(x)\in[0,1]$ is the probability of rejecting after observing $x$; values strictly between zero and one represent a separate random decision. Its size here is $E_0\phi$, and its power against the specified alternative is $E_1\phi$. A level-$\alpha$ rule satisfies size at most $\alpha$. For a composite null, level requires control at every law in that null, a different optimization problem.

The two errors play different roles: false rejection is constrained, and failure to reject under the alternative is reduced by maximizing power. Neither probability is the chance that the null is true after observing the data. A failure to reject may reflect low power rather than agreement with the null. Planning sample size therefore requires specifying alternatives or effect sizes worth detecting. An error budget alone does not tell you whether an experiment can reliably resolve the scientific question.

Assume the two laws have densities with respect to a common measure. Choose a nonnegative threshold $c$ and let $\phi^*$ reject where $f_1>cf_0$, accept where $f_1<cf_0$, and randomize as needed on equality so that its size equals $\alpha$. Outcomes possible only under the alternative can be rejected without spending null probability. The Neyman–Pearson lemma says this rule has at least as much power as every level-$\alpha$ competitor. This is a finite-sample result with no normal approximation.

The proof is an ordering argument. At each outcome, $(\phi^*-\phi)(f_1-cf_0)$ is nonnegative by construction. Integrating shows that the power difference is at least $c$ times the size difference, which is nonnegative because the chosen rule uses the full budget and the competitor cannot exceed it. This demonstrates optimality for the specified simple alternative. A single rule need not be most powerful for every alternative in a two-sided composite problem, where opposite changes can favor opposite tails.

For a binomial count, null tail probabilities jump in discrete increments. A deterministic cutoff may use less than the available error budget. Randomizing at one boundary count fills the gap exactly: divide the remaining budget by the null probability of that count. This is a mathematically valid decision rule, though some applications prefer a conservative nonrandomized test for reproducibility. The chosen convention should be specified in advance rather than changed after seeing the outcome.

A p-value is a calibrated tail measure derived from a specified test ordering, not the probability of the null and not generally the probability of the observed outcome alone. Discrete p-values are often conservative under the null. Changing the sampling rule, selecting the best-looking test after inspection, or repeatedly testing until a rejection can change the false-rejection probability. The finite-sample calculations below assume the declared fixed sample size, iid Bernoulli model, and one prespecified test.