Posterior predictive checking

A posterior predictive draw first samples $\theta$ from its posterior and then samples replicated observations from the sampling distribution at that parameter. For a replication of the original experiment, keep the same design, exposure, or predictors unless their generation is explicitly part of the model. Replicating a new group in a hierarchy may require drawing a new group effect, whereas replicating measurements from an existing group conditions on that group's inferred effect. Those are different questions with different predictive distributions.

The predictive law averages conditional distributions; it is not generally the conditional distribution at an averaged parameter. When the second moments are finite, total predictive variance includes average observation variance plus variance of the conditional observation mean. This separates irreducible sampling noise from parameter uncertainty. Conditional independence can disappear after integration: several future observations sharing one unknown parameter become dependent in the predictive distribution. Retaining a common parameter draw for an entire replicated data set preserves that dependence; independently redrawing the parameter for every observation changes the experiment.

A discrepancy should address a feature that matters and that the model is not guaranteed to match. A fitted location parameter often reproduces the sample mean, so checking only that mean may miss heavy tails or clustering. For counts, examine both dispersion and zeros; for time series, examine runs or residual autocorrelation; for grouped data, examine differences at the relevant group level. Plot multiple replicated data sets beside the observations before reducing every check to one scalar, since distinct failures can cancel in an aggregate summary.

A posterior predictive tail probability can summarize how often a replicated statistic exceeds its observed value. Because the observations help fit the posterior and also supply the comparison statistic, these values are not generally uniform under repeated data generation from a fixed model. They are not ordinary significance-test p-values and are not posterior probabilities that the model is true. An extreme value can reveal a mismatch; a moderate value says only that this particular discrepancy did not reveal one. Discrete statistics also introduce ties whose treatment should be stated.

An in-sample predictive check diagnoses whether the fitted model can reproduce features of the data used to fit it. Held-out prediction instead evaluates predictions for observations withheld during fitting, respecting any grouping or time ordering needed to avoid leakage. A model can pass a narrow in-sample check yet predict poorly on a new population. Conversely an unusual observed feature can be scientifically useful even when an overall held-out score is acceptable. Neither comparison replaces examining the assumptions that make the target population relevant.

Simulation-based calibration examines computation under the assumed generative model. Repeatedly draw parameters from the prior, simulate data, fit the model, and compare each generating parameter with its posterior draws using appropriately constructed ranks. Under correct computation and the required sampling conditions those ranks have a known reference behavior. This can detect implementation errors or biased approximation, but it cannot establish that the assumed model describes the real world. Predictive criticism, held-out assessment, and computational calibration complement one another because their conditioning and failure targets differ.