Goodness of fit and posterior predictive checks

Suppose independent Gaussian observations have known standard deviations and a fully specified mean vector. The sum of squared standardized residuals has a chi-square distribution with the number of observations as its degrees of freedom. If instead linear mean parameters are fitted from the same data with full-rank design and known covariance, the minimized quadratic residual has fewer degrees of freedom. This exact result depends on the linear Gaussian model; an arbitrary nonlinear fit or estimated noise scale needs additional justification.

For a discrepancy where larger values indicate worse fit, the frequentist p-value is the probability under the specified model of a discrepancy at least as large as observed. It is not the probability that the model is true, nor the probability that the data arose by chance. Discrete discrepancies have discrete tail probabilities, and nuisance parameters can make the reference law uncertain. A simulation calibration should reproduce fitting and selection as well as generate events; otherwise it may compare unlike procedures.

A Bayesian model check can generate a parameter from the posterior and then generate replicated data from the corresponding sampling model. This gives the posterior predictive distribution. The replicated experiment must match the aspect being checked: a new measurement, a repeated count at the same exposure, or a conditional shape sample are different targets. Holding a fitted parameter fixed instead produces a plug-in prediction and can understate predictive variation. The posterior predictive mixture retains uncertainty about the parameters within the chosen model.

Compare a chosen discrepancy of replicated and observed data. A discrepancy may depend on the unknown parameter, in which case both sides must use the same sampled parameter in the comparison. Posterior predictive tail areas use observed data both to fit and to check, so their null distribution is generally not uniform. They should not automatically be interpreted using the same repeated-sampling thresholds as a calibrated frequentist p-value. A prediction of a particular future count is also not itself a discrepancy-based p-value.

A total residual sum can miss time drift, asymmetric tails or a narrow local excess. Inspect residuals versus acquisition time, detector region and predicted scale when those comparisons are physically relevant. The covariance used to standardize residuals matters: correlated residuals cannot generally be treated as independent pulls. Quantile checks, held-out predictions and control-region comparisons can reveal different failures. Selection of the most surprising plot after many checks needs acknowledgment, rather than treating its tail probability as a predeclared single test.

A model can fail because the physical mean is wrong or because the measurement process is wrong. An omitted background or a time-dependent efficiency may mimic a new physical effect. Follow a failed check with a physically motivated revision and test its implications, ideally with independent information. A large p-value can also reflect low power, overestimated errors or an overflexible model. Report the discrepancy, reference construction and diagnostic limitations so another analyst can judge which alternatives the check could actually detect.