Variational inference and approximation error

Let $p(y,\theta)$ be a joint density with positive finite evidence $p(y)$, and let $q_\lambda(\theta)$ be a normalized approximating density. When the relevant expectations and divergence are well-defined and finite, expanding the reverse Kullback–Leibler divergence gives log evidence minus an evidence lower bound, or ELBO. Since the evidence is independent of variational parameters, maximizing the ELBO minimizes this divergence within the chosen family. The word lower refers to log evidence, not to every posterior probability or predictive interval.

The ELBO contains an expected log joint density and the approximation's entropy. The entropy term prevents the objective from simply being a search for a posterior mode when the family permits nonzero spread. A proper approximation must respect target support; placing positive approximation mass where target density is zero produces infinite reverse KL. Expectations estimated using stochastic gradients introduce numerical noise, while unconstrained parameter transformations still require density Jacobians. Optimizing an incorrectly coded objective does not become valid because its value improves smoothly.

Mean-field approximation factorizes the joint approximation into independent coordinate distributions. This is a computational restriction, not a conclusion that the true posterior coordinates are independent. For a correlated Gaussian target, optimizing a fully factorized Gaussian under reverse KL gives coordinate variances determined by diagonal precision entries, rather than the target's marginal variances. These can be much smaller. The missing covariance also affects sums and contrasts, so checking only coordinate means can conceal a serious uncertainty error.

For a bivariate zero-mean Gaussian with unit marginal variances and correlation $\rho$ satisfying $|\rho|<1$, its precision diagonal is $1/(1-\rho^2)$. A diagonal Gaussian approximation minimizes reverse KL at zero mean and both variances $1-\rho^2$. Different divergences or richer families yield different solutions. Reverse KL can favor concentrating in one mode when covering low-target-density gaps is expensive, but this is a tendency in particular settings, not a theorem that every variational approximation misses modes or always underestimates every uncertainty.

Optimization error means the algorithm has not reached the best available member of its family. Approximation error remains even at that best member, because the family or objective may exclude the target features needed. Monte Carlo error enters when expectations or gradients are estimated from finite random draws. These errors require different responses: more optimization may reduce the first, a richer family may reduce the second, and more or better draws may reduce the third. None of them represents the underlying model's substantive posterior uncertainty.

Compare independent initializations, inspect predictive quantities, and benchmark manageable examples against exact calculations or well-diagnosed sampling. Importance reweighting can sometimes assess or improve an approximation, but unstable weights reveal overlap problems and do not guarantee correction. Simulation-based calibration tests computation under the assumed model; real-data predictive checks test other implications. Report which summaries were checked and whether decisions remain stable under alternative approximations. An ELBO increase is useful optimization evidence, while accurate tail probabilities or calibrated intervals require evidence directed at those particular goals.