Information bottlenecks and variational bounds

Begin with a finite joint law for X and Y and choose a finite representation T through a stochastic encoder q(t given x). Its joint law factors as p(x,y)q(t given x), so Y to X to T is a Markov chain. We minimize I(X;T) minus beta times I(T;Y), with beta nonnegative and all information measured in bits. Compression penalizes retained detail about X; relevance rewards retained detail about the chosen Y.

Data processing bounds relevance by both I(X;Y) and I(X;T). For beta at most one, the objective is therefore nonnegative and a constant representation attains zero. A nontrivial tradeoff can appear for larger beta, but it is not guaranteed when X and Y are unrelated. Some papers maximize relevance minus beta-A times compression instead; for positive coefficients the equivalent parameter is beta-A equal to one over our beta. Comparing reported coefficients requires accounting for that convention.

Choose a reference law r(t) positive wherever the encoder's marginal is positive. The expected KL from q(t given X) to r equals compression mutual information plus the KL from the true representation marginal to r. It is therefore an upper bound on compression, tight when r equals that marginal. A reference distribution is an auxiliary modeling choice, not a claim that the encoder's output actually follows it.

Choose a predictive conditional law s(y given t) that is positive on the relevant joint support. Replacing the true conditional log-likelihood by log s lowers the relevance expression by an average conditional KL. Thus H(Y) plus the expected log predictive probability is a lower bound on relevance. Multiplying that lower bound by minus beta reverses its inequality, so combining it with the compression upper bound gives an upper bound on the minimizing objective. A zero predictive probability on a possible event instead gives an infinite cross-entropy penalty.

Dropping the constant minus beta H(Y), the upper-bound objective becomes the KL penalty U plus beta times predictive cross-entropy. A formulation using cross-entropy plus beta-A times KL has the reciprocal coefficient after rescaling by a positive constant. Both logarithm units must also match. Replacing the population expectation by a sample average introduces estimation error: a realized empirical loss need not remain an upper bound on the unknown population information objective.

The finite model avoids another issue: a deterministic representation of a non-atomic continuous input can have infinite mutual information even when it looks low dimensional. Adding noise or discretizing changes the mathematical model and requires stating that choice. Finally, a variational bound is not a universal generalization, robustness or causal-identification theorem. Those properties need additional assumptions and evidence beyond a favorable information objective or a lower training loss.