Entropy, chain rules and mutual information

For a finite random variable X with probabilities p(x), the information of a supported outcome is minus log-base-two p(x). Its expectation is H(X), measured in bits. We interpret zero times log zero as zero when summing. An impossible outcome has no contribution to that expectation; it is not an ordinary observation with a finite surprise. Replacing base two by the natural logarithm produces nats, and multiplies every information value by ln two.

Entropy is nonnegative and at most the logarithm of the support size, with equality at the uniform law. A deterministic variable has entropy zero even if its alphabet contains many unused labels. Splitting one label into distinguishable sublabels can change the random variable and its entropy; merely renaming labels bijectively does not. This is one reason to specify exactly what the outcome represents before comparing two numerical entropy values.

For each supported y, calculate the conditional distribution of X given Y=y, find its entropy, and average over y. The chain rule follows by writing a joint probability as a marginal times a conditional probability and expanding the logarithm. Repeating that identity yields a sum of sequential conditional entropies for a whole vector. The order of the expansion can change the individual conditional terms while leaving the joint entropy unchanged.

Conditioning reduces entropy on average: H(X given Y) is at most H(X). A particular rare conditional event can nevertheless have a more uncertain distribution than the original marginal. The theorem compares the weighted average, not every conditional slice separately. For discrete variables, conditional entropy is also nonnegative. These familiar inequalities should not be transferred unchanged to differential entropy, which uses densities and can take negative values after a change of measurement scale.

For finite variables, mutual information is zero exactly when the joint law factors into the product of its marginals. This is a stronger condition than zero covariance, which only tests a particular moment. Conversely, a deterministic function Y of X has conditional entropy zero and mutual information H(Y), even when the function discards much of X. Retaining every detail of X and retaining every detail relevant to Y are therefore different compression objectives.

The information in a pair of observations is not generally the sum of two separate information values. The observations may be redundant or may reveal a relationship only when combined. Conditional mutual information supplies the correct chain-rule correction. A statistical dependence identified this way can arise from a common cause, selection or direct influence; the information calculation alone does not choose among those mechanisms. It quantifies the supplied observational distribution.