Information and entropy

Define the surprise of an event with positive probability as minus the logarithm of its probability. Independent events have multiplying probabilities, so their surprises add. With logarithms base two, the unit is a bit. A certain event has zero surprise, while a rare event has more. Entropy averages this surprise over the possible outcomes, weighting each by how often the model says it occurs.

For a finite distribution, adopt the limiting convention $0\log 0=0$: impossible outcomes contribute nothing to the average. Entropy is nonnegative and is largest for a uniform distribution on a fixed finite support. A deterministic symbol has zero entropy; a fair binary symbol has one bit. Adding unused zero-probability labels does not create uncertainty. These are statements about a distribution, not a measure of the importance of the symbol names.

Suppose outcomes are drawn from distribution $p$ while predictions assign probabilities using $q$. Cross-entropy is the expected surprise measured using $q$. It separates into the unavoidable entropy of $p$ plus a mismatch penalty called KL divergence. In classification, minimizing average negative log probability corresponds to this type of objective; the data distribution and the model's allowed family determine what can actually be learned.

KL divergence is nonnegative but generally asymmetric, so it is not a distance metric. If $p$ assigns positive probability where $q$ assigns zero, the divergence is infinite: the predictor declares a possible event impossible. A small KL value also does not automatically guarantee acceptable errors for every rare subgroup or decision cost. The logarithmic objective evaluates a particular form of probabilistic mismatch rather than every practical notion of model quality.

Evidence reduces uncertainty on average; mutual information quantifies dependence through joint and marginal distributions. Correlated symbols can have less joint entropy than their separate entropies sum to. Thus a stream's one-symbol frequencies do not determine its full compressibility: relationships across time matter. This observation motivates a complete source model without proving a coding theorem here.

For continuous variables, a density integral defines differential entropy, which can be negative and changes under coordinate scaling. Thermodynamic entropy requires a physical microstate model with appropriate constants and units; quantum von Neumann entropy uses density operators. These structures share valuable mathematics, but their physical interpretations do not follow solely from the finite-symbol formula.