Measure-theoretic entropy and generating partitions

For a finite measurable partition, assign each atom its probability and sum minus probability times its base-two logarithm, with zero times log zero defined as zero. The result is measured in bits. Join the partition with its first N minus one pullbacks to record observations at times zero through N minus one. Each atom of this joined partition is an ordered name of length N. Distinct names can have the same number of visits to an atom, so a count histogram generally loses information.

Preservation and the entropy inequality for a join make the block entropies subadditive in block length. Consequently their entropy per time has a limit equal to the infimum of the normalized block entropies. This defines the entropy rate of that particular partition. The entropy of the transformation is the supremum over finite measurable partitions. A coarse observation can miss information, so computing one partition's rate normally gives a lower bound on transformation entropy until a generator theorem justifies equality.

For an invertible probability-preserving transformation, a partition is generating when its pullbacks over all integer times generate the measurable sigma-algebra modulo null sets. The finite-entropy generator theorem then identifies the transformation entropy with this partition's rate. We use this theorem on standard Borel probability spaces and invoke it rather than proving its conditional-entropy approximation argument. The finite-entropy assumption matters if a countable partition is used. A finite partition automatically has finite entropy.

For a two-sided finite-alphabet Bernoulli shift, the partition by coordinate zero is generating because shifting it through all integer times reveals every coordinate. Using only future coordinates would not generate the entire two-sided sequence sigma-algebra; the all-integer clause cannot be dropped from the generating claim. Independence makes the entropy of an N-symbol ordered block exactly N times the single-symbol entropy. The generator theorem therefore converts this direct block calculation into the entropy of the whole Bernoulli system.

A uniform finite cycle has positive entropy for the partition into individual states, but its first observation already determines all later observations. Joining further pullbacks creates no additional uncertainty, so its block entropy stays bounded and its entropy rate is zero. Every finite partition on a finite probability space has bounded block entropy, proving that the transformation entropy is zero as well. This example separates ergodicity from positive entropy without requiring a subtle infinite system.

A binomial instrument displays the distribution of the number of ones in a block, not the distribution on all ordered binary names. Many ordered names collapse into one count, which removes their positional uncertainty. Its bars can help check Bernoulli probabilities but cannot directly display the process entropy rate. Finally, changing logarithm base changes numerical units: multiplying a bit entropy by the natural logarithm of two converts it to nats. Comparing rates from different sources requires checking that convention before comparing the numbers.