Markov chains and stationarity

For a time-homogeneous finite-state chain, P(i,j) is the probability that the next state is j given the current state i. Every entry is nonnegative and every row sums to one. We use row probability vectors, so an initial law evolves by multiplication on the right. The Markov property says that conditioning additionally on earlier states does not change the next-step probabilities once the current state is known. It does not assert that successive states are independent.

State selection is therefore a modeling decision. A queue length might fail to be Markov if the remaining service time matters; adding an appropriate age or residual-time variable may repair the state. A fitted transition matrix that changes with hour of day is not time homogeneous unless the clock is included in the state or the analysis explicitly allows time-dependent matrices. Checking the state description comes before applying a convenient matrix formula to observed data.

A stationary distribution pi is a probability row vector satisfying pi P=pi. Starting the chain with this law leaves every later marginal unchanged. Individual sample paths still move. For two states, stationarity can be understood by balancing the probability mass flowing across their connecting edge. In a larger chain, the full balance equations allow net circulation around cycles; detailed balance on every pair is a stronger sufficient condition, not a requirement for stationarity in general.

Every finite chain has at least one stationary distribution. If the finite chain is irreducible, meaning every state can reach every other through some positive-probability path, the stationary distribution is unique and strictly positive. Reducible chains can have several stationary distributions supported on different closed classes. On infinite state spaces irreducibility alone is insufficient: positive recurrence is needed for an invariant probability distribution. A finite-state theorem should not silently become a theorem for all countable chains.

The period of a state is the greatest common divisor of its possible positive return times. In an irreducible chain all states share the same period. Aperiodic means this period is one. For a finite irreducible aperiodic chain, every initial law converges to the unique stationary distribution. Without aperiodicity, fixed-time distributions can oscillate forever. The two-state chain that switches deterministically at each step has a stationary uniform law but never forgets the parity of a deterministic starting state.

This does not make stationary analysis useless for periodic chains. In a finite irreducible chain, long-run occupation fractions and Cesàro averages of transition probabilities still identify the stationary law, even when fixed-time marginals fail to converge. The type of averaging is doing mathematical work. In simulation, saving every second observation can even hide transitions in a period-two example. Before interpreting a histogram as equilibrium, distinguish temporal averages, ensemble distributions at a fixed time, and the mixing assumptions that connect them. Even after convergence is established, a quantitative mixing-time bound is a separate question about how many transitions achieve a chosen error tolerance.