Information geometry and Fisher metrics

Consider a smooth parametric family of positive probability densities or masses p_θ with parameter θ in an open set. The score is the gradient of log p_θ evaluated at an observation. Fisher information is the expected outer product of that score with itself. It is therefore positive semidefinite wherever the expectation exists. Under conditions permitting differentiation through normalization and expectation, the score has mean zero and Fisher information also equals the negative expected Hessian of log likelihood. These alternate formulas can fail when support changes with the parameter or required derivatives are not integrably controlled.

For one Bernoulli observation X with success probability θ strictly between zero and one, the score is X/θ−(1−X)/(1−θ). Its squared expectation is 1/[θ(1−θ)]. This diverges toward the boundary in this coordinate, reflecting sensitivity in probability units. The endpoints are not ordinary interior points of this regular chart: events gain zero probability and the log score loses its stated form. For n independent identically distributed observations, information adds to n times the one-observation value. Dependence requires a joint model rather than automatic multiplication by sample count.

Suppose θ is a smooth invertible function of another local coordinate η. The chain rule transforms the score, and the information matrix becomes JᵀI_θJ, where J is the Jacobian of θ with respect to η. Consequently the quadratic infinitesimal length dθᵀI_θdθ is unchanged. The entries of the matrix need not stay the same; invariance belongs to the geometric quantity. Treating Fisher information as a scalar score without its parameter units can make two equivalent parameterizations appear to contain different amounts of information.

For Bernoulli log odds η=log[θ/(1−θ)], the derivative dθ/dη=θ(1−θ). The transformed information is θ(1−θ), rather than its reciprocal. At θ=1/2 it is 1/4, while probability-coordinate information is four. There is no contradiction: an increment dη corresponds to a different increment dθ. Under suitable smoothness, Kullback–Leibler divergence from p_θ to a nearby p_{θ+dθ} has leading term one half the Fisher quadratic form. KL itself is asymmetric and is not a metric; the quadratic local expansion does not change that global fact.

If two parameter directions produce the same distributional change, Fisher information can have a zero direction. For example, a model depending only on a+b cannot distinguish changing a upward while changing b downward. Its two-coordinate Fisher matrix is rank deficient even when the underlying one-parameter model is perfectly regular. Global identifiability alone does not guarantee nonsingular Fisher information at every point: a locally flattened parameter map can create a singular derivative. One needs a locally regular identifiable parameterization with independent score directions to obtain a positive definite Riemannian metric.

Natural-gradient methods use the Fisher geometry to rescale a differential objective, but an inverse Fisher matrix requires invertibility or a clearly stated regularized or quotient treatment. Approximate matrices and damping change the actual algorithm, and geometric motivation alone gives no global optimization guarantee. The probability lab shows exact binomial bars as success probability and trial count vary. Observe how an equal probability increment reshapes a distribution near different parameter values. It does not estimate Fisher information, invert a metric, or execute natural-gradient training; those conclusions require the calculations above.