Likelihood and innovation algorithms

Consider a jointly Gaussian observation vector with constant mean mu and a positive definite covariance matrix Gamma depending on model parameters. Sequentially predict each centered observation from those preceding it. The residuals are orthogonal, and joint Gaussianity makes them independent. Their variances are positive. Algebraically this is a lower triangular factorization Gamma=L D L-transpose, where L has unit diagonal and D contains the innovation variances; the residual vector is L-inverse times the centered observations.

The determinant of L is one, so the covariance determinant is the product of those variances. The inverse-covariance quadratic form becomes the sum of each squared residual divided by its variance. This factorization produces exactly the multivariate Gaussian density, using the mean and covariance of the entire finite vector. Natural logarithms appear in its log likelihood; the two pi normalization belongs to a normal density, not to the earlier choice of Fourier frequency units.

For a stationary AR(1), the first observation has variance sigma-squared divided by one minus phi-squared. Later observations have conditional variance sigma-squared and conditional mean mu plus phi times the previous centered value. An exact stationary likelihood includes the first density, whose variance depends on the parameters. A likelihood conditioned on the first observed value drops that factor. Neither convention should be silently substituted for the other during a finite-sample comparison.

For moving averages, setting unobserved presample shocks to zero creates a convenient conditional residual recursion. Those zero values are not the random presample shocks integrated out by the stationary finite-vector density. Invertibility can make initialization effects decay and support asymptotic approximations under further regularity, but it does not make the two finite likelihoods identical. A declared mean must also remain consistent: an AR intercept equals phi of one times mu, rather than generally equaling mu itself.

If a series is not jointly Gaussian, orthogonal linear innovations need not be independent, and a product of their normal densities is generally not its exact distribution. It can instead define a Gaussian quasi-likelihood criterion. Consistency or standard errors for that criterion require assumptions beyond the second-moment identities. Similarly, an estimated innovation sequence looking uncorrelated does not prove independence, correct conditional variance or a correct tail model.

Finite innovation recursions also distinguish available observations from the infinite history in Wold's theorem. Their earliest variances incorporate the missing past. Positive definiteness guarantees mathematical divisions are possible, but a covariance near singularity can still cause numerical difficulty. Practical fitting should document mean handling, initialization, parameter constraints and convergence, then examine residual behavior and forecast performance. An optimization routine returning a number is not evidence that every statistical assumption used to interpret that number holds.