Grassmann geometry and subspace learning

The real Grassmann manifold Gr(n,p) contains p-dimensional linear subspaces of Rⁿ. An orthonormal basis X represents one point, and XQ represents the same point for any orthogonal p-by-p matrix Q. Equivalently, the subspace is represented by its rank-p orthogonal projector XXᵀ, which is unchanged by that right action. A function on Grassmann must therefore satisfy F(XQ)=F(X). A cost that rewards one labeled column differently from another may fail this condition and belongs on a frame space instead.

Vertical Stiefel directions XΩ, with Ω skew-symmetric, rotate the basis without changing the quotient point. Horizontal tangent representatives satisfy Xᵀη=0 and encode genuine subspace changes. The standard Grassmann metric is the Frobenius pairing of these horizontal lifts. The induced and canonical Stiefel metrics agree on this horizontal space, making that common quotient metric unambiguous here. A horizontal lift changes equivariantly when the basis is rotated, while its quotient vector, length and directional effect remain the same.

For a real symmetric A, minimizing F([X])=−tr(XᵀAX)/2 finds a top p-dimensional eigenspace. The ambient gradient is −AX, and its horizontal projection is −(I−XXᵀ)AX. This vanishes exactly when the represented subspace is invariant under A. Changing X to XQ rotates the gradient lift by Q and preserves its Frobenius norm. Consequently the residual norm is a legitimate subspace stationarity measure, unlike a coordinate comparison that depends on a chosen basis orientation.

At an invariant subspace, the Hessian sends a horizontal η to −(I−XXᵀ)Aη+η(XᵀAX). In a selected eigenbasis, perturbations that mix a selected eigenvector with an unselected one have curvature equal to selected minus unselected eigenvalue. Positive gaps characterize the isolated top-subspace minimum. Repeated eigenvalues across the selection boundary produce flat directions and nonuniqueness. Eigenvalue multiplicities inside the selected subspace do not themselves create different Grassmann points, since basis rotations inside the same span have already been quotiented out.

Principal-component subspaces and several spectral relaxation problems fit this objective, but noisy data or nonlinear losses can produce different landscapes. One should first verify right-orthogonal invariance before using quotient formulas. Comparing two basis matrices directly by their Frobenius difference is not an invariant subspace error: a sign change or column rotation can make that difference large while leaving the subspace exactly unchanged. Projector differences or principal angles provide invariant alternatives, with their own normalization conventions.

A retraction on the representative frames must also be compatible with the right action so its equivalence class defines a quotient update. Polar normalization has this equivariance. Implementations can fix a convenient basis after each step, but the gradient and transported directions must transform consistently if that gauge changes. Quotient geometry removes artificial flat frame directions; it does not eliminate small genuine eigengaps or data-induced conditioning. Rate statements still depend on those gaps, smoothness bounds and the algorithmic step rule.