Alternating least squares and conditioning

Use real factors and squared Frobenius loss one-half ||T − Σaᵣ ⊗ bᵣ ⊗ cᵣ||F². With B and C fixed, define Z = C ⊙ B, whose r-th column is cᵣ ⊗ bᵣ. This order matches our first unfolding with j varying fastest inside the combined (j, k) column. The model unfolding is AZᵀ, so updating A solves a matrix least-squares problem with one shared design Z for every row.

First-order optimality gives A(ZᵀZ) = T₍₁₎Z. If the design has full column rank, the minimizer is unique; otherwise multiple factor updates can give the same fitted block, and a pseudoinverse selects one convention. The Gram matrix equals the entrywise product of BᵀB and CᵀC. This identity can reduce computation, but it should not hide the conditioning cost of forming normal equations.

For a full-column-rank design, the two-norm condition number of ZᵀZ is the square of that of Z. QR or SVD based least-squares solves can avoid explicitly squaring the conditioning in a computed Gram system. Nearly proportional factor columns often create an ill-conditioned design, even when each exact independence condition still holds. Column normalization helps control arbitrary factor scaling but cannot make genuinely dependent component information independent.

Adding λ||A||F²/2 with λ > 0 replaces the Gram matrix by ZᵀZ + λI and gives a unique regularized block update. If the full algorithm regularizes every factor, it optimizes a changed global objective, not the unregularized CP residual alone. Regularization can control component growth, at the cost of biasing the fit. Report the penalty and scaling conventions when comparing fitted components across runs.

An exact block minimization cannot increase the current objective because the old block is feasible. The nonnegative objective values therefore converge, but that does not itself prove factor convergence, boundedness, or global optimality. If complete sweeps converge and all smooth block first-order residuals tend to zero, the limit is stationary. Such conclusions require assumptions on the updates and limiting behavior; a small change in objective is not a stationarity certificate by itself.

ALS may stall in slowly changing regions, settle at a poor stationary fit, or approach a CP boundary with large cancelling components. The earlier border-rank example supplies a concrete reason a vanishing residual need not lead to finite factors. Useful diagnostics include the residual, block gradients, column norms, and sensitivity to starting points. Multiple initializations may improve a practical answer without proving that the best encountered fit is globally optimal.