Use variance arithmetic as a scoped initialization model

For one output z of a width-d linear map, sum d products of weights and inputs. If weights have zero mean, are independent across coordinates and independent of inputs, and cross-product terms vanish, the output variance is d times weight variance times input second moment. With zero-mean inputs, that second moment is their variance. Nonzero means, correlated weights, tied parameters, or data-dependent branches can invalidate the simple estimate. An initializer's standard deviation is not a complete description of the resulting activation distribution.

Choosing weight variance proportional to inverse fan-in can keep this idealized output variance comparable to the input variance. Glorot and Bengio's original analysis studies related signal-propagation considerations, while Transformer normalization analyses examine architecture-specific gradient behavior. Their conclusions rely on particular models and assumptions. Do not transplant a historical hyperparameter without its normalization placement, residual structure, embedding scale, and depth convention. Record which tensors receive which initializer and whether a later loading or tying step changes them.

For h_next=h+γF(h), the exact scalar-coordinate variance includes the variance of h, the scaled branch variance, and twice the scaled covariance. Even a zero-mean initialized branch is a function of h, so covariance cannot generally be discarded merely because weights were sampled independently at creation. A simplified analysis may assume negligible covariance and fixed branch variance to motivate a depth-dependent scale, but those are premises rather than results of writing a residual addition.

Under the further toy assumption that every layer adds an uncorrelated unit-variance branch, L additions with scale γ add Lγ² to the variance. Setting γ=1/√L keeps that added amount at one in this model. This does not establish gradient stability, attention-score behavior, or an optimal scale for a real decoder. Normalization placement changes which tensors are normalized and where scale can accumulate; Xiong and colleagues' analysis is an example of why those architectural details matter.

Run a small forward and backward fixture before expensive training. Record per-layer residual and branch RMS, selected quantiles, attention-logit scales, finite-value counts, and gradient norms under representative serialized batches. Separate normalization inputs and outputs so a normalized tensor does not hide growth elsewhere. Test multiple initialization seeds and input domains when practical. A single finite loss is weak evidence: a model can begin with saturated submodules or strongly depth-dependent gradients while still returning a number.

The network lab illustrates saturation in one nonlinear unit, not residual-stream variance. Reset, set both x=1,w=1, predict logit 0.5 and sigmoid slope factor about 0.2350, then Play or Step. Reset and set both x=3,w=3 for logit 8.5 and a much smaller factor about 0.0002034. The second setting is control-reachable but is not a recommended initializer. There are no random weights, layers, normalization, residual additions, or training updates in this instrument; derive variance separately and use it to guide actual measurements.