Residual networks and normalization

Let a block receive a row vector of width d and return the same width. Adding its learned branch to its input gives a direct identity contribution to the Jacobian. If the branch initially produces a small correction, the block initially behaves close to identity. This can make adding depth easier to optimize. It does not force the full Jacobian to have well behaved singular values: the learned derivative can still amplify, cancel, or rotate incoming gradients.

Addition requires matching shapes. A stage that changes channels or downsamples spatial dimensions needs a projection or another specified skip transformation. That projection has its own parameters and derivative; it is no longer a literal identity. Also distinguish an unmodified identity path from a post-addition nonlinearity. A ReLU applied after addition can zero a component's derivative. Trace the actual ordering when comparing residual formulations rather than treating every skip connection as mathematically identical.

For tokens shaped batch by sequence by features, a conventional LayerNorm computes a mean and variance over the feature axis separately for each token. It does not mix examples or average over sequence positions. Learned feature-wise gain and bias then restore flexibility. The small positive epsilon prevents division by zero and influences behavior when variance is tiny. Specify the axes because a numerically valid reduction over the wrong dimension can silently make tokens depend on their neighbors.

BatchNorm commonly uses batch and spatial axes to estimate a separate statistic per channel and usually uses running estimates for evaluation. LayerNorm uses each example's current features in both modes. This difference matters for small batches, distributed synchronization, variable sequence lengths, and deployment. Neither operation makes all activations independent, nor does zero mean imply good conditioning everywhere. Pre-normalization, where a residual branch consumes normalized input, preserves a direct identity path around that branch; post-normalization places another derivative on that path.

Residual addition is inexpensive relative to large matrix multiplications, but keeping activations for backward propagation costs memory. Normalization also reads and reduces tensors and can become visible in small-batch inference. Kernel fusion can reduce memory traffic without changing the architectural equation, subject to numerical tolerances. Checkpointing trades recomputation for activation storage; it does not remove parameters or optimizer state. Separate these memory categories before attributing a training limit to depth alone.

Compare alternatives using a fixed task and declared training budget. Record initialization, residual scaling, precision, gradient clipping, and normalization placement. A model that diverges under one learning rate has not automatically lost an architectural comparison if another model received more tuning. Inspect activation and gradient statistics across depth, then test whether an intervention repairs the diagnosed problem. A claim about optimization should be supported by training behavior as well as final validation accuracy.