Stable softmax: keep the normalizer consistent as tiles arrive

For a nonempty finite row, subtract its largest logit before exponentiation. The maximum shifted logit is zero, so no shifted exponential exceeds one in real arithmetic. The normalizer remains positive because at least one term is one. Very small shifted terms can still underflow in a finite representation, and approximate exponential instructions introduce their own error. The transformation improves stability; it does not make a floating-point implementation exact. Logits here are dimensionless, as required for the exponential.

Masks need a defined semantic contract. Invalid tail lanes can use an appropriate exclusion value for a maximum and exponential reduction, but a row with every position masked cannot blindly compute negative infinity minus negative infinity. Positive infinity, NaN, and empty rows also need an explicit API policy or supported reference behavior. Choose the accumulator type and final output conversion separately from the input type. Check normalization and selected probability errors against a trusted reference rather than only checking that the kernel returned finite values.

Summarize a block by its maximum and the sum of exponentials shifted by that maximum. To merge two blocks, choose the larger maximum and rescale both sums to it before adding. Adding the original shifted sums directly is wrong when their maxima differ. A weighted-value accumulator can be rescaled the same way, allowing the normalized weighted result to be formed after processing more blocks. The summaries refer to represented logits and the chosen arithmetic; their mathematical combination is associative in real arithmetic, while floating-point grouping can still differ.

Online normalization is one mechanism enabling attention algorithms to avoid storing the entire score or probability matrix in external memory. FlashAttention combines tiling, online softmax, and careful data movement; it is not a claim that attention performs no quadratic arithmetic or that every sequence length benefits equally. Causal masks, head dimensions, dropout, and backward computation add further contracts. Distinguish exact attention as a mathematical operation from bitwise equivalence to a particular materialized implementation. Different rounding and accumulation schedules require tolerance-based validation.

A fused row softmax can load a row, compute its maximum and sum, and write normalized results without round-tripping several intermediates through device memory. Large rows may not fit the chosen register or shared-memory budget, requiring multiple stages or a different partition. More retained values can cause spills or reduce parallelism. Include any rereads and temporary storage in the actual traffic boundary. A small illustrative tutorial's in-row fusion strategy is not a universal implementation for arbitrary shapes and hardware.

The network lab shows exactly the two-logit softmax family [0,z] through p=σ(z), where z=wx−0.5. Reset, set x=1 and w=1, predict p≈0.622, then Play or Step. Separately Reset, set x=1 and w=3, predict p≈0.924, then Play or Step. The complement is the probability for the zero logit. The instrument does not expose arbitrary rows, large-logit arithmetic, masks, or online block merging. Use the hand example to verify rescaling and the lab to inspect only this fixed binary normalization family.