Shift exponential losses before they overflow

For a nonempty vector of finite logits, choose its largest value m. Factoring exp(m) from the sum gives log-sum-exp as m plus the logarithm of a sum of exp(z_i minus m). Every shifted exponent is at most zero, so its exponential is at most one, and at least one term is one. This avoids overflowing each large positive exponential individually. It can still lose negligible tiny terms to underflow, and the accumulator, log function, and final addition need their own precision and range checks.

The sum can grow with class count, so using a low-precision accumulator for an enormous number of terms is not automatically safe just because each term is at most one. Likewise, a true final result outside the destination range cannot be made representable by rearranging it. A carefully implemented library routine can choose suitable intermediate arithmetic and edge-case handling. Consult that operation's actual contract rather than assuming an output dtype tells the whole story. Compare both log-sum-exp and derived probabilities against a reference over shifted and extreme but valid fixtures.

For a target class y, cross-entropy is log-sum-exp minus its target logit. A useful evaluation groups the expression as m minus z_y plus the logarithm of the shifted exponential sum. This avoids first rounding a tiny softmax probability to zero and then taking log zero. It can also avoid subtracting two separately rounded large outputs near one another. The same principle motivates combined binary-logit losses that use stable expressions rather than a separately materialized sigmoid followed by logarithms.

For a binary target in zero through one, one stable logit-loss identity uses max(z,0) minus y times z plus log1p(exp(−abs(z))). Here log1p is designed to evaluate log(1+t) accurately for small t. The implementation still needs attention to multiplication and subtraction with exceptional values. An all-masked vector of negative infinities does not satisfy the finite nonempty premise and can make a naive maximum shift produce infinity minus infinity. Define whether such a case is invalid, excluded, or given another explicit application meaning before computing a loss.

If two distinct logits already became identical in a coarse input representation, shifting them afterward cannot recover their original gap. Test input casting separately from the stable evaluation. Check the loss and its gradients, because a finite plausible scalar does not establish useful gradient accuracy. Include nearly equal large logits, one strongly dominant class, many classes, and explicitly invalid or masked cases. Preserve the same target and masking semantics in the reference rather than comparing two implementations of different mathematical functions.

The sigmoid lab shows the moderate-logit function only. Reset, set both input x=1 and weight w=2, predict logit 1.5 and output about 0.818, then Play or Step. Reset and re-enter both x=1 and w=−2 to predict logit −2.5 and output about 0.076. Its fixed graph is sigmoid(wx−0.5), and playback traces the computation rather than training it. It uses no selectable low-precision format and does not reproduce exponential overflow, loss computation, all-masked inputs, or gradient accuracy. Use the worked log-space calculation to study those numerical questions.