Classification, probabilities, and decision costs

A binary logistic model produces a real-valued logit $z=w^\mathsf{T}x+b$. The sigmoid maps it between zero and one. A zero logit gives probability one-half; positive logits favor class one. The logit is the log odds, so adding a fixed amount multiplies the odds rather than adding a fixed probability. This allows a linear score to represent a nonlinear probability curve without allowing impossible negative probabilities or values above one.

For mutually exclusive classes, softmax converts a vector of logits into a categorical distribution. Subtracting the same constant from every logit leaves probabilities unchanged; numerical implementations exploit this by subtracting the largest logit before exponentiation. Multilabel tasks are different: an image may contain both a bicycle and a person, so separate sigmoid outputs are often appropriate. The label semantics, rather than the number of output neurons, determine the correct probability model.

For a binary label, the Bernoulli likelihood assigns probability p to a positive outcome and one minus p to a negative outcome. Negative log likelihood therefore penalizes assigning very little probability to what actually happened. A correct label guessed at 0.51 and one predicted at 0.99 have the same thresholded accuracy but different losses. Conversely, a confident error is much more costly under log loss than a hesitant one. This makes the training objective sensitive to information discarded by accuracy.

Under the stated binary model, differentiating cross-entropy with respect to the logit gives probability minus label. Use a numerically stable logits-based loss rather than manually taking logarithms of rounded probabilities. For example, PyTorch's BCEWithLogitsLoss combines the sigmoid and binary loss; its input should be logits, not probabilities already passed through sigmoid. Class weighting or resampling changes the effective training objective and can alter probability interpretation, so evaluate and calibrate on the intended population.

A threshold of one-half is justified only under particular assumptions, such as equally costly false positives and false negatives with calibrated deployment probabilities. Let a false alarm cost one unit and a missed positive cost nine. Acting positive has expected cost one minus p; acting negative has expected cost nine p. Comparing them gives a threshold of 0.1. The probability model has not changed, but the rational action policy has.

Evaluate a threshold using counts of true positives, false positives, and missed positives. Precision asks how many predicted positives are correct; recall asks how many actual positives were found. A rare class can produce high accuracy with zero useful recall. Choose operating points on validation data, document capacity constraints, and then evaluate once on the held-out test. Costs, prevalence, and intervention effectiveness can change, so a threshold is an operational assumption to revisit rather than an eternal model property.