Proper scoring rules and decision costs

For a binary event Y, the Brier loss is the squared difference between a reported probability p and the observed zero-or-one outcome. If the true event probability is q, expand the expected loss to obtain a squared forecasting error plus q times one minus q. The second term does not depend on the reported probability. The unique expected-loss minimizer is therefore p equal to q, even though an individual realized outcome can make a less accurate forecast look better.

A strictly proper loss rewards truthful distribution reporting in expectation over the specified outcome distribution. It does not guarantee that finite data identify that distribution, that the model can express it, or that the optimizer finds it. Specify whether a score is a reward to maximize or a loss to minimize; the terminology otherwise creates sign errors. For multiclass Brier scores, also record whether components are summed or averaged, because numerical scales differ even when the minimizer is unchanged.

Suppose predicting positive incurs cost C_FP only when the event is absent, while predicting negative incurs C_FN only when it occurs. With zero cost for correct decisions, compare the two conditional expected costs. Their intersection gives a threshold determined by the cost ratio. An expensive missed failure lowers the positive-action threshold. If correct actions also have costs, or several actions are available, compute the full expected cost for each action instead of reusing this special binary formula.

This rule is optimal for the supplied conditional probabilities and costs, not for an unexamined confidence score. Calibration averaged over a broad confidence bin may be insufficient when costs vary by input or group. Likewise, a stable probability model can need a different action policy after costs change without needing retraining. Record the forecast, cost version, selected action, and eventual outcome so later analysis can distinguish probability error from a poor or outdated utility specification.

Log loss supplies another strictly proper criterion and penalizes assigning tiny probability to an observed event strongly. Clipping probabilities for numerical safety changes the evaluated rule, so document the clipping floor and report severe failures separately. Compare mean proper losses with uncertainty appropriate to the sampling unit, alongside the actual action costs. A model can improve ranking or accuracy while worsening probability quality, and a lower average probability loss need not improve a particular constrained decision policy.

The quadratic lab shows the reducible part of expected Brier loss rather than fitting a classifier. Reset, set coefficient a to 1 and position x to 0.3, predict y = 0.09, then Step. Compare a fresh Reset with both a at 1 and x at 0 before Step: zero. Interpret x as a hypothetical probability-reporting error p minus q within its feasible range. The plot omits the constant q(1−q), decision costs, labels, and estimation uncertainty.