Evaluation, uncertainty, and calibrated confidence

Count true positives, false positives, true negatives, and false negatives at the chosen threshold. Precision is the fraction of predicted positives that are real positives; recall is the fraction of real positives found. Their denominators answer different questions. A precision score without its count can hide enormous uncertainty, and an undefined ratio when there are no predicted positives should not silently become a successful result. Record the convention used by the reporting implementation.

Prevalence affects interpretation. Even a small false-positive rate can produce many false alarms when negatives greatly outnumber positives. A ranking metric can compare score ordering across thresholds, but deployment still needs a specific action policy. Estimate performance at the intended traffic mix and capacity. Report important slices and uncertainty, especially when examples are clustered by user or incident; a confidence interval that assumes independent rows can be overconfident for strongly correlated observations.

For binary event probabilities, calibration asks whether outcomes occur about 80% of the time among cases assigned probability 0.8, and analogously elsewhere. One confident correct answer proves little: calibration is a distributional property. Reliability diagrams approximate it by grouping predictions, but bin boundaries and small group counts affect the picture. Calibration within broad bins can also hide failures in particular subgroups, so inspect the intended use rather than trusting one aggregate visualization.

The Brier score averages squared probability errors; log loss also evaluates probabilities and penalizes confident mistakes strongly. These scores reflect more than calibration, including how informative predictions are. A constant base-rate predictor can be calibrated while making little useful distinction between cases. Conversely, a well-ranked model can be systematically overconfident. This separates discrimination, which concerns useful separation, from calibration, which concerns whether stated probabilities match empirical frequencies.

Temperature scaling divides multiclass logits by a positive fitted temperature before softmax. Increasing temperature generally flattens the distribution while preserving the class ordering for each example. Fit the temperature on a held-out calibration set, or within a properly separated development procedure, and assess it on data not used for fitting. It cannot fix every kind of miscalibration, improve the available features, or guarantee calibrated probabilities after a population shift.

Freeze the model, preprocessing, calibrator, and policy before final evaluation. Quantify uncertainty using a method consistent with the sampling unit; for grouped data, resample or analyze groups rather than pretending each row is independent. After deployment, labels may arrive late or only for selected cases, making monitoring biased. Keep the original evaluation report reproducible, document the label window, and treat changes in prevalence or collection as reasons to recheck both threshold performance and probability reliability.