Separate score movement, confidence, and observed correctness

For a binary classifier, a probability assigned to class one differs from confidence in the selected class. A probability of 0.20 selects class zero under a 0.50 threshold and has top-class confidence 0.80. The predicted positive rate counts threshold crossings, while the mean probability averages values on both sides. These summaries can move independently. Record the exact threshold and tie rule because a policy change can alter decision rates without changing any model scores. For multiclass systems, preserve class identities when comparing score vectors or confusion patterns.

Stratify prediction summaries by model artifact, calibration artifact, feature version, and important request populations. A rollout that mixes versions can move the aggregate histogram even when each version behaves consistently. Class-order permutations can make an apparently plausible probability vector refer to the wrong labels. Monitor normalization and finite-value invariants alongside distributions, and retain enough sampled examples to explain sharp changes. Comparing anonymous score arrays without their output contract can conceal an integration error behind an attractive dashboard.

Once trustworthy outcomes mature, compare each prediction with its aligned label and evaluate accuracy or a task-appropriate loss. Calibration asks whether claimed probabilities match outcome frequencies in a specified population. A single global confidence gap is only a coarse diagnostic: overconfidence in one group can cancel underconfidence in another. Reliability plots by score band and relevant slices offer more detail, while proper scoring rules assess probability predictions beyond a threshold. Fitting a new calibration transformation and evaluating it on the same labels would overstate the evidence.

A historical calibration result is scoped to its evaluation distribution and procedure. It does not guarantee calibration after a population shift, nor does calibration imply a useful ranking or low decision cost. A constant prevalence predictor can be calibrated while offering little discrimination. Conversely, an uncalibrated score may rank requests well. Monitor these properties separately, and compare like-for-like mature cohorts rather than mixing a current confidence statistic with last month's accuracy. Missing outcomes and selective label collection can bias even a neatly joined calibration plot.

A larger fraction of high scores can reflect a genuine increase in positive cases, more extreme inputs, a changed preprocessing scale, or a new model. Before changing a threshold, examine affected examples, population composition, output invariants, and delayed labeled metrics. Conversely, a stable score histogram cannot rule out concept changes that alter outcomes at fixed inputs. Prediction monitors are particularly useful for identifying sudden saturation, constant outputs, lost classes, or changes concentrated in one serving version; their operational value does not require pretending that they measure accuracy.

Use the sigmoid instrument to inspect one mechanism that changes confidence. Reset, set input x=1 and weight w=1, predict logit 0.5 and output approximately 0.6225, then Play or Step. Separately Reset, set x=1 and w=2, predict logit 1.5 and output approximately 0.8176, then Play or Step. The fixed bias is −0.5. This single computation has no labels, training, calibration estimator, or quality measurement. Its increased output therefore illustrates a change in model arithmetic without supplying evidence that the decision became more correct.