greatitself.
Continue
research29 min studyLesson 863 in Engineering

Temperature scaling and calibration splits

Fit one positive temperature on held-out logits, preserve class ordering, and reserve independent data for policy tuning and evaluation.

Before you begin
On this page

The question for today

What you’ll understand

  • Compute softmax probabilities after positive temperature scaling.
  • Explain what a single temperature can and cannot change.
  • Assign training, calibration, threshold-selection, and testing data distinct roles.

A confidence correction can look successful simply because it was assessed on the labels used to fit it. Temperature scaling is small enough to make this leakage easy to overlook: it learns only one number, but that number still adapts to data. A sound procedure freezes the base model, fits a positive temperature on a separate calibration split, chooses any operating threshold without consulting the final test labels, and then evaluates the entire frozen pipeline.

Scale logits, not a single displayed percentage

For multiclass logits z, divide every component by the same positive temperature T before applying softmax. Values above one flatten the distribution; values below one sharpen it. Because division by a positive scalar preserves logit ordering, the predicted class and exact ties are unchanged under the same tie-breaking rule. The transformation changes confidence and can consequently change acceptance decisions at a fixed probability threshold, even though the classifier’s ordinary top-one predictions do not change.

Apply the transformation to the logits that actually reach the deployed output. Scaling an already rounded maximum probability generally loses information, and scaling only one class changes the method. Store the model hash, logit conventions, class vocabulary, precision, and temperature together. If an upstream ensemble, retrieval stage, prompt, or classifier changes, the old calibration is an empirical hypothesis that needs rechecking. A temperature is not a portable property of a model name independent of its inference pipeline.

pT(y=j∣x)=exp⁡(zj(x)/T)∑k=1Kexp⁡(zk(x)/T),T>0p_T(y=j\mid x)=\frac{\exp(z_j(x)/T)}{\sum_{k=1}^{K}\exp(z_k(x)/T)},\qquad T>0
Logits and temperature are dimensionless in this convention, K is the fixed class count, and exponentials are evaluated stably after subtracting the maximum scaled logit. Positive T preserves within-example class ordering; it does not guarantee calibrated probabilities or preserve every cross-example confidence ranking in multiclass problems.↔ Scroll wide equations horizontally.

Fit a proper objective on held-out labels

The original temperature-scaling procedure chooses T by minimizing negative log likelihood on held-out labeled cases with the base predictor fixed. Optimize a positive parameterization and inspect boundary behavior rather than permitting negative or zero temperatures. A tiny, unrepresentative calibration split can produce an unstable temperature even with only one parameter. Record the fitting sample count, source period, and label process, and compare held-out probability loss before and after the transformation.

Do not fit temperature by repeatedly minimizing the final test set’s binned calibration error. Expected calibration error depends on bin boundaries and sample counts, and its empirical value is not a strict proper fitting objective. A lower calibration-set loss also does not establish improvement on future cases. For a practical release, separate temperature-fitting labels, policy-selection labels, and final evaluation labels, or use a precisely specified cross-fitting protocol whose analysis matches all adaptive uses. Renaming a reused split does not restore independence.

T^∈argmin⁡T>0  1ncal∑i=1ncal−log⁡pT(yi∣xi)\widehat T\in\underset{T>0}{\operatorname{argmin}}\;\frac{1}{n_{\mathrm{cal}}}\sum_{i=1}^{n_{\mathrm{cal}}}-\log p_T(y_i\mid x_i)
The base logits are fixed independently of these calibration labels; n_cal is a positive integer and natural logarithms give mean loss in nats per case. The equation describes fitting, not an unbiased final test estimate. A finite minimizer need not exist without additional constraints or regularization.↔ Scroll wide equations horizontally.

Test the fitted pipeline and its limits

A single global temperature corrects only a restricted pattern of confidence distortion. It cannot repair wrong class ordering, recover missing information, or ensure calibration in every subgroup. Measure class-specific and operational slices after fitting, and distinguish a changed acceptance population from a changed classifier. Under distribution shift, reassess probability quality and decision costs with fresh labels. Neither a smooth reliability diagram nor a positive fitted temperature proves that a downstream abstention or conformal procedure has its required guarantees.

The sigmoid lab can reproduce two chosen binary logit values without fitting T. Reset, set input x to 1 and weight w to 2.5, predict z = wx−0.5 = 2 and output approximately 0.8808, then Step. Compare a fresh Reset with both x at 1 and w at 1.5 before Step: z = 1 and output approximately 0.7311. This mirrors dividing the first logit difference by two; the instrument has fixed bias and no calibration dataset or optimizer.

Work it through

A binary confidence changes while its label stays fixed

For a binary classifier, the logit difference between class one and class zero is 2. Consider the illustrative temperature T = 2; it has been supplied, not estimated from the example. Compute the probabilities and loss when the observed label is one.

  1. At T = 1, probability of class one is 1/(1 + exp(−2)) ≈ 0.8808. Its binary argmax is class one.

  2. At T = 2, the logit difference becomes 1 and probability is approximately 0.7311. The class prediction remains one because the difference stays positive.

  3. For label one, negative log likelihood rises from approximately 0.1269 to 0.3133 nats on this case. A temperature that helps an entire fitting split can still worsen individual outcomes; this calculation does not establish which T should be fitted.

The result

Temperature modifies probability sharpness without changing this class decision, and its empirical value requires separate fitting and testing.

The living notebook

Explore the model

Interactive lab

Interactive lab · network

Two logits without a temperature fitter

Reset, set x = 1 and w = 2.5, predict logit 2 and sigmoid ≈ 0.8808, then Step. Compare a fresh Reset with both x = 1 and w = 1.5 before Step: logit 1, sigmoid ≈ 0.7311. Fixed bias is −0.5; this constructs two logits and performs no temperature estimation.

Two logits without a temperature fitterOne inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Current readings: Product · wx: 1.2; Logit · wx − 0.5: 0.7; Sigmoid output: 0.6682. All parameters are adjustable with the labeled controls below.One input. One weight. One sigmoid.x1.20× w1.20+ b0.70σ0.668output0.668w = 1.00b = −0.50σ(z) = 1 / (1 + exp(−z))
Computation graphSignal trace
Product · wx
1.2
Logit · wx − 0.5
0.7
Sigmoid output
0.6682
t = 0 s

One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA correct logit transformation is a mechanism, not evidence of calibration.

Pause & reason

Check your understanding

A positive global temperature was fitted to a frozen multiclass model on calibration labels. Which claim follows from the transformation alone?

Your turn

Try it yourself

A binary logit difference is −2 and a supplied temperature is 2. Compute class-one probabilities before and after scaling, and explain whether a new final test set may be used to choose T after its labels have been inspected.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use the sigmoid at −2 and −1. The sign preserves the predicted class; adapting T to inspected labels changes those labels’ statistical role.

Compare with the solution

Class-one probability changes from approximately 0.1192 to 0.2689, while the class-zero argmax is unchanged. Once test labels influence T or any choice of method, their reported loss is affected by selection. Treat that split as development evidence and obtain an untouched final evaluation set, or use a preplanned method with valid correction for the adaptive procedure. Keeping the same filename test does not make the assessment independent.

Keep the idea

Take it with you

  • Positive temperature preserves each case’s class ordering.
  • Fit the temperature on labels that did not train the base predictor.
  • Freeze calibration and policy choices before evaluating final test labels.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.