Out-of-distribution score limitations

Define whether OOD means a new class, a changed sensor, a different language, an unsupported workload, or another specified departure. These targets can overlap without being equivalent. A classifier trained on product categories may correctly recognize a familiar category photographed by a new camera, yet fail on an ambiguous image from the original camera. Deciding which event should trigger abstention requires an operational goal rather than only a geometric distance from training embeddings.

A scalar score can group many inputs together. Two distributions can induce exactly the same score distribution while differing in features the score ignores or in their conditional label laws. A threshold on that score then cannot distinguish them better than its identical alarm rates permit. This is a concrete nonidentifiability example, not a claim that every restricted OOD problem is impossible. Learnability depends on assumptions about available training data, the hypothesis class, and which unseen distributions are allowed.

Maximum softmax probability is a useful baseline whose empirical behavior must be measured. Softmax normalizes relative class logits and can be large far from labeled training support. A generative likelihood evaluates input density, not the probability of a correct task label or of belonging to a semantic class. Published counterexamples show some trained density models assigning higher likelihood to another dataset than to their own. That finding motivates controls; it is not a theorem that likelihood can never help detection.

Report sensitivity on named OOD families and false-positive rate on representative in-distribution inputs at the chosen threshold. Area-under-curve summaries average over thresholds and can conceal a poor deployment operating point. Alarm precision also depends on novelty prevalence: a small false-positive rate can overwhelm rare true novelties. Estimate that prevalence and its uncertainty, and separately measure task errors among accepted and alarmed cases. Detecting a benchmark’s dataset identity is not automatically the same achievement as preventing costly deployment mistakes.

Keep novelty examples used for detector selection separate from final shift-family tests. Include plausible near-distribution changes, label ambiguity, unsupported classes, and in-distribution hard cases. State how test families were selected and avoid extrapolating their performance to every unknown future distribution. Verify that the fallback triggered by an alarm is available and beneficial. Monitoring input-score drift can request investigation, but labels or other task evidence remain necessary to establish a change in correctness or expected harm.

The sigmoid instrument shows a saturated output with no novelty awareness. Reset, set x to 3 and w to 3, predict logit 8.5 and output approximately 0.9998, then Step. Compare a fresh Reset with both x at 0 and w at 3 before Step: logit −0.5 and output approximately 0.3775. Its fixed mathematical sigmoid has no training distribution, class semantics, or labels. Saturation demonstrates only a score mechanism; neither displayed input can be classified as real OOD from this toy alone.