Engineering / research
Make uncertainty useful to a decision
Connect predictive scores to coverage, abstention, shift checks, and operational thresholds while stating exactly which calibration guarantees apply.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Make uncertainty useful to a decision
Connect predictive scores to coverage, abstention, shift checks, and operational thresholds while stating exactly which calibration guarantees apply.
- 01
Predictive versus parameter uncertainty
Decompose uncertainty within a declared predictive model while separating posterior approximation, data variation, and model misspecification.
29 min - 02
Proper scoring rules and decision costs
Evaluate probability forecasts with proper losses and derive an action from a separate, explicit cost matrix.
29 min - 03
Temperature scaling and calibration splits
Fit one positive temperature on held-out logits, preserve class ordering, and reserve independent data for policy tuning and evaluation.
29 min - 04
Calibration by class and subpopulation
Expose cancellation of calibration gaps across classes and groups while reporting bin definitions, counts, and sampling uncertainty.
29 min - 05
Conformal score construction and quantiles
Implement a split-conformal rank threshold with explicit ties and the infinite-threshold boundary rather than an interpolated sample quantile.
29 min - 06
Exchangeability and coverage boundaries
State the randomization and exchangeability behind marginal conformal coverage, and identify subgroup, dependence, and shift claims it does not provide.
29 min - 07
Risk-coverage curves and abstention
Calculate error among accepted cases, retain the coverage denominator, and compare abstention policies using explicit costs.
29 min - 08
Out-of-distribution score limitations
Distinguish novelty from task error, expose score nonidentifiability, and evaluate alarms under deployment prevalence and explicit shift families.
29 min - 09
Uncertainty-aware human review queues
Combine selective routing with review capacity, reviewer errors, delay, and an auditable fallback instead of treating abstention as free correctness.
29 min - 10
Selective prediction threshold validation
Freeze the full selective policy before final testing, quantify accepted-case error uncertainty, and prevent threshold search from reusing the claimed test evidence.
29 min