Conflicts and abstention: measure when evidence is sufficient to answer
Resolve source scope and version conflicts, define abstention behavior, and evaluate calibrated risk versus coverage without treating model confidence as a guarantee.
On this page
The question for today
What you’ll understand
- Distinguish apparent conflicts from real disagreement in applicable evidence.
- Compute answered coverage and selective error risk.
- Apply a stated cost-based answer threshold only under its calibration assumptions.
Retrieved passages can disagree because they refer to different dates, scopes, units, or authority levels. Sometimes the evidence is genuinely unresolved or insufficient. A reliable pipeline needs a defined response in those cases: answer with qualified supported facts, explain an authorized conflict, ask for missing scope, or abstain. Evaluate these decisions with labeled tasks rather than equating confident language with adequate evidence.
Check applicability before trying to reconcile text
Compare source identity, effective date, revision status, jurisdiction or organizational scope, units, and the exact claim being discussed. A newer draft does not automatically supersede an older approved record, and publication time may differ from effective time. Two values can be compatible when one refers to a different product version or population. Preserve those qualifiers in the final answer instead of averaging incompatible numbers or selecting whichever passage had the highest retrieval score.
When a true disagreement remains, apply a declared source-precedence policy if the task has one. Otherwise, retain the conflict and identify what additional authorized evidence would resolve it. A generated summary can accidentally merge the disagreement into an unsupported compromise, so inspect the original spans. Do not expose a restricted document's existence through an explanation to an unauthorized principal. Abstention and clarification templates must respect the same access boundaries as ordinary answers and citations.
Abstention needs a measurable decision rule
A model's self-reported confidence, retrieval similarity, or maximum token probability is not automatically calibrated to answer correctness or evidential support. Fit and validate a decision score using held-out labeled examples from the intended task, including insufficient and conflicting evidence. Check calibration and risk by data slice and after corpus or model changes. A threshold that works on one distribution can fail on another. Distinguish failure to answer an answerable question from correctly refusing an unsupported one.
A simple decision model can compare the expected cost of an incorrect answer with a fixed abstention cost. That derivation assumes a calibrated correctness probability and the stated costs; neither is supplied by a raw language-model score. Real applications may distinguish partial answers, clarification, citation defects, and different error severities. Report risk-coverage curves and task utility together, because reducing answered error rate by answering almost nothing can be a poor product outcome. Evaluate the actual text as well as the binary decision.
Evaluate uncertainty under controlled evidence conditions
Construct cases with sufficient consistent evidence, a missing required fact, contradictory current records, superseded material, and correct facts placed among distractors. Hold the generator and prompt fixed while changing only the evidence condition. Review whether qualifiers and uncertainty match the sources, and count unsupported confident answers separately from valid qualified ones. Preserve the answerable denominator when comparing thresholds. A change in corpus access can make a formerly answerable question unanswerable for one principal without changing public factual truth.
The sampling lab illustrates uncertainty in an independent error-rate estimate. Reset, set p=0.10 and n=20, predict sample-mean expectation 0.10 and variance 0.0045, then Play or Step. Separately Reset, set p=0.10 and n=100, predict the same expectation and variance 0.0009, then Play or Step. Playback highlights exact binomial outcomes rather than collecting real evaluations. The model assumes independent identical errors and has no conflict resolver, confidence score, or abstention threshold. Use it to distinguish a measured finite-sample rate from a guaranteed future risk.
Work it through
A lower selective risk can trade away coverage
On the same 100 labeled questions, policy A answers sixty with six adjudicated errors. A stricter policy B answers forty with two errors. Use cost ten for an erroneous answer, cost two for abstention, and zero for a correct answer.
Policy A has coverage 60/100=60% and answered risk 6/60=10%. It abstains on forty cases.
Policy B has coverage 40/100=40% and answered risk 2/40=5%. It abstains on sixty cases.
Under the stated costs, A costs 6×10+40×2=140 units and B costs 2×10+60×2=140. Lower selective risk alone did not improve this particular measured utility.
Coverage, risk, and utility are separate evaluation quantities. The preferred policy depends on validated behavior and the declared task costs.
The living notebook
Explore the model
Interactive lab · sampling
An estimated error rate has sampling uncertainty
Reset, set p=0.10 and n=20, predict mean K/n=0.10 and variance 0.0045, then Play or Step. Separately Reset, set p=0.10 and n=100, predict variance 0.0009, then Play or Step. The exact iid binomial distribution is not a calibration model, real evaluation run, or abstention controller.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forFinite evaluation counts support an estimate under assumptions, not a guaranteed future error rate.
Pause & reason
Check your understanding
Your turn
Try it yourself
In the teaching decision model, c_wrong=8 and c_abstain=1. Find the correctness-probability threshold. Compare calibrated p=0.90 and p=0.80.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The threshold is one minus the cost ratio; compare expected wrong-answer cost with one.
Compare with the solution
The threshold is 1−1/8=0.875. At p=0.90, expected answer cost is 0.10×8=0.8, below abstention cost one. At p=0.80, it is 0.20×8=1.6, so abstention has lower stated cost. This requires calibrated probabilities and says nothing about an unvalidated confidence phrase.
Keep the idea
Take it with you
- Resolve source applicability and precedence before synthesizing disagreements.
- Abstention policies need calibration, labeled criteria, and coverage reporting.
- Lower selective risk is not automatically better task utility.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.