Error attribution: change one evidence stage with a controlled oracle

First determine whether the required authorized evidence exists in the evaluated corpus snapshot. A retriever cannot return a source that was never ingested or is inaccessible to this principal. Next inspect candidate retrieval, parent expansion, deduplication, reranking, and final packing. A relevant hit in the initial list can disappear before generation. For a multi-fact question, finding one answer string is not necessarily complete evidence coverage. Labels should identify the required facts and source spans rather than infer adequacy from keyword overlap alone.

Then evaluate whether the generated answer is correct, supported, appropriately qualified, and cited under the declared criteria. A model may answer correctly from parametric memory even when the context lacks the required evidence; that does not establish retrieval grounding. Conversely, adequate evidence can be present while the answer misreads a date, merges conflicting versions, or cites the wrong span. Record both evidence availability and answer outcomes. Separate unsupported correctness from a response meeting the full attributed-answer requirement.

A retrieval oracle supplies labeled relevant candidate identities from the same authorized snapshot, then leaves expansion and packing unchanged. A context oracle supplies a valid bounded evidence context directly, preserving the generator, prompt format, and output budget as far as the experiment permits. These diagnose different stages. If relevant facts cannot fit the declared budget, an unlimited gold context changes the problem rather than isolating retrieval. Label oracle construction and any unavailable cases; do not silently inject information from outside the principal's permitted corpus.

A reference reader or human adjudicator can inspect retrieved evidence to assess whether it is sufficient, but its performance is not an automatic numerical ceiling for another generator. Likewise, an oracle-context score is an observed intervention result: it can change ordering, distractors, and wording as well as coverage. Use paired cases and documented controls. Compare a frozen generator with actual and oracle evidence, then separately compare generator variants on the same contexts. Avoid attributing every difference in a multi-component change to the retriever.

Keep corpus and ACL versions, query transformations, candidate identities and ranks, expansion maps, packing decisions, rendered input, output, and claim-level judgments. For a failure, ask where the first required fact became unavailable or was misused. Run controlled cases with no evidence, sufficient evidence, distractors, conflicting evidence, and evidence moved between positions. Original RAG and Fusion-in-Decoder evaluate specific retrieval-generation architectures; their results do not imply that a prompt-only pipeline has the same training or attribution behavior.

The probability lab represents a hypothetical evidence-coverage rate only. Reset, set p=0.60 and n=10, predict mean six successes, then Play or Step. Separately Reset, set p=0.80 and n=10, predict mean eight, then Play or Step. Exact binomial bars assume independent identical cases, while real benchmark errors can be clustered. The instrument has no corpus, retriever, generator, or oracle. Use it to distinguish expected counts under a model from the paired observed intervention needed to diagnose an actual pipeline.