Long-context citation and attribution tests

Split a response into meaningful factual claims and identify exactly which source spans support each. A document mentioning the same entity is insufficient if it states a different date, condition, or relation. Preserve document identity, version, and location through retrieval, reordering, and compression so citations resolve to what the model actually saw. A summary’s link to a source is not a substitute for checking the original passage. Also distinguish source entailment from external truth: a source can support a claim while itself being outdated or wrong.

Citation presence is a formatting property. Support precision asks whether cited links provide the required evidence under a declared policy, and coverage asks which claims have enough cited support. Jointly supporting passages and partially supporting links make the policy nontrivial. ALCE uses an entailment-based procedure with special treatment of combined citations; its exact metric should not be replaced silently by a simpler count. Automated checks can help triage, but human review and uncertainty labels remain necessary for ambiguous scope or partial support.

Create controlled records containing fresh values whose correct answer depends on the provided source. Change one relevant value and update the expected answer; a context-using system should follow that change under the task definition. Remove the only support and test the declared abstention or uncertainty behavior. Replace a citation with a similarly worded but non-supporting source to test attribution. Preserve token length, position, and unrelated content where possible, because a perturbation that changes several properties can produce a misleading diagnosis.

A changed answer alone is not a complete causal explanation. Random generation, contradictory instructions, lexical cues, or a corrupted prompt can also change outputs. Use matched controls and repeated trials under a recorded decoding policy, and inspect whether the change is correct in the intended direction. An unchanged answer can likewise reflect a valid alternative source or prior knowledge rather than definite evidence neglect. Remove those shortcuts in synthetic controls, while evaluating real tasks separately. Attention visualizations and citation syntax alone do not establish faithful source use.

Report unsupported claims, uncited claims, wrong source links, and unresolved judgments separately. One score can hide that a system cites accurately when it cites but leaves most important claims uncited. Inspect long-position and language slices, where truncation or reference resolution may differ. Re-evaluate after changing compression, context order, retrieval depth, or generation prompts; each can alter the available evidence and citation mapping. A quality improvement must preserve provenance and support rather than merely increase the number of bracketed references.

The probability lab illustrates a hypothetical independent support model only. Reset, set per-link support probability to 0.8 and link count to 10, predict mean supported links and probability all support, then Step: eight and approximately 0.1074. The rate and independence are assumptions, not labels inferred by the instrument. It does not read sources, run entailment checks, generate citations, or demonstrate actual source use. Correlated citation errors can make the binomial model inappropriate for a real answer.