An overlap test measures a threat model, not the absence of contamination

Inventory the evaluated unit and plausible leakage paths: full prompt-answer pairs, answer keys, long passages, code solutions, near-duplicate questions, and documents from which examples were generated. General background knowledge is not automatically equivalent to exposure to a specific test answer. State the distinction in the evaluation protocol rather than assuming one universal rule. The C4 documentation study found examples from evaluation datasets in a web corpus, while Lee and colleagues examined train-test overlap and deduplication. Those observations establish realistic leakage mechanisms, not that every shared phrase necessarily explains a model's score.

Keep raw and normalized query forms under a versioned matching process. Normalization can improve recall for punctuation or formatting changes but also erase meaningful distinctions and increase false positives. Split source families or duplicate clusters before assigning ordinary development and test partitions when those relationships would leak the target. Temporal cutoffs help only if source dates and later copies are understood; an old-looking document can have a newer edit containing benchmark material. Reserve an independent final evaluation and avoid repeatedly selecting filters or mixtures to maximize its score.

Use complementary mechanisms appropriate to the threat: exact substring indexes for verbatim spans, approximate candidate retrieval for edited text, and carefully evaluated methods for paraphrases or translations. Verify candidate matches against the intended overlap rule and inspect false positives. A standard instruction such as choose the best answer can occur across unrelated tasks, whereas a rare solution and its reasoning may be informative evidence. Record detector versions, thresholds, preprocessing, benchmark versions, and which training records were excluded. Near-duplicate corpus removal alone does not guarantee the evaluation boundary was examined.

A negative search is conditional on corpus coverage, representation, retrieval recall, and the chosen threat model. It cannot establish that no paraphrased or unindexed answer was seen. Evaluate the detector on known inserted overlaps where feasible, while recognizing that synthetic insertions may be easier than natural leakage. Keep benchmark content and audit artifacts under appropriate access controls so the test itself is not distributed into another training pipeline. Report limitations plainly, including undetected forms, rather than replacing a bounded engineering test with the unqualified label contamination-free.

Disclose full-set results alongside a clearly defined overlap-excluded analysis when appropriate, including counts, thresholds, and uncertainty. Removing detected examples changes the test distribution; the resulting number should not silently replace the original benchmark's published metric. Overlapping items can differ in difficulty or subject matter, so higher accuracy on them is observational evidence rather than a causal estimate of memorization. Stronger causal claims need controlled training comparisons or other justified designs. Maintain a reproducible exclusion manifest so reviewers can distinguish changed data from changed scoring.

The probability lab illustrates an ideal detector's chance of missing every known overlap. Reset, explicitly set BOTH detection probability p=0.8 and n=5, predict P(K=0)=0.00032, then Play or Step. Reset and re-set BOTH p=0.5 and n=5; predict 0.03125. Edits retain t and playback highlights exact binomial counts. These are independent detection trials with a stipulated p, not searches through text or estimates from an empty report. Shared paraphrase blind spots can make all misses correlated and invalidate this reassuring independent calculation.