Preserve test independence across training, retrieval, and feedback channels

Define what an overlap check actually recognizes: identical normalized bytes, shared identifiers, n-gram matches, semantic candidates, or another documented relation. Retain the normalization version and the relevant training, retrieval, and test snapshots. Exact matches can be meaningful evidence, but paraphrases, translations, answer keys, or derived explanations can bypass simple string checks. Yang and colleagues demonstrate such limitations with rephrased benchmark samples. A nonmatch therefore means the declared detector found no match in the inspected material, not that all possible exposure has been ruled out.

Exposure is also different from proving memorization or quantifying a causal performance gain. A known-overlap subset can be easier or harder for reasons unrelated to exposure. Removing it changes the evaluation population and can change weights or difficulty. Report the inspected inventory, known-overlap rate, subgroup results, and uncertainty without presenting a filtered score as a perfectly decontaminated counterfactual. When training data are unavailable, state that limitation and use additional evidence such as fresh independently authored cases or collection-after-cutoff samples under a documented protocol.

Keep development material available for ordinary iteration while restricting confirmatory prompts, labels, and outputs to the authorized evaluation workflow. Separate the identities that submit a model, execute a test, inspect protected content, and approve a report where practical. Enforce dataset and purpose scope in the actual service and storage permissions, rather than relying on a folder name such as secret-test. Record who accessed which snapshot and why. Audit whether debug traces, retries, cached prompts, or generated annotations can flow back into training or prompt development.

Even aggregate results reveal information about a test. If those results drive repeated selection, the final model becomes dependent on that feedback despite never reading raw labels. Use a declared evaluation budget, independent confirmation, or a justified adaptive-analysis method with its assumptions; simple query counting is not itself a theorem of validity. Incident handling should preserve the exposure timeline and affected model runs, restrict further reuse, and determine which claims need a fresh independent assessment. Avoid silently replacing a compromised test and reporting the new score as though the protocol never changed.

Version access policies and overlap detectors with the evaluation artifact. Preserve an immutable source inventory or bounded evidence about what was available to inspect. Distinguish approved use of development examples from accidental exposure of confirmatory cases, and document changes after an incident. Fresh cases can improve independence but still require representative construction and correct scoring. A robust report names remaining unknowns and limits the claim accordingly instead of equating a private dataset, a successful scan, or a high score with uncontaminated general capability.

The probability lab illustrates only a hypothetical exposure-membership model. Reset, set both p=0.05 and n=20, predict the chance of at least one flagged item, then inspect one minus the zero-success bar. Under independent identical flag events it is 1−0.95^20≈0.64151. The model has no overlap detector, corpus, access policy, or training history. Real flags are often clustered and detector misses are unknown, so this probability cannot certify an evaluation. It simply shows how a small per-item chance can accumulate across a collection under an explicit toy assumption.