Deduplicate before trusting a split

Exact hashing can detect identical canonical records, but canonicalization itself needs a contract: whitespace or formatting changes can be meaningful for code and structured-output tasks. Near-duplicate checks find similar strings or token sequences, while semantic grouping may connect paraphrases with little literal overlap. No single similarity threshold perfectly separates useful variation from leakage. Preserve source IDs and transformation lineage so generated variants can remain linked even when their text differs substantially.

Deduplication can reduce repeated exposure and memorization risk, but it is not a complete privacy guarantee or proof that a benchmark is uncontaminated. A model may have seen related content during pretraining, and a fine-tuning dataset can contain sensitive information once without any duplicates. Review provenance, permitted use, and task quality separately. Record which duplicates were removed and why, including effects on task and language distributions rather than silently letting a large source dominate the remaining mix.

A conversation-level split keeps turns from the same conversation together. A source-document split keeps derived questions and answers together. A template-family or task split can measure a broader generalization goal. The right unit depends on the claim: new rows from known tasks are different from unseen task families. Keep validation for model and hyperparameter selection and preserve a final test for the stated use, with an audit trail of any later contamination discoveries.

Avoid using test labels to decide cleaning thresholds and then reporting the same test as untouched evidence. Contamination checking may legitimately identify overlap, but its policy and disposition should be documented. Related groups can have unequal sizes, so a target proportion of groups does not necessarily produce the same proportion of rows or tokens. Report all three counts and the resulting task balance. Downstream loss normalization can amplify any remaining long or frequently repeated group.

Run exact and chosen near-duplicate checks across split boundaries after formatting, while also checking original source and conversation IDs. A template that injects an answer into the prompt can leak within an individual example even when group IDs are disjoint. Evaluate deduplication samples manually for false merges and missed variants. Retain reproducible hashes and rules without publishing sensitive contents. A clean split is evidence for a stated boundary, not permission to claim universal unseen-task performance.

The probability lab can show the row-assignment assumption. Reset and set both p=0.8, n=10, then predict mean eight rows in train. Counts zero and ten mean the group stayed in one split;interior bars mean it crossed. The exact binomial model assumes independent row assignment, not a fixed 80/20 quota, and Play highlights outcomes rather than splitting data. It does not perform text deduplication or semantic grouping;use it to explain why related records should be assigned together when that matches the evaluation goal.