Duplicate detection needs a representation, a similarity rule, and a retention policy

An exact duplicate can mean identical raw bytes, identical normalized text, or an identical token sequence under a pinned tokenizer. These criteria differ. Use a digest to group candidates efficiently and verify the required equality where the contract calls for exactness, accounting for hash-collision assumptions. Keep length and representation metadata. A table of short boilerplate fragments can be useful to remove repeated spans without deleting an entire otherwise distinct document; conversely, whole-document hashing misses a copied article embedded inside a different wrapper.

The original training-data deduplication research studies repeated documents and substrings and their effect on memorization and evaluation. Its findings motivate a measurable pipeline, not a universal claim that all similar text should disappear. Repeated definitions, code templates, or legally required notices can be meaningful in some contexts. Define whether the goal is reducing redundant exposure, preventing train/evaluation leakage, or clustering source copies. The same threshold need not serve every purpose, and a quality-based representative choice should not discard source provenance or evade the documented use policy.

MinHash estimates Jaccard resemblance by comparing minima under suitable random permutations or their practical hash approximations. With an ideal minwise permutation, equal minima occur with probability J. A banded sketch can raise the chance of proposing sufficiently similar pairs while limiting all-pairs comparisons. The familiar candidate formula assumes independent rows grouped into bands. Practical hashes, collisions, finite sketches, and data distribution affect behavior, so a candidate match is not an exact similarity measurement and a noncandidate is not proof of dissimilarity.

Verify candidate pairs with the intended similarity or containment rule before applying the removal policy. Approximate pairwise similarity is not transitive: A can match B and B match C while A does not match C. A connected-component cluster can therefore merge a broad chain, and keeping just one representative may remove more diversity than the pair threshold suggests. Test threshold behavior and cluster sizes on trusted pairs, including templated but meaningfully distinct documents. Record cluster identifiers, similarity evidence, and representative selection rules for later audit and source retirement.

A bounded cache of recently seen hashes provides only a bounded-memory duplicate check. Once a digest is evicted, an old duplicate can appear new unless another index retains it. Global corpus deduplication needs a method whose retained or partitioned index covers that declared scope, such as sorted hashes or an appropriately designed external structure. Distributed work partitioning must route equivalent candidates consistently and coordinate representative decisions. Count removed documents and tokens separately, and inspect whether one source or language loses disproportionate useful material.

The cache lab isolates forgotten exact identities. Reset, explicitly set BOTH C=4 and s=4, predict twelve hits among N=16 requests at t=0, then Play or Step. Reset and re-set BOTH C=3 and s=4; predict zero hits because each repeated identity follows eviction. N=min(4096,16+floor(2t)); edits retain t and recompute from empty. This LRU toy has no text, hashing, MinHash, pair verification, or global corpus index. A hit represents remembered identity only; it cannot decide which similar documents are interchangeable.