Canonical text is a versioned representation choice

Start from a named raw artifact and its byte hash. Decode with an explicit encoding and error policy, then extract relevant content under a versioned rule. A crawl's raw response, metadata, and pre-extracted text are different artifacts; Common Crawl's overview documents their roles. An extractor may remove menus and boilerplate while preserving article paragraphs, but it can also lose tables, mathematical notation, or code indentation. The desired representation depends on the training task and content type, so one generic text-density threshold should not silently define quality for every source.

Unicode normalization changes how equivalent or compatible characters are represented. NFC addresses canonical equivalence, while NFKC also folds compatibility distinctions that may matter in identifiers, mathematics, or styled text. The Unicode standard defines the transformations; it does not determine which is appropriate for a particular corpus. Whitespace folding and lowercasing add further choices and can change code or case-sensitive names. Record each step and test representative fixtures instead of assuming that fewer distinct byte strings always means a better or more faithful dataset.

Hashing raw bytes identifies one captured representation, while hashing extracted normalized text defines another grouping criterion. Two raw pages with different navigation can share extracted text; two visually similar pages can differ after a meaningful symbol or table cell is preserved. Store separate raw and derived identifiers rather than overwriting the first with the second. A cryptographic digest supports integrity and practical content addressing under its collision assumptions, but the digest does not certify extraction quality, establish permission, or anonymize a document against membership tests.

Extraction yield should be measured with compatible units and a labeled target. Retained characters, decoded bytes, documents, and tokenizer tokens are different quantities. Removing duplicated boilerplate may lower volume while improving useful training content, whereas dropping every short mathematical answer can damage coverage. Use human-reviewed or otherwise trusted examples across formats and languages. Inspect both the retained and discarded portions, and keep reason codes so a regression can be attributed to a particular rule rather than hidden behind one final token-count change.

Create fixtures for malformed markup, missing encodings, nested lists, tables, code, formulas, repeated sidebars, and multilingual text. Verify deterministic outputs under pinned dependencies and inspect whether the intended content survives. A parser can deterministically produce the wrong article, so reproducibility is necessary but not sufficient. If extraction rules change, derive a new content version and compare retention and downstream performance rather than mutating an already referenced corpus shard in place. Preserve lineage to the raw artifact even when raw access is restricted by the project's storage policy.

The probability lab illustrates defects in an audit sample rather than extracting text. Reset, explicitly set BOTH defect probability p=0.05 and sample count n=20, predict at least one defect with probability about 0.6415, then Play or Step. Reset and re-set BOTH p=0.1 and n=20; predict about 0.8784. Edits retain t; playback highlights exact independent counts. The model has no HTML, Unicode, tables, or tokenizer. A clean sample does not prove every extraction is faithful, and repeated templates can make audit errors correlated rather than independent.