A quality filter needs a target, an audit, and a threshold tradeoff

Write a rubric that reviewers can apply to representative material: for example, coherent relevant content with sufficiently preserved structure for a specific training objective. Separate this criterion from source admission, sensitive-data policy, and benchmark overlap. A document can be informative yet still fail another release requirement. Record borderline examples and reviewer disagreement rather than treating every uncertain case as an annotation error. If the rubric changes, the old labels and threshold no longer measure exactly the same target, so version the annotation instructions alongside the classifier.

Build evaluation samples from both accepted and rejected regions, with enough coverage of rare sources and formats to detect large systematic mistakes. A sample consisting only of accepted text estimates neither the useful material lost nor the full confusion table. If sampling rates differ by score bin or source, use the design's weights when estimating population metrics. Keep threshold tuning separate from the final held-out audit. Document the human or model-labeling process, since a judge can inherit the same style preferences or blind spots as the filter it evaluates.

Calibration asks whether predicted probabilities agree with observed rubric outcomes in a stated population. A reliability bin with mean predicted probability 0.9 and eighty positives among one hundred independently labeled examples has an observed frequency of 0.8 and a gap of 0.1. That estimate has uncertainty; it also says little about a tiny subgroup hidden inside the bin. The calibration work of Guo and colleagues studies such probability reliability and post-processing methods. It does not imply that every modern corpus filter can be calibrated once and trusted under arbitrary distribution changes.

A threshold chooses a tradeoff among retained volume, useful-material recall, false positives, and compute cost. Good ranking can exist with poorly scaled probabilities, and calibration can improve without changing ranking. Raising a threshold need not improve every downstream task, particularly if it concentrates one genre or removes rare but valuable text. DCLM provides controlled dataset experiments for comparing curation strategies under training and evaluation recipes. Follow that experimental logic: compare relevant alternatives with controlled budgets, protected evaluation data, and reported coverage instead of treating a score histogram as a model-quality experiment.

Compare source, language, length, and genre distributions before and after filtering, and manually inspect concentrated losses. Deduplication or extraction changes can alter the score distribution even when the quality model's weights stay fixed. Freeze model artifacts, tokenizer, input truncation, thresholds, and reason codes in the release manifest. Use small controlled training studies to assess downstream effects, then check transfer to the intended scale. A threshold that worked for one architecture, token budget, or evaluation suite should not be promoted as a universally optimal rule without new evidence.

The network lab shows a threshold acting on a fixed mathematical score, not a trained quality model. Reset, explicitly set BOTH x=1 and w=2, predict sigmoid(1.5)≈0.8176, then Play or Step. Reset and re-set BOTH x=0.5 and w=2; predict sigmoid(0.5)≈0.6225. A paper threshold of 0.8 would accept only the first. Edits retain t. The fixed bias is −0.5; there are no documents, labels, fitting, or calibration. A sigmoid-shaped output alone cannot establish that 0.8176 means an eighty-two percent chance of useful text.