Language labels change the corpus mixture when their errors are uneven

Specify whether the unit is a whole document, paragraph, or segment, and whether the desired result is one label, several labels, or an unknown outcome. A page with one English menu and a long article in another language should not necessarily inherit the menu's label. For code-switched text, a single majority label can conceal useful minority-language spans. Keep the model version, input transformation, label inventory, and segmentation policy so the same raw text can be traced to its admitted representation and routing decision.

The official fastText language-identification documentation describes models recognizing 176 languages and expecting UTF-8 input. That documented coverage is not a claim of equal accuracy across every supported language, dialect, genre, or input length. Languages outside a closed label set may still be assigned an available label rather than automatically recognized as unknown. Establish an abstention or review policy using representative labeled examples. A confidence threshold should be justified by measured error and retention in the relevant setting, including unfamiliar scripts and mixed-language documents.

A language filter can distort a mixture through unequal rejection even when the surviving labels look plausible. If one group has more short messages or nonstandard spelling, a global threshold may remove a larger share of its useful text. Measure retained and rejected samples by language, length, source, and relevant varieties using a defensible annotation process. The C4 case study documented disproportionate removals from and about minority individuals under its blocklist filtering; that particular result motivates checking selection effects rather than assuming every filter has identical consequences.

For fixed true groups, multiply each initial share by its retention probability and renormalize by total retained mass. This accounting does not require the filter to explicitly use a language label. A quality score, boilerplate heuristic, or source restriction can also change language composition. Document the measurement unit: ten long documents can outweigh hundreds of short ones in tokens. Do not correct a suspected bias simply by trusting noisy predicted labels as ground truth; first estimate classification and retention errors on an independently reviewed sample.

Separate ranking accuracy from probability calibration. Among items assigned confidence near 0.9, approximately ninety percent correctness would support calibration for that bin and population; a high average confidence alone would not. Inspect confidence reliability by length and language where sample sizes support it, report uncertainty, and keep calibration data separate from the final evaluation. Changing extraction or source composition can change those relationships. If uncertainty is concentrated in valuable low-resource material, preserve it for targeted review or separate routing rather than silently discarding it as low quality.

The probability lab uses an assumed accuracy, not a classifier confidence score. Reset, explicitly set BOTH p=0.8 and n=20, predict sixteen correct labels on average, then Play or Step. Reset and re-set BOTH p=0.9 and n=20; predict eighteen. Edits retain t; playback highlights exact binomial outcomes without labeling documents. The model assumes independent identical correctness events and cannot represent language-specific confusion, code switching, calibration drift, or unequal retention. Use it to distinguish an assumed population accuracy from the count observed in one small audit.