Training data poisoning and backdoors

Poisoning changes the training process through manipulated inputs or related artifacts. A backdoor is a learned behavior associated with a particular condition or trigger; it can originate through poisoned data or compromised model supply. Distinguish attacker-controlled examples, label changes, preprocessing, and directly supplied weights. These capabilities affect what defenses are relevant and prevent claims that one data filter protects every possible supply-chain path.

BadNets and clean-label poisoning research establish that satisfactory ordinary behavior can coexist with targeted failures in studied settings. This does not mean every model contains such a failure or that any particular defense always fails. The practical lesson is to include targeted evaluations and provenance checks alongside clean accuracy. Keep research scenarios explicit: dataset, model access, training method, and attacker knowledge strongly affect results.

Record source identities and collection times, constrain dataset write permissions, review labeling changes, and publish immutable dataset manifests. Quarantine suspicious inputs for authorized analysis rather than automatically deleting rare examples that might represent legitimate users. Deduplication and outlier checks can reveal anomalies, but an unusual record is not necessarily malicious and an ordinary-looking record is not necessarily trustworthy.

Keep validation data independent of the collection process being assessed, and protect evaluation code and labels as assets. Inspect changes by source, subgroup, and task behavior, not only overall accuracy. A model release should connect to dataset and preprocessing identities, making it possible to compare and roll back. A valid artifact signature authenticates its producer; it cannot establish that its training data was free of poisoning.

For a hypothetical contaminated fraction f and independent sampling with replacement, auditing n examples misses all contamination with probability (1 − f) to the power n, assuming perfect recognition when sampled. Sampling without replacement from a finite dataset instead follows a hypergeometric calculation. Real review is imperfect and poisoning may cluster by source, so random sampling alone can substantially overstate practical coverage.

Combine random audits with source-based reviews and targeted behavioral tests derived from the threat model. Document reviewer sensitivity and false positives, and preserve approved evidence for incident response. If a suspected model is already serving, isolate the release, retain relevant manifests, and compare against a known accepted version before resuming use. Retraining without fixing the compromised data path can recreate the same problem.