Check that the perturbation preserves the task

A semantic or spatial threat can use rotations, translations, lighting changes, controlled text edits, or another declared family. Define parameter ranges, interpolation, padding, cropping, and encoding. Those details change which pixels or tokens reach the model. A translation that removes the only relevant object does not necessarily preserve the original category, while padding can create artifacts that dominate a classifier. Save the transformed artifact actually evaluated, not just a parameter string that may render differently in another library.

Separate transformation feasibility from semantic validity. A point can satisfy an angle bound and still become ambiguous or label-changing. Conversely, a large pixel-norm displacement can preserve the intended category. Norm-bounded and semantic evaluations therefore address different sets rather than stronger and weaker versions of one universal task. Include any intersection explicitly when both constraints matter. Never present a certificate for one mathematical set as evidence for a different transformation family without a justified containment argument.

Use a task-specific rule, trusted simulator, or blinded human adjudication where appropriate. Document annotator instructions, expertise, disagreement handling, and the information available to adjudicators. The evaluated classifier should not be the sole validity oracle, because that would exclude precisely the cases on which it disagrees. A second model can assist screening, but its errors and shared biases do not establish ground truth. Preserve uncertain cases instead of silently assigning them whichever label favors the defense.

Validity assessment after candidate generation can create selection effects. Report how many candidates were generated, how many passed parameter constraints, how many were judged label-preserving, and how many of those failed. Also report case-level coverage: several valid failures for one original example should not become several independent failed examples in a robust-accuracy table. If a generator rarely finds valid candidates, its conditional failure rate alone cannot describe the overall performance of the generation-and-testing procedure.

Two individually acceptable edits may produce an invalid combination, such as a crop followed by a rotation that removes the target. Validate the final composed input, and record transformation order. For text, an apparently harmless word replacement can change negation, reference, or the requested output. Use the actual task semantics rather than a generic similarity threshold as the final criterion. A fixed embedding-similarity cutoff is a model-based proxy, not a proof that the label remains unchanged.

Compare clean and transformed performance using the same label policy and preserve the untouched examples for review. Report ambiguous or label-changing outputs in separate diagnostic categories so useful observations are not lost. A system may fail to recognize transformed objects, reject unusable inputs appropriately, or correctly change its answer after the task itself changes. Those outcomes have different implications. Reproducibility requires the validity protocol and adjudication evidence alongside attack code, because the semantic claim cannot be reconstructed from numerical perturbation size alone.