Release a robustness result that others can audit

Identify model weights by a content hash, architecture and code by immutable revisions, and the dataset by version plus ordered example IDs. Include preprocessing, input range, color order, normalization, output aggregation, rejection policy, precision, and evaluation mode. Save the exact external input representation used to measure perturbations. A decoded image produced by a different library or a normalization performed twice can change the evaluated classifier while leaving a checkpoint filename unchanged.

Pin attack implementations and their resolved configurations. A suite name is insufficient because defaults and components can change over time. The official AutoAttack README at the cited commit supplies a concrete implementation reference; record the chosen norm, epsilon, version preset, seed, batches, and any custom changes. The pinned RobustBench convention restricts its standardized norm benchmark to generally nonzero input gradients, deterministic forward passes, and no optimization loops. For other defense mechanisms, add defense-aware tests rather than assuming a generic configuration covers every case. Freeze the final protocol before reporting held-out results.

For a fixed deterministic threat, combine attacks by taking the union of valid failed example IDs. Averaging individual attack accuracies discards overlap structure and can overstate survival. Count clean errors as failures under the ordinary robustness definition, then preserve all newly discovered feasible failures. Keep attack provenance so reviewers can see whether a gradient-free method found cases missed by derivative-based methods, or whether one attack merely repeated failures already found by another.

Release per-example clean correctness, tested-candidate status, actual norm, failure predicate outcome, consumed queries, and relevant random seeds. Candidate tensors or reconstructible artifacts allow independent forward verification when distribution permits. Invalid candidates should be recorded as invalid rather than silently counted as either attack successes or robust survivors. A crashed evaluator or missing result is an incomplete case and must affect reported coverage. Separate coverage from the score over completed cases.

Place clean accuracy, finite-suite survival, certificate-derived accuracy, and semantic validity coverage in separate labeled columns. A certificate has its own classifier, radius, confidence procedure, and assumptions; it cannot be merged with unrelated empirical candidates into an unqualified robust score. For stochastic attacks, report the repeated-evaluation protocol and statistical uncertainty. For dataset-level uncertainty, explain the sampling population and any dependence rather than deriving confidence from attack randomness alone.

Include regression fixtures for preprocessing, norm checking, candidate replay, and failure aggregation. Recompute summaries from saved per-example records so a reviewer can reproduce every denominator without rerunning expensive searches. Compare releases on common example IDs and inspect changed decisions after artifact updates. Keep the claims scoped to the tested or certified threat. Public release makes evidence inspectable and reusable, while broader robustness still requires additional experiments or a theorem that actually covers the proposed extension.