Evaluate the defense that is actually deployed

If the service predicts with a classifier after a transform, every candidate must be evaluated after that transform. If it rejects suspicious inputs, define whether rejection is a permitted safe outcome and include that rule in the success predicate. Optimizing only the classifier loss may find candidates that the detector rejects, while optimizing only detector acceptance may leave classification correct. A useful adaptive objective must address the actual conjunction of conditions required by the declared failure event.

A nondifferentiable operation may require a backward-pass differentiable approximation to obtain a search direction. Preserve the genuine operation in the forward pass and substitute only a justified derivative approximation during candidate generation. Replacing the transform in both directions evaluates a different system. The approximation is a heuristic, so inspect several surrogate choices and always verify final candidates through the original forward path. A successful feasible counterexample remains valid regardless of whether its search derivative was exact.

For randomized preprocessing, a fixed-seed gradient can optimize one realization that does not represent the service’s output distribution. An expectation-over-transformation estimator averages loss gradients over independently drawn transformations when the relevant derivatives and differentiation-under-expectation assumptions hold. The expected surrogate loss is still different from misclassification probability. Use it to search, then separately estimate the declared failure probability or aggregated-output failure on fresh random draws.

Count every transformation sample in the attack’s resource budget. Twenty iterations with eight sampled forward-backward pairs per iteration use 160 such pairs, before candidate validation. Compare estimator variance across sample counts and investigate unstable directions. Reusing random draws can reduce variance in some comparisons but changes dependence and must be documented. Do not tune a candidate on a set of seeds and then present its performance on those same seeds as an unbiased estimate of future randomized behavior.

Read the defense’s claimed mechanism and identify why ordinary attacks might fail computationally. Optimization loops, hard quantization, randomization, or unusual logits each suggest different diagnostic experiments. Adapt losses, estimators, and initialization to that mechanism. Keep norm, epsilon, attacker access, and success policy fixed unless a separate experiment explicitly changes them. Showing failure under a larger threat does not refute a smaller stated guarantee, while using an artificially weaker attacker can inflate apparent robustness.

Separate development cases for attack design from the held-out report set. Save the selected adaptive protocol and its exact budgets before the final evaluation. Preserve intermediate evidence when a defense-aware search improves on a generic suite, including candidate feasibility, true forward outputs, and the role of the approximation. An automated suite provides a reproducible baseline, but a novel defense may need additional analysis. No finite collection of adaptive attack failures converts the search into a proof of the universal robustness property.