Measure robustness under a real information budget

A perturbation that fools a surrogate may fail on the target because their boundaries differ. Check it through the target’s full preprocessing and output rule, using the target threat’s coordinates. Report transfer success over a clearly specified set, such as all target-clean-correct examples, rather than only examples selected after observing target attack outcomes. If you condition on surrogate success, publish that selection fraction as well; otherwise the number hides how many cases the surrogate never attacked successfully.

Using several surrogates or candidate objectives can improve empirical coverage, but each target evaluation still consumes the declared budget. A genuinely transfer-only protocol can generate candidates without target feedback and submit a fixed candidate list. Selecting the next surrogate candidate after inspecting target scores instead becomes an adaptive query procedure. Both are legitimate test settings, but their resource and knowledge contracts differ and should not be compared as if they were identical.

A central coordinatewise finite-difference estimate in d dimensions requires two score queries per coordinate before any candidate validation. Random-direction estimators use fewer directions but introduce their own variance and smoothing bias. Label-only responses do not directly supply a differentiable loss value, so a score-based formula cannot simply be applied to integer labels and interpreted the same way. Use a method compatible with the interface and state exactly which target outputs it consumes.

For randomized outputs, repeated calls can estimate a score or success probability, multiplying query cost. Batched requests still contain multiple queried inputs unless the threat explicitly budgets batches instead. Include initial classification checks, retries that reached the model, rejected candidates, and final confirmations according to the predefined policy. Keep a per-example ledger of consumed calls. Runtime limits and rate limits are practical constraints, but they are not interchangeable with a fixed total number of model evaluations.

Plot cumulative valid failures against consumed queries, retaining successful candidates. Report clean errors separately and keep the denominator fixed across budgets. A case with no discovered failure after fifty queries is a fifty-query search survivor, not a certified point. Success can depend on score precision, hidden randomization, input encoding, and response aggregation, so freeze those interface details with the model version. A server update during the run otherwise changes the target while the query counter continues.

Include complementary methods whose assumptions differ. A transfer probe can expose a broken white-box gradient path; a score-based random search can find failures that local derivatives miss. The Square Attack paper supplies one example of structured score-based search, while limited-query work studies estimators under restricted interfaces. These methods are references for designing controlled tests, not promises that a given budget finds the worst case. Release per-example outcomes and counts so others can distinguish query efficiency from merely attacking an easier selected subset.