Agent comparisons need matched resource rules and complete execution cohorts

Record model input and output tokens, tool calls, tool compute, elapsed time, parallel workers, retrieval, verifier work, and human intervention where relevant. A wall-clock limit can reward parallelism that consumes much more total compute, while a token limit can hide expensive browser or code-execution actions. Define what happens at each limit and whether unfinished effects are reconciled before grading. Keep policy revision, hardware, service versions, and observation interface fixed or explicitly report differences. A benchmark name alone does not identify these resource conditions.

Do not infer a bounded-run outcome by deleting the expensive suffix of a completed trace unless that counterfactual is actually valid. A policy operating with a shorter budget may choose different actions, and a truncated task may leave asynchronous effects unfinished. Run the intended bounded policy and grade its state under the declared termination contract. If setup failures receive a separate category, predeclare that rule and report its count alongside scheduled and graded attempts. Excluding timeouts or invalid outputs after seeing them creates a success-only cohort rather than a fair comparison.

Use an explicit cost model with its time and price version when converting resources into one scalar. Tokens, tool actions, and elapsed time are not naturally interchangeable units. Report the raw resource vector as well as any weighted cost so another reader can apply a different constraint. For repeated-candidate systems, include failed candidates and the mechanism used to select or verify the final one. A hidden evaluation oracle is not a free deployment component, and a system that cannot recognize its successful candidate may not achieve its reported one-of-k rate operationally.

Total cost per verified success divides all consumed cohort cost by verified successes, not merely the cost of successful runs. It is undefined when there are no successes. Even a properly computed ratio does not reveal tail latency, critical failures, or whether one system dominates another across every budget. Plot or tabulate success under several matched budgets where useful and identify tradeoffs rather than forcing a universal ranking. Selection of a preferred operating point should reflect the actual application objective and the limits of the measured task distribution.

Use the same task versions and initial-state rules, with repeated seeds or conditions paired where that design is justified. Keep task families together in uncertainty analysis and disclose benchmark selection or tuning. A higher rate under twice the budget is evidence for that operating point, not necessarily a better policy at the smaller budget. Report interrupted and unfinished work, environment errors, and whether results come from one attempt or a multi-attempt strategy. AgentBench and SWE-bench provide task-driven evaluation settings, while the experiment's resource and cohort definitions still need explicit reporting.

The optimization lab illustrates comparison under an equal update count. Reset, explicitly set BOTH η=0.5 and x₀=2, predict x=0.125 after four updates, then Play or Step four times to t=2. Reset and re-set BOTH η=2.1 and x₀=2; predict x=2.9282. Edits retain t; n=min(1000000,floor(2 t)), and values beyond |x|=4 are explicitly out of range. This fixed quadratic descent has no agent tasks, token costs, verifiers, or deployment score. Equal modeled updates expose a parameter effect here without normalizing a real agent experiment.