Benchmark design and contamination

Define what counts as useful work. For training, time to an agreed validation quality can be more informative than raw steps per second, because an altered optimizer may need more steps. For inference, specify input and output lengths, concurrency or arrival process, latency objectives, and quality constraints. Closed-loop clients that wait for completion change offered load when the server slows. An open-loop arrival process asks a different question and needs explicit handling of queues and dropped requests.

Separate cold-start, compilation, warm-up, and steady-state measurements, then report the combination relevant to deployment. A kernel benchmark excluding transfers is legitimate for studying a kernel, but it cannot establish an end-to-end service gain. Use a strong baseline configured with comparable tuning effort. Record software versions and allowed algorithmic changes so a faster result does not quietly substitute a smaller model, looser tolerance, easier dataset, or lower quality threshold.

A single average conceals which workload changed. Publish per-workload results and the weighting rule before combining them. If fractions of total requests are known and each request has one service time, an arithmetic mean estimates expected service time for that mixture. A geometric mean of normalized speedups answers a different comparison question and does not by itself predict fleet capacity. Tail latency requires distributions, not an average of workload means.

Include adversarially inconvenient but representative cases: long contexts, skewed access patterns, sparse batches, and recovery after a failure. Avoid selecting only the workloads where the proposal wins. Distinguish a benchmark intended to isolate a mechanism from one intended to represent production traffic. The former can deliberately simplify conditions, while the latter needs evidence that its workload mixture resembles real demand. Both should state excluded regimes and the consequences of those exclusions.

Contamination occurs when evaluation information influences the system in a way that invalidates the intended independence claim. Exact duplicate examples in training are one route; near duplicates, answer explanations, benchmark-specific instruction tuning, and repeated manual prompt selection are others. Record dataset versions, collection dates, deduplication criteria, and tuning access. A hidden holdout or newly collected temporal test can reduce exposure, but temporal change may also alter task difficulty.

Membership or contamination detectors provide imperfect evidence. A likelihood-based detector can confuse familiar writing style or easy text with training membership, while failing to detect paraphrased exposure. Validate detector sensitivity and false positives on suitable controls rather than treating its score as proof about one example. Report both audited benchmark performance and uncertainty about exposure. If a test has become a development target, build fresh evaluation evidence instead of claiming that secrecy can be restored by renaming the split.