Baseline equivalence and tuning fairness

Check that both configurations produce equivalent accepted outputs at the required accuracy, consistency, and failure semantics. A database that omits durability or a model server that reduces output length performs a different task unless the study explicitly treats that tradeoff. Match data preprocessing, request definitions, timeouts, warmup policy, and accounting of compilation or maintenance. If a feature is unsupported, report the limitation rather than silently removing those requests from only one arm. Correctness guardrails remain part of every timed configuration.

Resource parity depends on the question. Equal accelerator count, equal rented cost, and equal power cap define different studies. State hardware models, usable memory, host resources, network placement, and concurrent tenants. Software defaults are not necessarily comparable because libraries choose different thread counts, batching policies, and precision. A tuned baseline should be a credible implementation of the same task under the declared budget. Also report absolute outcomes, since a large ratio against a slow baseline may say little about an achievable deployment.

A fixed number of trials does not imply equal effort when trial durations or machine sizes differ. Sum the relevant resource consumption, such as accelerator-hours, and separately report wall-clock deadlines or human intervention when those matter. Include failed and interrupted tuning trials according to a declared policy. Equal budgets are a transparent experimental choice rather than proof that two search spaces are equally difficult. Publish search ranges, algorithms, seeds, stopping rules, and the selected configurations so the comparison can be challenged.

Choose configurations on development workloads and evaluate them on held-out workloads or fresh runs. Selecting the fastest noisy trial and then reporting that same measurement creates selection optimism. Re-evaluate the selected baseline and candidate with a comparable protocol. If a candidate received months of developer specialization while a baseline received one afternoon, disclose the asymmetry and seek expert feedback where feasible. A fair end-to-end tuned-system comparison still bundles many differences; it does not isolate which algorithmic component caused the gain.

Keep a configuration table containing revisions, flags, compilation options, dataset preparation, search budget, and measured execution boundary for each arm. Explain why the baseline is relevant and whether a newer implementation changes the conclusion. Present quality-resource curves when no single setting makes every constraint equal. An equal-cost result can be useful even if absolute latency differs, provided the claim names that decision. Distinguish a study of attainable deployed performance from a controlled experiment that intentionally freezes most implementation choices.

The optimization instrument illustrates an equal-update comparison without implying equal real compute. Reset, set both η=0.5 and x₀=2, predict x=0.25 after three Steps, then Step three times. Separately Reset, set both η=0.25 and x₀=2, predict x=0.84375 after three Steps, then Step three times. The exact recurrence is x_n=x₀(1−η)^n on one quadratic. Equal iterations are visible here, but this toy supplies no training-time, quality, tuning-search, or hardware evidence for an actual optimizer baseline.