Systems claims and falsifiable hypotheses

Identify the system version, proposed change, comparison implementation, workload distribution, machine configuration, and measurement boundary. Mean request completion time, aggregate job makespan, and throughput at a latency target are different estimands. An optimization can improve one while worsening another. Define when timing starts and stops, whether compilation or warmup belongs to the user-visible operation, and which failed requests remain in the outcome. A description sufficiently precise for another investigator to implement the same measurement is more useful than an impressive adjective.

For paired runs, define a difference with a consistent sign: baseline time minus candidate time makes a positive value an improvement. Pairing requires a meaningful shared experimental unit, such as the same input and machine-time block; adjacent raw requests are not automatically independent pairs. Declare a practical threshold separately from zero. A tiny positive difference can be statistically distinguishable while irrelevant to deployment, and an important point estimate can still have too much uncertainty to establish the promised minimum benefit.

Write down which outcomes would contradict the claim: a confidence interval incompatible with the target, a correctness failure, or a regression in a required guardrail. Declare exclusions and stopping rules before seeing favorable results. Exploratory experiments are valuable, but label subsequent hypotheses as exploratory and confirm them with fresh evidence when possible. Repeatedly changing the workload until an effect appears selects on the answer. Reporting the final test as if it were the only planned comparison conceals that selection.

An interval describes uncertainty under its construction assumptions. For an illustrative large-sample normal approximation to an independent paired mean, estimate its standard error from paired differences, not from two unrelated lists of error bars. Heavy tails, serial dependence, or a small number of independent machines can require another analysis. A confidence level is a property of a repeated-sampling procedure, not a posterior probability that a fixed parameter lies inside this particular interval. Statistical calculation does not repair biased workload selection or unequal treatment contracts.

Randomized assignment within suitable blocks can support a causal contrast when treatment is implemented consistently, outcomes are comparable, and interference or carryover is controlled. A checklist, an artifact badge, or an interval alone establishes none of those assumptions. State whether evidence is a randomized intervention, an observational comparison, or a deterministic model calculation. Also distinguish the local measured effect from its transfer to other workloads. A careful experiment can estimate the wrong target precisely if the stated population does not match the intended decision.

The function instrument makes a narrow falsification exercise possible. Reset, set both a=1 and x=1, predict y=1, then Play or Step. Separately Reset, set both a=1 and x=1.5, predict y=2.25, then Play or Step. A proposed rule y=x agrees with the first point and disagrees with the second. The instrument evaluates the fixed noiseless formula y=ax²; it supplies neither performance samples nor causal identification. Its purpose is to show why a hypothesis needs tests where competing explanations predict different observations.