Experiment design and statistical power

The experimental unit is the independently assigned entity: perhaps a complete training run, a machine-day, or a randomized service request. Ten thousand steps inside one run do not automatically provide ten thousand independent replications. They share initial weights, cache state, hardware, and often strongly correlated load. Treating them as independent understates uncertainty. Record nested measurements for diagnosis, but estimate uncertainty at the assignment level or use a model that represents the dependence.

Blocking can reduce noise by comparing alternatives within the same hardware or time window. A paired experiment might run both schedulers on each of several workload seeds, with execution order randomized within each pair. Pairing is useful only when the pairing variable induces meaningful correlation. Avoid always running the baseline first, because warm caches, thermal drift, or background traffic would then be confounded with the treatment. Define exclusion rules and failure handling before discovering which runs favor the proposal.

Power is the probability that a specified test rejects its null under a specified alternative. It depends on effect size, variability, sample count, and the analysis procedure. Decide the smallest improvement that would justify adoption, rather than choosing the effect after inspecting results. Estimate variability from an independent pilot or conservative historical evidence. If the pilot is tiny or the workload has heavy tails, a normal approximation can be optimistic; simulation or a design-specific calculation may be needed.

The usual two-sided normal planning expression for paired mean differences assumes independent, approximately normal differences and a known or reasonably estimated standard deviation. It is an approximation when variability is estimated. A small-sample t calculation may require more runs. Repeatedly checking significance and stopping at the first favorable result also changes the error rate. Predeclare a fixed sample plan or a valid sequential procedure. Detecting a nonzero effect with power calculated at δ does not establish an improvement of at least δ; an adoption rule should compare an appropriate effect interval with its practical threshold.

Present the effect estimate and an interval, along with individual run outcomes, rather than only a p-value. A statistically detectable one-millisecond gain can still be operationally irrelevant. Conversely, an inconclusive result is compatible with a useful improvement if the experiment is too noisy. Report quality failures and resource costs alongside speed; a scheduler that silently drops difficult examples has changed the task. Distinguish an intention-to-run comparison from an analysis restricted to successful runs.

If many configurations were explored, identify that stage as exploratory and reserve fresh runs for confirmation. Include negative outcomes and sensitivity to workload composition. An interval from one machine family describes uncertainty within that experiment, not automatic generalization to every cluster. The next experiment should challenge the weakest assumption: alternate hardware, a different load regime, or independent implementation. This makes the research program cumulative rather than a sequence of isolated favorable screenshots.