Repeated runs and variance decomposition

Builds can contain executions, and executions can contain timed iterations. Data-center studies may add machines, allocation windows, or clusters above those levels. Record identifiers for each observation’s parents. When a treatment is assigned to an entire process, iterations within that process are not independent treatment assignments. Warmup and steady-state behavior also require a declared policy: discarding initial iterations after inspecting each run’s favorable region can select the answer. Retain the original observations and document any justified stabilization rule.

A simple balanced random-effects model represents one run-specific offset plus independent within-run noise. The run offsets are independent across runs, the residuals are independent across iterations and runs, and the two families are independent. These assumptions are illustrative, not automatic properties of benchmark data. Autocorrelation, machine drift, unequal variances, and heavy tails may require richer models or different estimators. A variance component associated with builds or hosts must be modeled at that level rather than folded into a fictitious independent iteration count.

Under this balanced model, averaging k iterations reduces within-run noise, but it does not average away the offset shared by all iterations of one run. Independent runs reduce both components in the overall mean. This explains why collecting more iterations in a single process can reach diminishing returns. Pilot measurements can estimate where uncertainty is concentrated, subject to their own sampling error. Use that evidence with startup, rebuild, or allocation costs to plan repetition rather than applying a fixed count to every benchmark.

Allocate independent units to both comparison arms and preserve meaningful pairing when they share inputs or machine-time blocks. Analyze paired differences at the level where pairing actually occurs. For a bootstrap, resample the appropriate independent clusters or use a hierarchy-aware method; resampling individual requests ignores a common run effect. Confidence intervals for a mean do not describe the full latency distribution or the uncertainty of a high percentile. Report the estimand, variance model, unit counts, and method together.

Publish run-level summaries and enough raw data to inspect drift, multimodality, and failures. A minimum execution time may answer a best-case question but cannot silently stand in for a representative mean. Report any exclusions, machine replacements, or unsuccessful starts. If a large allocation permits only a few independent runs, state the resulting limitation rather than multiplying its internal samples into false certainty. Repetition improves precision under an appropriate sampling model; it cannot remove systematic bias shared by every run.

The sampling instrument isolates only independent Bernoulli averaging. Reset, set both p=0.5 and n=25, predict mean 0.5 and standard deviation 0.1 for K/n, then Play or Step. Separately Reset, set both p=0.5 and n=100, predict mean 0.5 and standard deviation 0.05, then Play or Step. The bars are exact probabilities and playback highlights outcomes rather than generating samples. This model has no process offsets, build variability, warmup, or correlated request timings, so its narrowing must not be used as a hierarchical benchmark uncertainty estimate.