Performance counterfactuals and mechanism tests

Define what the proposed change preserves and what it alters: bytes transferred, instruction count, synchronization, scheduling, numerical accuracy, or offered load. A counterfactual such as halving a serial component’s duration is different from deleting its work. Under a fixed serial decomposition, Amdahl-style accounting predicts a maximum gain from accelerating that component while everything else remains unchanged. That model can guide a test, but it becomes inappropriate when the intervention also changes overlap, queueing, cache behavior, or the amount of work.

Compare the predicted mechanism with observable mediators at compatible boundaries. A compression hypothesis may predict fewer network bytes and more CPU work; an allocator hypothesis may predict fewer contended acquisitions without changing application output. Measure whole-operation completion as well as the mediator, since expensive work can move outside an instrumented region. A predicted mediator changing in the expected direction is corroboration, not proof that no other path caused the outcome. Include controls for instrumentation overhead and changed execution scheduling.

Let each experimental unit have potential outcomes under an implemented control and treatment. With the same nonzero treatment probability strictly below one for every unit, randomized assignment can make observed groups comparable in expectation within the chosen population when consistency, suitable independence, and absence of relevant interference or carryover hold. Blocking can improve precision while preserving the assignment rule. If assignment probabilities differ across blocks, combine within-block contrasts with target-population weights rather than blindly using an unweighted group difference. A before-and-after deployment without comparable load may instead combine treatment with a workload shift. Clearly identify which changes were randomized and which assumptions connect measured differences to the desired causal effect.

Coz provides an original example of an intervention-based performance tool: it estimates virtual speedup effects using controlled delays of other concurrent execution, under its mechanism and measurement assumptions. That is more specific than reading ordinary profiler time shares. It does not imply every delay experiment reproduces every real optimization, or that a real code change preserves the same scheduling and resource effects. Validate a promising intervention with the actual implementation and compare its end-to-end result to the predicted response.

If a speedup is attributed to memory traffic, vary problem size across a cache boundary, keep outputs fixed, and check the expected traffic response. If synchronization is blamed, vary contention independently where feasible and predict when the benefit should disappear. Include a negative control that should not respond to the proposed mechanism, while considering whether it changes other resources. Counterexamples narrow the explanation. A mechanism supported at one size, topology, and load should not be generalized to all executions without new evidence.

The roofline instrument permits a bounded counterfactual inside its fixed model. Reset, set both I=2 and B=100, predict a 200 GFLOP/s ceiling, then Play or Step. Separately Reset, set both I=2 and B=200, predict 400 GFLOP/s, then Play or Step. Declare I at the main-memory transfer boundary and retain the synthetic compute ceiling of 1000 GFLOP/s. These are model ceilings, not measured throughput or a causal profiler. Doubling B in this model cannot establish that a physical application will receive twice the usable bandwidth or complete twice as fast.