Separate transport capability, collective scheduling, and kernel cost with controlled tests

Record collective type, participant count, rank placement, topology, message size, dtype, and synchronization protocol. A small-message latency result and a large-message bandwidth result probe different regimes. Initialization and warmup should be separated from steady operations unless startup is the question. Preserve correctness checks: a fast transfer that produces invalid data is not a performance result for the intended operation. Low-level link tests can establish a transport baseline, while a collective test includes algorithm and multi-participant effects absent from a point-to-point measurement.

Define what bytes mean. A payload divided by elapsed time gives one algorithm-bandwidth convention, while a collective's topology-aware bus-bandwidth estimate can include an explicit operation-specific factor. It is not generally the same as measuring every physical link's traffic counter. Mixing decimal GB with binary GiB creates another avoidable discrepancy. Keep benchmark version and its reporting formula with results; do not compare a normalized collective metric directly with one wire's nominal rate without establishing that their byte counts and scope are compatible.

Replay representative compute kernels with the same shapes, strides, arithmetic mode, and important workspace constraints. A large square matrix multiplication may achieve an impressive rate while saying little about the smaller, irregular kernels that dominate a particular model. Device clocks, thermal state, cache state, and interference can change the result. Record whether inputs are already resident and whether launches, transfers, or synchronization are inside timing. Match the experiment to the suspected mechanism rather than selecting a benchmark because it produces the highest advertised arithmetic rate.

Then compare the isolated components with a concurrent replay and the full training schedule. If compute and communication each slow when concurrent, shared execution or memory resources are a plausible hypothesis. If isolated communication is healthy but the training collective begins late, inspect readiness and launch dependencies. If only a low-level link test fails, investigate the fabric baseline before tuning an unrelated matrix kernel. A useful test should make competing hypotheses predict different observable changes, while preserving the mathematical workload and validating its outputs.

Build a short evidence chain: transport baseline, matched collective, matched compute, concurrent replay, and end-to-end training. Change placement or one configuration only after a hypothesis predicts which layer should respond. Repeat enough completed windows to distinguish a persistent effect from transient load. Save topology and configuration alongside timing distributions so the experiment can be reproduced. A passing lower layer narrows the search but does not validate every higher-layer interaction; similarly, a slower isolated operation can still improve the completed job if its scheduling exposes less of it.

The roofline lab helps separate device-memory bandwidth from compute ceilings, but its B is not network bandwidth. Reset, set both I=4 FLOP/byte and B=50 decimal GB/s, predict 200 GFLOP/s, then Play. Reset and set both I=4 and B=100 for 400. Reset again and set both I=20 and B=100 to predict the fixed 1000 compute cap. It models no network topology, collective, contention, or measured kernel. Use the unit-checked ceiling as a question about a hypothetical kernel, then gather the real hierarchy of evidence before naming a network or compute bottleneck.