Collective microbenchmark interpretation

The nccl-tests performance documentation defines algorithm bandwidth as logical operation size divided by measured time. For the AllReduce examples here, size is the complete input tensor bytes per rank. Preserve the benchmark's exact definition for other collectives, whose count arguments and full output sizes differ. Distinguish decimal GB/s from binary GiB/s, and do not multiply an input element count by a datatype width twice. Record whether in-place or out-of-place execution, datatype, reduction operation, and validation mode match the intended workload.

Time a completed operation at a defined boundary rather than only the host enqueue. Warmup can avoid conflating communicator setup, allocation, or initial transport behavior with steady operation, but cold-start cost may itself matter and should then be reported separately. Iteration averages can hide tails and rank asymmetry, so preserve the tool's rank aggregation mode and useful variation. GPU event clocks have device and recording semantics; do not subtract unrelated device timestamps as though they shared one calibrated wall clock.

For AllReduce, nccl-tests multiplies algorithm bandwidth by 2(p−1)/p to form its bus-bandwidth column. That factor reflects a conventional per-rank transfer normalization and enables a more useful comparison across rank counts under the model discussed in its documentation. It does not prove that the implementation selected a ring. Other operations use their own factors and size conventions. A single formula should not be copied to ReduceScatter or AllGather without checking whether size refers to the full array or one local contribution.

Normalized bus bandwidth is not a direct physical-port counter, a per-link utilization percentage, or total fabric-hop throughput. Multiple links, channels, rails, asymmetric paths, and hardware collective assistance can change how the logical normalized work relates to actual traffic. An apparently surprising ratio should trigger checks of units, topology, operation definition, algorithm, and timing before a product claim. Compare a measured physical bottleneck only when its counted direction, aggregate scope, and transfer assumptions match the normalization.

Record accelerator placement, NIC or rail mapping, node count, library/runtime/driver versions, selected configuration, message-size range, warmup and measured iterations, and correctness results. Sweep payloads corresponding to real training buckets instead of publishing only the best large-array point. Compare a one-node baseline with cross-node and contended runs where the application needs those conditions. A benchmark with incorrect output, a hidden fallback path, or a different datatype cannot support the desired training-performance claim even if its bandwidth column looks impressive.

The queue lab provides an intentionally separate measurement-boundary exercise. Reset, set both λ=2 and μ=3, predict drainage of its initial four work units at t=4, then Play or Step. Separately Reset, set both λ=4 and μ=3, predict growth at one unit/s from four, then Play or Step. The fluid model has no bytes, accelerator links, or collective timing and cannot generate nccl-tests bandwidth. Use the explicit units and initial condition as a reminder to define what a real numerator, denominator, and boundary count before interpreting their ratio.