Collective algorithms meet physical placement

All-reduce combines values from all ranks and makes the result available to each participating rank. A ring implementation typically separates reduce-scatter and all-gather phases, while tree or hierarchical algorithms arrange different exchanges. Libraries can change algorithm, protocol, channel count, and chunking with message size and topology. A hand calculation for one ring is a stated model, not a claim that a particular library always follows that exact schedule for every size.

For a conventional evenly partitioned ring with n ranks and S bytes of input per rank, each phase sends n−1 chunks of S/n bytes per rank. Summing the two phases gives 2(n−1)S/n sent bytes per rank, with matching received traffic. Do not add send and receive indiscriminately when comparing with a one-direction link rate. Startup latency, pipelining, reduction compute, and protocol overhead are additional to this payload accounting.

Map ranks to GPUs, GPUs to NICs, and NICs to switches. A remote NIC path can cross a local interconnect or NUMA boundary before packets reach the network. Multiple ranks may share one port, while different ports can share an upstream cut. Hierarchical reduction may keep some exchanges within a node before transferring inter-node results, changing external traffic without changing the all-reduce's mathematical output. Verify library topology discovery and actual routes rather than assuming rank numbering matches physical proximity.

Placement tradeoffs depend on the workload. Tensor-parallel exchanges may be frequent and latency sensitive; data-parallel gradient transfers can be larger and overlap compute; expert routing can create all-to-all skew. A placement favorable to one operation may overload a shared cut for another. Evaluate a realistic iteration schedule with concurrent storage and communication when those overlap in production. A clean isolated bulk test is useful evidence about one path, not a guarantee about synchronized job completion.

Sweep message sizes, rank counts, and placements while retaining correctness checks and recording the actual library settings. Small transfers expose startup and scheduling overhead; large transfers reveal throughput constraints. Compare per-rank completion spread, link utilization, and the slowest iteration, not only an average bandwidth column. A degraded link may affect all ranks through synchronization even when most links remain idle. Repeat under representative background traffic and report both normal and degraded placement behavior.

The Roofline lab is a separate resource-bound exercise, not a collective simulator. Reset and set both I=4 FLOP/byte and B=80 GB/s, predicting 320 GFLOP/s under its fixed 1,000 GFLOP/s compute ceiling; Play traces that roof only. Its bytes are the selected memory-boundary bytes of a hypothetical kernel, not ring payload or NIC traffic. Explain why substituting normalized collective bus bandwidth for memory bandwidth would change the accounting boundary and invalidate that calculation unless a new model justified it.