Hierarchical collectives and topology

A logical neighbor exchange may traverse several physical links, and several logical edges may share one NIC or switch uplink. A rank-to-device placement determines which paths are used. Include GPU proximity to NICs, host NUMA placement, peer-access support, and routing policy when interpreting a slowdown. A topology diagram with many connections does not prove that all links can simultaneously sustain their individual headline rates. Direction, contention, protocol, and supported transport mechanisms determine which capacities can be combined.

For a chosen algorithm, count the data that must cross a specific cut or bottleneck in one direction. Dividing those bytes by the usable aggregate directional capacity gives a transfer lower bound. The counted D must match the selected schedule or a justified semantic lower bound; do not simply use the total tensor bytes resident in every GPU. In-network reduction, sharding, and local aggregation can change crossing traffic. Startup and local processing add time beyond this bandwidth bound, and multiple sequential bottlenecks may require a richer path model.

A simple hierarchical design reduces data within each node, performs an inter-node reduction among selected representatives, and distributes the final result locally. Other designs shard work across several representatives or rails. A single representative can simplify cross-node traffic but become limited by its own local memory or NIC path. More representatives do not create more external bandwidth if they share the same constrained uplink. Count both communication volume and the resources serving each phase before attributing a result solely to the word hierarchical.

The phases may execute sequentially, pipeline by chunks, or overlap in supported implementations. A sequential sum is a useful explicit model, while a pipelined design needs its critical-path and shared-resource analysis. Local reduction consumes accelerator memory bandwidth and arithmetic resources too. When an inter-node network is very fast, that local stage can become the dominant cost. Implementation and architecture versions matter: NCCL may choose different supported algorithms and transports depending on discovered topology and configuration, rather than following a hand-drawn two-level schedule.

A communication-heavy application can still be limited by device memory movement while reducing or staging data. Measure that category separately from NIC bytes and protocol time. An optimization that lowers cross-node traffic can increase local reads, copies, or temporary storage. Record the selected rank mapping, network interfaces, library/runtime versions, and payload when comparing runs. Use hypothetical rates for teaching unless measured evidence supports a particular product claim; a vendor link label alone is not the sustained per-job bandwidth of a shared fabric.

The roofline lab isolates a local numeric-kernel ceiling, with bytes counted between the last-level cache and main memory. Reset, set both I=0.5 FLOP/byte and B=100 GB/s, predict 50 GFLOP/s, then Play or Step. Separately Reset, set both I=0.5 and B=200, predict 100 GFLOP/s, then Play or Step. The compute roof remains a synthetic 1,000 GFLOP/s. This does not model NIC bandwidth, a topology, or a collective; it illustrates a possible local stage that must be measured separately from cross-node traffic.