Collective semantics and participation

A rank identifies a participant within a communicator, rather than a universal physical GPU number. Rank-to-device mapping affects roots, gathered layout, and topology. In the NCCL 2.32.3 documentation used here, each participating rank is associated with its selected CUDA device and operations must follow the supported communicator rules. Do not assign the same device repeatedly as different ordinary ranks of one communicator and assume that this creates extra independent hardware. Initialization and recovery are distributed protocols whose membership must agree before data movement is meaningful.

Every required rank must make a matching call in the collective sequence. NCCL's documented equal-count collectives require the matching count and datatype contract, while MPI 4.1 has operation-specific type-signature and count rules, including variable-count routines. Equal buffer byte totals alone do not prove compatibility. Distinguish element counts, datatype widths, reduction operation, root where applicable, and communicator identity. A rank using a tensor from a different training step can violate the logical contract even when its count and datatype happen to match.

A sum AllReduce returns sums; a framework may divide later to obtain an average, but that normalization is another operation. If every worker contributes a mean gradient over an equal local batch, averaging rank means gives the mean over all samples. Unequal local batch sizes require appropriate weighting. Padding, masked tokens, and gradient accumulation can change the effective sample count further. Define whether the objective averages examples, valid tokens, or another unit before assuming that division by communicator size implements the intended global loss.

Floating-point addition is not associative. Different legal algorithm orders can produce small numerical differences, so bitwise reproducibility is a separate requirement from the logical sum contract and numerical-tolerance checks. Datatype choice also changes payload and accumulation behavior. Compare final training correctness and convergence under the selected numerical rules instead of treating identical bandwidth with a lower-precision datatype as a free improvement. A user-defined MPI reduction has its own associativity and commutativity contract; an algorithm requiring commutativity cannot be applied indiscriminately.

NCCL collective calls use CUDA streams; successful posting is not equivalent to device completion. Preserve the documented stream dependencies and buffer lifetime before an optimizer reads reduced gradients or a producer overwrites send storage. MPI nonblocking completion has its own request and buffer rules, so do not transfer one API's completion assumptions to another. A collective also need not behave as a general barrier for unrelated actions. Name the exact local completion observation and what, if anything, it establishes about peers.

The reliability instrument deliberately exposes a misleading analogy. Reset, set both n=5 and k=1, predict four available replicas and a majority threshold of three, then Play or Step. Separately Reset, set both n=5 and k=0, predict five available, then Play or Step. A majority is still available in the first display, but an unchanged five-rank collective cannot silently omit its missing rank. This toy majority counter does not run NCCL, establish matching calls, or perform membership recovery. Explain why coordinated creation of a different supported communicator changes the contract rather than repairing an old call by wishful counting.