Fabric congestion and collective tail latency

A shared link serves the aggregate traffic mapped onto it, including other jobs and protocol overhead. Sustained offered work near its usable capacity leaves little room for bursts or service variation. A long-term rate below capacity is a necessary stability-style condition in simple queue models, not a guarantee of a low percentile latency. Message size, congestion control, routing, backpressure, and head-of-line effects can change the observed tail. Count traffic at the selected boundary rather than adding logical tensor sizes and assuming they all traverse that link exactly once.

Rank mapping and collective algorithm determine which flows coincide. Moving ranks or changing a schedule can spread demand, but the change may also create more traffic across a constrained cut or reduce local reuse. Correlate fabric counters with the particular communicator, payload, and job timing where possible. NCCL's versioned diagnostic guidance separates system checks and transport evidence from tuning knobs. A parameter that improves one isolated benchmark may worsen contention when many jobs or layers communicate together.

Suppose each rank has a probability distribution for a readiness or completion time. If those times were independent and identically distributed, the probability that all p are below a threshold would be the single-rank probability raised to p. The maximum becomes more likely to exceed a high individual percentile as p grows. This is a teaching independence model, not a description of shared-fabric reality. Shared switches, common input storage, and synchronized workloads create correlations, so the actual joint distribution must be measured or modeled appropriately.

A late rank can hold peers at a collective even when its eventual transfer path is healthy. Measure when inputs become ready and calls are reached, not just when a waiting rank starts its own timer. CPU scheduling, data loading, previous kernels, or thermal and resource variation can create skew before network work. Conversely, aligned entries followed by long completion may point toward communication or shared-resource work. A collective includes dependencies on several ranks, so one local timing interval should not be interpreted as pure physical-link transit time.

Report distributions and a clearly defined aggregation interval, not only one best or mean time. Compare synchronized workload phases and record whether measurements include rank readiness waits. An average of local p99 values is not the p99 of their maximum. Repeated iterations can also be correlated, so a large sample count does not necessarily give that many independent observations. A useful investigation combines rank timelines, transport errors, congestion indicators, memory activity, and useful step throughput before choosing a mitigation.

The probability lab makes the independent-count assumption inspectable. Reset, set both p=0.05 and n=20, predict one slow rank on average and probability of none equal to 0.95^20, then Play or Step. Separately Reset, set both p=0.05 and n=30, predict 1.5 and no-slow probability 0.95^30, then Play or Step. Exact binomial bars count hypothetical independent slow ranks; they contain no shared switches, latency durations, or real fabric traffic. Explain which observed correlations would invalidate using this distribution as a training-tail forecast.