A barrier follows the slowest required participant, not the average rank

For a simple barrier, take each participant's arrival time relative to a common step origin and define release by the latest required arrival. A participant that finishes early waits for that maximum. Aggregate waiting can be expressed in rank-time units, which is different from elapsed job time. This model excludes barrier implementation overhead unless it is added explicitly. A collective has a richer execution pattern than an instantaneous barrier, but matching readiness and completion boundaries still prevents confusing an early rank's wait with its own productive computation.

Use aligned rank and step identities, and include the actual required participant set. Pipeline stages, uneven expert routing, and different local work can make ranks legitimately heterogeneous. A rank appearing slow at a later synchronization may have inherited an earlier delay, so trace backward to its first unusual readiness or service interval. Compare multiple steps to see whether the same rank is persistently slow or the maximum moves around. Both patterns can hurt throughput, but they suggest different hypotheses about placement, work imbalance, and shared disturbances.

If rank durations are independent, the probability that all finish before a threshold is the product of their individual completion probabilities. Identically distributed independent ranks reduce this to a common CDF raised to device count. A small per-rank slow probability can then become a noticeable probability that at least one rank is slow. This is a model of repeated comparable steps with stated distributions; it is not a license to extrapolate a measured single-device tail unchanged into a different distributed workload.

Common storage stalls, synchronized checkpoints, thermal or power constraints, and shared network congestion can correlate timings. Under perfectly correlated rank durations, adding ranks does not multiply independent chances of a slow event. Other dependence structures can change both frequency and severity. Temporal correlation across steps also affects uncertainty in estimated tail rates. Keep joint per-step measurements when possible instead of reconstructing a maximum distribution only from separately recorded rank histograms. Marginal distributions alone generally do not determine the distribution of their maximum.

Compare per-rank input readiness, kernel execution, collective readiness, and completion to locate the earliest divergence. A slow collective on every rank may reflect one late entrant rather than uniformly poor links. A repeated long compute kernel on one rank may reflect different work or an execution-resource difference. Separate these hypotheses with fixed-shape replays, placement changes, and repeated measurements. Report tail quantiles with sample count and dependence assumptions; a maximum from one short trace is not a reliable estimate of a rare-event distribution.

The probability lab can illustrate an explicitly independent slow-event count. Reset, set both p=0.1 and n=4, predict probability of no slow participants 0.9⁴=0.6561 and at least one 0.3439, then Play or Step. Reset and set both p=0.1 and n=8 to predict at least one 0.56953279. Its bars are exact binomial masses and playback only highlights outcomes; it neither samples rank timings nor models correlated training participants. n is limited to thirty. Use this chosen independent model to understand the product rule, then check real dependence separately.