Communication-computation overlap

A gradient bucket becomes eligible only after all its contributing producers have finished. The communication operation can then read that bucket on a stream whose dependencies include those producers. An optimizer must not consume the reduced result before the appropriate completion dependency. Separate streams permit independent scheduling but do not create missing data or remove a true dependency. An incorrectly omitted event can appear fast by reading incomplete gradients, while an unnecessary global synchronization can serialize work that was genuinely independent.

NCCL 2.32.3 documents stream-ordered collective submission and group behavior. MPI 4.1 nonblocking collectives have a different request-completion interface and implementation-dependent progress characteristics. In either case, a successful host return is not the same measurement as communication completion. Confirm the linked library/runtime and progress environment before inferring that time spent on the CPU is available to drive another operation. A queued asynchronous call may wait behind earlier stream work or consume resources needed by the computation it was supposed to overlap.

A collective kernel can read and write device memory while the next backward kernel needs the same bandwidth. Both standalone durations can then change when run together. Concurrent launch does not imply independent-resource execution, and actual combined time can be worse than a naïve sum of isolated measurements if the schedule creates extra contention or stalls. Compare end-to-end step time and useful progress as well as the fraction of a profiler timeline containing visually overlapping activity. Overlap is a means of shortening the critical path, not a cosmetic occupancy target.

Bucket size trades startup overhead against readiness and scheduling. A large fused bucket amortizes communication launch but waits for its last gradient producer. Smaller buckets can begin earlier yet pay more per-call startup and retain additional state. The best partition depends on parameter order, backward compute times, datatype, network behavior, and resource contention. Preserve matching collective order across ranks while changing bucket layout; a faster local readiness sequence is not valid if different ranks launch incompatible collectives.

Record when each bucket becomes ready, when its communication actually starts, and when the next dependent consumer can proceed. The last collective often leaves an exposed tail after backward computation ends; optimizing earlier collectives may not shorten that tail. Include CPU launch gaps, event waits, and variation across ranks. Use synchronized measurement boundaries appropriate to the tool, while avoiding instrumentation that forces serialization and destroys the behavior being studied. Validate that numerical output and rank sequence remain correct after every scheduling change.

The queue lab isolates one contention-style capacity mismatch. Reset, set both λ=4 and μ=3, predict one normalized work unit of backlog growth per second from four, then Play or Step. Separately Reset, set both λ=2 and μ=3, predict drainage at t=4, then Play or Step. These separate fluid runs do not execute CUDA streams, kernels, or overlapping communication. They show why additional concurrent submissions can increase waiting rather than service capacity. The M/M/1 mean has separate assumptions; a real overlap claim still needs its actual dependency and shared-resource timeline.