Collective deadlocks and mismatched calls

Record a communicator or generation identity and a collective sequence number together with operation, count, datatype, root where applicable, and logical tensor identity. Compare the same sequence position across ranks. Equal total byte size is insufficient: 1,024 four-byte elements and 512 eight-byte elements have equal payload bytes but different count and datatype contracts. Equal signatures can still conceal a tensor-semantic error, such as reducing gradients from different training steps. A compact debug signature helps locate a mismatch; it is not a replacement for the actual API contract.

Conditional control flow can skip a call on one rank or change bucket order. An exception before enqueue, an uneven data-loader exit, or a failed peer can leave other ranks waiting at a valid-looking operation. In MPI 4.1, collective ordering must match per communicator, and interactions across overlapping communicators or point-to-point calls must not form blocking cycles. Nonblocking operations offer scheduling tools but do not permit mismatching blocking and nonblocking collective calls or arbitrarily reusing active communication buffers.

Draw an edge from A to B when A cannot progress until B performs a required action. A directed cycle can be a deadlock when every edge is a current blocking dependency, no alternative progress path exists, and no external recovery breaks it. A cycle drawn merely from communication neighbors is not sufficient: a valid ring algorithm also has cyclic connectivity while its operations progress correctly. Distinguish a topology graph from a wait-for graph and document the exact action that prevents each participant from advancing.

Single-thread multi-device launch can require grouping under the selected NCCL API so initialization or operations can make the necessary concurrent progress. Thread-safety and multiple-communicator ordering have documented version-specific rules too. Inspect those rules instead of assuming all asynchronous-looking calls are host-nonblocking under every configuration. A rank that stopped at an earlier compute kernel can cause others to look blocked in communication; observe stream dependencies and host progress alongside collective logs before assigning the cause to the fabric.

Collect operation identity, first error, last completed step, rank exit status, and relevant runtime or fabric diagnostics. Do not discard the earliest failure merely because many later ranks report a timeout. A longer timeout can be appropriate for a legitimate long operation, but it cannot make a missing participant execute a call. Recovery must use the documented library and job semantics, including coordinated abort or supported communicator reconstruction where applicable, and restore consistent training state. A unilateral retry of an unresolved collective can create a different mismatch.

The modular lab contrasts a connectivity cycle with a deadlock proof. Reset, set both m=3 and a=1, predict a three-residue orbit, then Play or Step. Separately Reset, set both m=4 and a=2, predict a two-residue orbit, then Play or Step. The instrument performs enabled arithmetic steps and therefore keeps progressing despite its cycles. It is not a wait-for simulator or MPI/NCCL debugger. Explain which additional blocking dependencies would be needed before a similar drawn cycle could demonstrate an actual deadlock.