RoCEv2, loss recovery, and congestion response

The selected reliable transport detects and recovers from missing or otherwise invalid progress according to its implementation and configured retry behavior. Congestion control changes the offered sending rate to avoid persistent overload. PFC applies hop-by-hop priority pausing to protect buffers under its assumptions. These mechanisms can interact, but they solve different problems. A network that does not drop packets can still suffer long queues or stalls, while a network that permits loss needs recovery that performs acceptably for the workload.

RoCEv2's IP and UDP encapsulation enables routed fabrics and can supply fields for forwarding decisions. It does not make the RDMA operation behave like an arbitrary application UDP datagram, nor does it inherit TCP's congestion controller. Verify actual path MTU, addressing, priority classification, and endpoint support. A mismatch can cause failures that look like congestion but are configuration or reachability problems. Begin with correctness and compatible endpoint state before interpreting bandwidth numbers.

DCQCN uses congestion indications to adjust sender behavior in RoCE deployments. In the original design, switch marking, receiver-generated congestion notifications, and a sender-side rate-control process form a loop. Notification timing, rate limits, and parameter choices affect its response. Naming DCQCN does not establish that switches mark at suitable thresholds, notifications return on a functioning path, or the sender's actual settings match the intended controller. Inspect the implemented version rather than copying constants from another cluster.

PFC can suppress some congestion drops while that loop reacts, yet can also introduce blocking and cyclic dependencies. It is not a fundamental semantic prerequisite for every possible RDMA transport design: supported lossy RoCE configurations and alternative recovery designs exist. Removing it from a particular system still requires a validated loss and recovery budget, not a generic claim that Ethernet handles everything. Evaluate short-message tails, sustained collectives, incast, and receiver pressure under representative faults and traffic mixes.

Test clean point-to-point operation first, then introduce multiple senders, mixed directions, and synchronized collectives. Record loss, retransmission, ECN, congestion-notification, and pause counters with their device-specific definitions. Compare end-to-end completion and error status with those signals. A receiver that cannot drain its local resources can trigger upstream symptoms even when switch links are not over-subscribed. Preserve the actual receiver and memory path used by the AI workload instead of benchmarking a faster substitute.

The queue lab demonstrates only the need to match offered work with service. Reset and set both λ=4, μ=2; predict growing backlog and Step. For an independent lower-load case Reset and re-set λ=2, μ=4 before predicting and stepping. The fluid trajectory begins at four jobs and control edits retain time unless reset. It has no RoCE packets, retransmissions, DCQCN state, ECN notifications, or PFC. Use it to state the rate imbalance, then identify which missing feedback and recovery measurements the real deployment needs.