Peer-to-peer transfer and ordering

The CUDA Runtime 13.4 peer-access interface distinguishes whether one device can access another's memory and whether that access has been enabled. The documented enablement is directional; the reverse direction requires its own supported setup. Platform, topology, allocation, and runtime constraints still apply. A capability result does not mean every possible device pair has the same bandwidth, that a remote allocation is immortal, or that every cross-device atomic operation has the desired semantics. Use the supported access and synchronization contracts of the deployed architecture.

NCCL 2.32.3 two-sided point-to-point communication uses matching sends and receives with compatible peer ranks, counts, and datatypes. Its grouping interface can express sets of operations that need concurrent progress. Those rank-based communication calls are distinct from merely enabling CUDA peer memory access. A transport may use different supported physical paths based on topology. Validate the actual selected route rather than inferring that a high-level peer operation necessarily uses one particular local link or never touches host resources.

First establish producer completion before the communication stream or device reads the source. Then establish transfer completion before a consumer reads the destination. Finally, allow source or destination reuse only after all operations that still reference the corresponding storage have finished. Supported stream ordering, events, waits, or communication requests provide those dependencies. A host-side flag set immediately after an asynchronous submission does not automatically order device work across unrelated streams. An incorrect dependency can produce intermittent old data instead of a clean error.

Avoid inferring correctness from one successful transfer of a zero-filled array. Use identifiable values and vary producer timing, streams, buffer reuse, and iteration sequence while retaining the documented contracts. A stream wait can order later work without a global device synchronization, but the event must refer to the intended operation and revision. Cancellation and timeout need special care: software deciding to stop waiting does not prove that a device or peer stopped accessing the buffer. Teardown must quiesce the applicable operations before storage is freed or recycled.

Double buffering can let one transfer proceed while another buffer is produced or consumed, but it doubles the retained slots for that particular buffer category. Count source and destination pools separately and include other outstanding iterations. A pipeline that reuses slot i modulo a small ring needs evidence that the earlier use of that slot is complete. Increasing slot count can hide latency until another resource binds, but it also consumes memory and can make a stalled peer retain more state. Bound in-flight work instead of treating unlimited asynchronous submissions as free overlap.

The Markov instrument is useful as a contrast to real completion evidence. Reset, set both p=0.5 and q=0, predict modeled completion-state probability 0.75 after two Steps, then Step twice. Separately Reset, set both p=0.9 and q=0, predict 0.99 after two Steps, then Step twice. These are exact probabilities for an assumed two-state chain, not observations from a CUDA transfer. Even a high modeled completion probability does not authorize consuming or recycling a real buffer without the required event or request completion.