Streams: order work without assuming overlap

A host-to-device copy, kernel, and device-to-host copy submitted to one stream can express that the kernel uses the copied input and that the output copy follows the kernel. The host still needs the appropriate completion observation before reading the destination or reusing buffers whose transfers may be active. A launch returning successfully does not mean its output exists on the host. Record which stream owns each operation and which completion point protects each buffer’s lifetime.

An Async suffix does not mean every argument combination behaves identically with respect to the host. CUDA12.6 documents copy synchronization behavior depending on transfer direction and memory type; pageable host memory can involve staging or blocking. Proper pinned memory can support intended asynchronous transfer behavior, but hardware support, resources, dependencies, and workload still determine overlap. Do not build correctness around a particular amount of host blocking that an implementation happened to exhibit during one test.

If producer P writes a buffer in stream S1 and consumer C reads it in S2, host submission order alone is not a sufficient cross-stream dependency in the general case. Record an event after P and make S2 wait on the appropriate event before C, or use another documented ordering mechanism. If no data dependency exists, separate streams permit overlap but do not promise it. Resource pressure, copy-engine limits, kernel characteristics, and other contexts can serialize operations without violating correctness.

Default-stream mode matters. The legacy default stream has implicit synchronization with blocking streams in the same context, while explicitly nonblocking streams avoid that legacy synchronization. A per-thread default stream is local to the host thread and context and behaves like an ordinary stream with respect to other ordinary streams; it is not the same as a legacy global ordering shortcut. It is also not a nonblocking stream relative to a legacy default stream. Configure and document the mode, especially when separately compiled libraries participate.

A correct multi-stream program should not deadlock merely because independent work is scheduled serially. Avoid cross-stream waits whose graph contains a cycle, and keep buffer ownership explicit across reuse and deallocation. Benchmark device work with suitable events or completion-aware timers; a CPU timer around enqueue calls measures submission plus any host blocking, not necessarily kernel execution. Include required output completion when reporting application latency, and distinguish it from the steady-state throughput of a pipelined transfer schedule.

The queue lab illustrates submission outrunning service at a much simpler level. Reset, set λ=4 and μ=3, predict backlog 4+t, then Play or Step. Reset and set λ=2, μ=3 to predict a drain at t=4. This one fluid queue has no CUDA streams, DMA engines, events, pinned buffers, or default-stream mode. It explains why fast host submission need not mean completed device work; actual stream order and transfer lifetime must be verified with the documented API.