Pinned host memory and transfers

Page-locked host memory can be used for DMA without relying on an ordinary pageable buffer remaining resident in a convenient physical arrangement. CUDA exposes allocation or registration APIs for this purpose. Pinning is a limited system resource: excessive page locking can reduce memory available for ordinary paging and degrade the host. Prefer a bounded pool with an explicit owner and measured reuse rather than repeatedly pinning unbounded batches as demand rises.

Pinned and mapped are related but different properties. A host allocation can support an explicit copy to device memory, while mapped access allows a kernel to access host-backed storage under the platform’s rules. That access still traverses the relevant interconnect on a discrete GPU and does not magically become local device DRAM. Check capabilities and API flags rather than inferring placement from a pointer value. For training tensors, repeated device reuse often makes one explicit transfer preferable to repeatedly reading remote host memory, but measure the actual workload.

An asynchronous-named copy API does not promise identical host behavior for every allocation and direction. Pageable memory can require staging or synchronization described by the CUDA runtime’s API synchronization rules. For reliable transfer/computation overlap, establish suitable pinned buffers, device support, independent streams or operations, and enough execution resources. Legacy default-stream interactions can impose ordering that serializes a seemingly parallel schedule. Record the actual dependency graph rather than assuming two API calls automatically overlap.

The source of a host-to-device copy must remain valid and unmodified until its required completion condition, and a device-to-host destination must not be consumed early by the CPU. An API returning after enqueueing is not that completion. Use stream or event completion with the correct device and operation association, then transfer buffer ownership back to the producer. The same reasoning applies to recycling a pool slot: a later batch must not overwrite bytes still being read by an earlier transfer.

Separate CPU decoding, pinning or registration, transfer queueing, DMA duration, and device consumption. Increasing pinned buffers can hide short fluctuations but cannot raise a saturated link’s sustained throughput. Host NUMA placement can also affect the path to the GPU. Include a realistic batch size distribution and avoid timing only enqueue latency. Keep correctness checks for buffer reuse and transfer completion while profiling; a pipeline that skips a needed wait can look fast by consuming data before it is ready.

The queue lab isolates pending-work pressure. Reset, set arrival rate to 4 and service rate to 3, predict backlog at one second, then Step twice. Compare a fresh Reset with both arrival rate at 2 and service rate at 4, predicting before two Steps. Each starts with four abstract equal batches. This fluid model has no pinned pages, DMA engines, streams, or ownership states. Use it to explain why more queued transfers cannot fix a sustained rate imbalance, and draw a separate completion timeline for safe buffer reuse.