Device allocator fragmentation

A pool can retain backing memory after an application allocation becomes safely reusable. CUDA exposes reserved-memory and used-memory attributes with documented meanings; a framework’s similarly named counters need not have identical accounting boundaries. Compare counters over repeated equivalent workload phases and after appropriate completion points. A stable high-water reserve differs from unbounded growth of genuinely live allocations. Record which stream owns the final use and whether outstanding operations still need the allocation when collecting a diagnostic snapshot.

For a declared simplified pool, partition the reserved bytes into mutually exclusive live allocations, safely reusable free space, and other accounted overhead. A pending release remains unavailable until its ordering conditions permit reuse; do not subtract it just because a host call returned. Real allocators may distinguish allocation granularity, metadata, inaccessible regions, or external backing. An accounting equation is useful only if its terms cover the same pool at the same observation boundary, without counting one region twice.

cudaMallocAsync and cudaFreeAsync place allocation and release into stream order. An allocation must become available before its first use, and its release must follow every relevant use. When another stream accesses it, establish the necessary dependency through supported synchronization rather than assuming that host submission order across streams is sufficient. Memory-pool reuse policies can use dependencies or introduce documented dependencies to enable reuse, but those policies do not excuse an application that frees memory before a still-unordered consumer finishes.

Caching released backing can avoid expensive allocation activity at the cost of retaining a larger reserve. Trimming a pool can release eligible unused backing under its documented behavior; it cannot make an actively used tensor disappear safely. Aggressive trimming or synchronization can also hurt throughput, so memory reduction and latency form a workload-specific tradeoff. For variable-length inference batches, bound shape diversity or reuse suitable buffers when that preserves semantics, then measure both peak eligible storage and request latency.

Enough total free bytes need not imply one suitable block in a particular suballocator. For example, a fixed arena requiring a contiguous virtual subrange can fail when free regions are separated by live allocations. That example does not imply that CUDA generally requires physically contiguous device DRAM. Virtual-memory mapping, coalescing policies, size classes, alignment, and driver allocation can change the outcome. Describe the constrained allocator instead of diagnosing every out-of-memory error with the same fragmentation story.

The queue lab gives only an analogy for pending releases. Reset, set arrival rate to 0 and service rate to 4, predict the backlog at one second, then Step twice; its initial four units drain to zero. Compare a fresh Reset with both arrival at 2 and service at 4 before two Steps; two units remain. Changing a control preserves elapsed time, so the explicit Reset matters. The instrument contains no pointers, free lists, byte sizes, or CUDA streams.