Attribute memory growth at the tensor, allocator, pool, and device boundaries

PyTorch 2.9 documents memory_allocated as tensor-occupied memory and memory_reserved as capacity managed by its caching allocator. A backing KV-pool tensor can remain allocated even when application blocks become free. Conversely, a logical reference leak can exhaust that fixed pool without changing either tensor counter. Device-level tools can include runtime contexts, allocations outside the framework, and other processes, so their totals need not equal the framework's reservation. Compare counters on the same device and time window, with the allocator backend and process scope recorded.

The same documentation explains that empty_cache releases unused allocator cache rather than live tensor storage. It cannot free objects still held through output lists, closures, callbacks, views, or other references. A snapshot's apparent free blocks may also be constrained by pending stream events or allocation size. Do not add allocated and reserved bytes as independent categories, since the former normally sits inside the latter under the standard native caching allocator. Capture their difference as allocator-held capacity not currently reported as tensor allocation, then inspect its state instead of naming all of it a leak.

Run a controlled repeated workload with fixed model, shapes, request limits, prefix policy, and bounded output retention. Warm up expected kernels and graph pools, then drain requests and synchronize at declared diagnostic checkpoints before comparing residual live usage. Synchronization can change overlap and performance, so use it deliberately for attribution rather than leaving it in every production path. Compare tensor counters with active block references, idle cached blocks, outstanding transfer buffers, and request registries. An increasing count of completed requests still owning blocks is different evidence from a stable cache reaching its intended capacity.

Use the versioned PyTorch memory-history and snapshot facilities to trace framework-managed allocations and their call stacks. Their documented visibility excludes allocations made directly through other CUDA APIs; NCCL is one common example. A device-level residual suggests further investigation, not automatic attribution to that particular library. Record driver and runtime versions, backend configuration, concurrent processes, and exact failing allocation size. Keep diagnostic metadata useful without dumping raw user prompts or sensitive cached contents into an uncontrolled report. A trace should support an ownership explanation and a reproducible fix.

A live-tensor leak calls for finding and releasing unintended owners; allocator fragmentation calls for evidence about sizes and inactive splits; a fixed KV-pool leak calls for repairing block-reference transitions; an external allocation needs visibility outside the framework; and a transient prefill peak may require a different concurrency limit. Backend-specific allocator options should follow measured evidence, not be applied as a universal OOM cure. Reproduce cancellation, rollback, prefix reuse, and offload completion paths, because resource leaks often occur on exceptional exits rather than the successful short-request path.

The queue lab contrasts a draining workload with a stable nonzero backlog, not allocator memory. Reset, explicitly set BOTH λ=2 and μ=3, predict initial q=4 drains by t=4, then Play or Step eight times. Reset and re-set BOTH λ=3 and μ=3; predict q=4 at t=4. Edits retain t and recompute the original constant-rate trajectory. This fluid model has no tensors, caching allocator, device counters, or leaks. A nonzero steady quantity can reflect ongoing work; the interpretation requires the correct workload boundary and ownership evidence.