Peak activation memory depends on overlapping lifetimes and allocator scope

For a defined memory scope, assign each unique allocation a size and an interval from allocation until it can actually be reused. A tensor view usually shares an existing storage rather than creating another full allocation, so counting logical tensor sizes alone can double count. Conversely, a small view can retain a much larger underlying storage. Saved forward values, retained graph references, and asynchronous stream use can extend lifetimes beyond the Python variable's apparent last use. Use the allocator and execution semantics rather than source indentation to establish release.

Peak live memory is the maximum simultaneous sum of the counted storage sizes. Define endpoint conventions so a free followed by a same-time allocation is handled consistently. Separate persistent parameter, gradient, and optimizer-state storage from activations and transient workspaces, but combine their lifetimes when computing the actual total. The maximum of category totals taken at different times is not the same as the maximum of their sum. A peak snapshot should therefore include time or operation identity and the relevant live allocations, not only a ranked list of allocation sizes.

A caching allocator can keep blocks reserved after individual tensors are freed so later allocations can reuse them efficiently. Reserved capacity is therefore different from bytes currently allocated to live tensors. Fragmentation can prevent an allocation even when some free space exists inside the reserved pool. Resetting a peak statistic changes the observation window, not the allocator's physical state. A measurement that begins after warmup should state which persistent allocations already exist and whether the reported peak is a current amount, a maximum since reset, or an allocation-volume counter.

Framework allocation history may omit device memory acquired directly by another library. Communication buffers are a practical example; a framework snapshot can be internally accurate while explaining only part of total device usage. Compare measurements with documented scopes instead of assuming their difference is automatically a leak. Context overhead, workspaces, graph pools, and driver-visible external allocations can contribute. APIs and allocator backends evolve, so verify the deployed version's definitions and support before treating a particular counter or snapshot field as universally complete.

Activation checkpointing trades retained values for later recomputation, which can reduce a particular peak but add arithmetic and change timing. Offloading trades device residency for transfers and synchronization; it can introduce another peak if staging buffers overlap. Compare the complete live-memory trace and completed training time rather than subtracting every removed activation from one static number. Check saved-tensor semantics and numerical equivalence as well. A configuration that fits memory by changing the batch or effective update count needs a separate quality and throughput comparison.

The cache lab illustrates bounded residency in a much smaller, different model. Reset, set both cache capacity C=4 and request stride s=4, predict four misses and twelve hits in the initial sixteen requests, then inspect or Play. Reset and set both C=2 and s=4 for sixteen initial misses. The fully associative LRU model requests address (i×s) mod16 from an empty cache, adds two requests per playback second, and caps simulation at 4096 requests. It tracks toy entries, not activation bytes, tensor aliases, lifetimes, a GPU allocator, or a memory benchmark.