Host offload exchanges device capacity for a dependency-bound transfer path

A complete offload protocol allocates host storage, copies a valid state, waits for the relevant copy completion before recycling device storage, and records how to restore compatible state later. A restore needs a destination device allocation and completion ordering before its consumer runs. Pinned host buffers, staging pools, copy-engine queues, and temporary duplicate copies can raise peak memory on both sides. Host RAM is finite, and pinning removes the operating system's freedom to page that region normally; excessive pinned allocation can harm the surrounding system even when GPU occupancy improves.

FlexGen is a primary example of managing inference state across GPU, CPU, and disk for throughput-oriented generation; its workload and scheduling assumptions should not be promoted as guarantees for interactive token latency. The original PagedAttention experiments also compare swapping and recomputation under specific model and trace conditions. Measure the actual path: PCIe topology, NUMA placement, simultaneous directions, transfer batching, and competing traffic can change effective rates. A peak link specification is not necessarily the sustained bandwidth of many small KV block copies.

The CUDA 12.8.1 programming guide describes page-locked host memory and device capability conditions for overlapping transfers with kernel execution. Streams and events express ordering; hardware support and available resources determine what can run concurrently. Copying a request's missing KV values cannot overlap with the attention operation that already needs those same values, although it may overlap with independent work for another ready request. Using a nonblocking API alone neither supplies the required independence nor proves that a pageable source avoids hidden staging or host-side blocking.

For a single dependent restore followed by compute, add the stages along that critical path. For an ideal steady-state pipeline processing independent items with adequate buffers and no shared bottleneck, the slower stage can dominate the interval after pipeline fill. That optimistic pipeline bound is different from one request's end-to-end latency and may fail under variable sizes or memory contention. Measure transfers and computation on a synchronized timeline, and distinguish copy submission, completion, queue wait, and first useful consumer execution.

Track host occupancy, device restoration slots, transfer bytes per generated token, queue delay, and repeated offload of the same requests. A scheduler can oscillate between copying out and immediately copying back, consuming bandwidth while little decoding progresses. Bound outstanding transfers and ensure cancellation safely releases host and device ownership after in-flight operations finish. A lower GPU-resident byte count is not automatically a better serving result if tail latency, host pressure, or transfer amplification becomes unacceptable. Compare against recomputation and admission alternatives under the same workload.

The queue lab isolates a constant-rate transfer-job backlog. Reset, explicitly set BOTH arrival λ=4 and service μ=3, predict q(2)=6 from initial backlog four, then Play or Step four times. Reset and re-set BOTH λ=2 and μ=3; predict q(2)=2 and drain at t=4. Edits retain t and recompute q=max(0,4+(λ−μ)t), so these are separate fresh runs. The fluid toy has no bytes, DMA engines, pinning, or burst sizes. Its rates are hypothetical jobs per second, not measured offload bandwidth or a simulation of individual copies.