Buy memory capacity with an explicit access policy

A remote-memory system may fetch pages, objects, or smaller transactions. A large fetch can amortize fixed latency and support sequential reuse, but it can waste bandwidth for isolated small accesses. Track useful application bytes separately from fetched bytes, including metadata and writeback. An embedding workload that reads one small vector per remote page can have substantial amplification even if the advertised network bandwidth is high. A cache hit ratio alone does not describe that amplification.

Specify whether a cache probe is paid on both hits and misses. In the simple blocking model below, every access pays a local probe and a miss adds a remote fetch penalty. That is different from a weighted average where the quoted miss latency already includes the probe. State the convention to avoid counting or omitting the local path twice. Real systems with prefetching, asynchronous object access, or multiple outstanding misses need a richer concurrency model than one serialized mean.

Moving data to local memory incurs copy, metadata, eviction, and possibly writeback costs. An estimated latency saving per future reference must repay those costs before the object becomes cold or disappears. This is a prediction about future reuse, not a property of the memory technology. Include the opportunity cost of evicting another useful object: a migration that helps one tensor can hurt another more. Preserve the capacity constraint rather than calculating every object’s benefit as though local memory were unlimited.

Access phases can change faster than a tiering controller reacts. Newly hot data may suffer repeated remote misses before promotion, while recently cold data occupies local capacity. Rate-limit migration so a burst of misplaced pages cannot monopolize the same bandwidth needed for demand reads. Measure adaptation time, queueing, and tail latency during workload transitions. A steady-state hit rate from a warmed benchmark can conceal the exact cold-start and phase-change behavior a production service must tolerate.

Infiniswap studies remote memory through an RDMA-based swapping design, while AIFM integrates far-memory management with application objects. Their mechanisms illustrate different points in the design space rather than interchangeable performance numbers. A CPU load to CXL memory has another path and coherence contract. Reusing a remote-paging result as a CXL latency claim, or assuming an object cache inherits transparent shared-memory semantics, would change the system being analyzed.

Evaluate complete job or service behavior at a fixed working set and quality target. Capacity expansion may prevent out-of-memory failure or slower storage spill even when every remote access is slower than a local hit. Include remote-memory server resources, network contention, replication or recovery requirements, and memory reclamation in cost. Report failure behavior when the remote tier is unavailable. A large capacity pool is valuable only if the application can use it within its correctness, latency, and recovery constraints.