RDMA completion, ordering, and buffer ownership

Registration associates a memory region with permissions and keys used by the interface to validate access. Local and remote operations use the appropriate keys and addresses within allowed bounds. The region's lifetime must cover outstanding operations, and registration costs or pinned-resource limits belong in the capacity budget. A valid key does not determine which business user should access data; application authentication and authorization still decide who receives capabilities and for how long.

An RDMA write, an RDMA read, and a send/receive exchange have different participation requirements. An ordinary write can place data into permitted remote memory without producing the same remote receive completion as a send; write-with-immediate has an explicit notification behavior and receive-resource requirements. Queue-pair transport modes also differ in reliability and supported operations. Name the exact operation instead of assuming every verbs call automatically wakes the remote application with a usable message.

Posting work successfully means it was accepted for execution, not that its source buffer can immediately be overwritten. The application must observe the relevant successful completion or another documented retirement guarantee before reusing storage still needed by the operation. Error completions and queue teardown need explicit handling. Inline sends have distinct source-buffer semantics when supported, so never generalize that exception to arbitrary registered operations or to memory that remote peers may still access.

A local successful completion can establish the operation's documented local reuse condition and, for a reliable transport, associated transport guarantees. It does not prove the remote application has polled, validated, or consumed the result, nor that bytes are durably stored. Ordering may hold for specified operations on a queue while not extending across independent queues, different memory agents, or CPU/GPU visibility without further rules. Build an explicit publication and acknowledgment protocol for the application's stronger requirement.

Track every live region and operation through normal completion, cancellation, and connection failure. A ring of buffers is safe only if reuse waits for the correct generation's retirement; wrapping an index does not prove the previous operation finished. Batching completions can reduce polling overhead but makes ownership bookkeeping more important. Benchmark registration, warm steady transfers, and teardown separately when the production path pays different subsets of those costs, and preserve end-to-end correctness checks.

The cache lab is a limited state-history contrast, not an RDMA model. Reset and set both C=4, s=4, predict four misses and twelve hits in its initial sixteen requests, then Step to add a hit. The empty-start LRU model tracks address identifiers, capped at 4096 requests. It has no operations, registration keys, completions, or asynchronous ownership. Use the trace to explain why a numerical slot index alone cannot establish a buffer's generation or retirement, then describe the missing lifecycle state explicitly.