Zero-copy I/O and ownership

Consider an immutable payload built in an application buffer. With a copying send path, the kernel can often retain its own copy after accepting bytes. With copy avoidance, the application must preserve the relevant region until the API supplies the reuse signal. Keep the allocation alive and prevent writes through aliases, including slices handed to another task. A memory-safe language can still expose a logically unsafe design if an unsafe I/O boundary or external buffer pool ignores these access obligations.

Linux MSG_ZEROCOPY provides a notification when the shared storage can be reused; asking for copy avoidance does not guarantee that copying is avoided. Its notification is not proof that bytes reached the peer, and deferred copying can release storage before transmission finishes. Treat notification processing as part of the send path, not optional housekeeping. Registered or pinned pages are likewise not magically immutable: registration optimizes resource setup, while your ownership protocol must prevent accidental overwrites.

For io_uring zero-copy sends, inspect the initial completion flags. If that result carries IORING_CQE_F_MORE, the operation promises a later notification; IORING_CQE_F_NOTIF identifies that notification. Keep the required memory unchanged until the appropriate notification arrives. Do not unconditionally wait for a second entry when the initial result does not promise one. Maintain an operation record capable of handling both events, errors, and their documented ordering without returning the same allocation to its pool twice.

The initial result describes accepted bytes or an error, so partial sends still require explicit handling of the unsent region. Buffer reuse, local transmission, transport acknowledgement, and application acknowledgement answer different questions. If your product requires durable peer storage, use a peer protocol that confirms that condition. Cancellation cannot replace this evidence: a request to cancel a send is neither a promise that the device stopped nor permission to overwrite pages that remain referenced.

A useful benchmark measures more than payload throughput. Include CPU time, copied-byte fallback rate, notification lag, peak held bytes, and application latency across payload sizes. Keep the workload’s network path representative; a local loopback test can exercise different behavior from an actual network device. Compare total work at equal correctness guarantees. Dropping notifications to improve a benchmark merely removes the mechanism needed to reclaim memory, producing a misleading result and eventually exhausting the pool.

Try the queue lab with arrivals interpreted as payload buffers becoming retained and service as reuse notifications processed. Press Reset, set both arrival and service rates to 3, predict whether the initial four-unit backlog drains, then press Play. For a growth comparison, press Reset again, explicitly set arrival rate to 4 and service rate to 3, predict the new slope, then press Play. A separate Reset with both rates set to 2 and 3 permits a draining comparison; predict before pressing Play. Each experiment starts afresh at four units. This comparison motivates a bounded pool and admission control, but the lab has no page pinning, packet sizes, partial sends, or notification batching. Its stable M/M/1 readout is a separate idealized queue estimate, not a measured network or memory-release latency.