Completion queues and buffer lifetimes

In io_uring, submission entries describe work and completion entries report results with an application-supplied identity. Do not infer completion order from submission order unless the particular API and dependency mechanism guarantee it. Associate each operation with a stable request record. A slot number plus a generation counter is a useful identity scheme when records are recycled: a stale completion for generation seven must not be applied to the new request occupying that slot at generation eight.

Consuming a completion entry returns space in the completion ring; it does not tell your parser to stop reading the payload. Copy fields you need before advancing the ring, then advance exactly as the library contract requires. A conventional one-shot read’s terminal completion ends that operation’s kernel access, but the application may retain the returned bytes. Other operation types can produce multiple entries or extra notifications, so use their documented terminal conditions rather than assuming the first entry ends every lifetime.

For an ordinary read pool, write the transitions as free to submitted, submitted to application-held on terminal completion, and application-held to free after the final consumer releases it. A failed read still needs a terminal transition and cleanup; error status does not make ownership disappear. Cancellation likewise initiates another protocol step. It does not permit freeing the target operation’s memory while that target can still access it. Keep the request record until all required terminal evidence has been handled.

If several consumers share one payload, track their references explicitly or transfer ownership along a single chain. The right condition for recycling is that neither the I/O mechanism nor an application reader may still access the region. Registered buffers reduce repeated registration work, but registration is not permission to overwrite an in-flight payload. Keep registration lifetime and per-operation access lifetime separate; eventual unregistration or replacement must also follow its own documented resource-release rules.

Pool exhaustion can be healthy backpressure. When every buffer is owned, stop issuing reads until a consumer releases one, or apply an explicit rejection policy. Allocating unbounded replacements hides the bottleneck until memory is exhausted. Record time spent waiting for I/O separately from time spent queued for parsing. A pool larger than submission depth may be necessary, but larger is not automatically faster: it can increase retained memory and queueing delay without improving the slow stage’s throughput.

Use the queue lab to explore this bottleneck only as an analogy. Let arrivals represent completed payloads reaching a parser and service represent parser releases. Press Reset, set arrival rate to 4 and service rate to 3, predict the count after one second, then press Step twice. For an independent draining case, press Reset again, explicitly set arrival rate to 2 and service rate to 3, predict the new count, then press Step twice. Each run starts at four units. The model has no finite pool, generation tags, DMA, or ownership states. Its M/M/1 residence readout applies only under that separate stochastic model’s stable assumptions; it is not a measured buffer lifetime.