Cache handoff: separate copy completion from ownership commitment

A possible sequence is producing, prepared, copying, receiver-ready, ownership-committed, and source-released. Prepared identifies an immutable cache manifest and handoff attempt. Copying retains every source allocation still read by transfer. Receiver-ready means the required destination blocks and metadata have been validated and are usable under the stated completion contract. Ownership-committed changes which attempt may advance the request or emit output. Source-released happens only when its local readers are finished and the handoff protocol permits reclamation. These are different events even if a fast path combines their messages.

Specify whether readiness means device-visible data, a volatile process buffer, replicated memory, or durable recoverable state. An acknowledgment must name the model and cache identity, request generation, position frontier, destination, and handoff epoch it confirms. A lost acknowledgment makes state uncertain; it does not prove copying failed or succeeded. Query or retry the same idempotent protocol step against authoritative state rather than immediately create an unrelated active owner. Include every required worker shard before declaring the logical request ready.

During handoff, both source and destination may retain the same logical cache. Duplicate bytes are necessary in many safe copy protocols and do not imply duplicate output authority. Conversely, retaining only one copy does not prove that two workers cannot both emit from stale metadata. A proposed ownership ledger can commit a new monotonic epoch through a correctly coordinated operation, while consumers and the output gateway reject stale epochs. Merely attaching a larger number to a packet is insufficient if the resource or output path never checks it.

The source must not free or reuse memory while a transfer still reads it. vLLM v0.10.2’s save-completion interface explicitly addresses avoiding overwrite before saving finishes, while its other hooks expose asynchronous lifecycle decisions. Those local hooks do not establish a cross-process durable transaction or client-visible exactly-once delivery. If crash recovery is claimed, define which ownership and readiness records survive each failure. A volatile receiver-ready acknowledgment followed by a receiver crash cannot be treated as durable state without additional storage or recomputation assumptions.

Inject failure before receiver-ready, after receiver-ready but before ownership commitment, after commitment but before acknowledgment delivery, and during source release. Check that the authoritative owner is unambiguous and that stale attempts cannot emit or reclaim another attempt’s blocks. Retrying a status query should not create another generation. Cleanup must wait for actual transfer completion and respect retained references. A trace should reconstruct which identity and epoch each acknowledgment covered, including delayed messages from a previous destination incarnation.

The reliability lab illustrates only majority availability. Reset, explicitly set replicas five and unavailable one, predict four available and threshold three, then Play or Step. Compare a fresh Reset with replicas five and unavailable three explicitly set, yielding two available. The model implements no ownership ledger, cache copies, acknowledgment durability, fencing, or reclamation. A majority indicator cannot establish this handoff invariant; the actual state-machine and failure trace must do that work.