research29 min studyLesson 745 in Engineering

Partial retries: a byte count is not a valid cache prefix

Track generation-scoped chunk identity, contiguous coverage, integrity, and idempotent acknowledgments before resuming a transfer.

On this page

What you’ll understand

  • Define a contiguous verified position frontier across all required shards.
  • Distinguish duplicate receipt from an idempotent state transition.
  • Handle stale generations and in-flight memory lifetimes during retry.

A receiver can hold many cache bytes and still lack the next required position in one layer or head shard. Retrying a partial transfer therefore needs semantic coverage, not just a received-byte counter. Define immutable chunk identity over a request, generation, model/cache manifest, shard, and position range, then specify how duplicate and delayed messages are handled. We describe an illustrative protocol and use vLLM v0.10.2’s connector lifecycle as a scoped completion reference, without claiming that every connector implements this retry or durability design.

Represent what is complete, not merely what arrived

A chunk describes exact logical positions, layers or head shards, representation, byte length, and appropriate integrity evidence. Receiving a packet does not mean its asynchronous device copy has completed or its contents match the expected manifest. Integrity checks also have explicit assumptions; a checksum is not a universal proof against malicious substitution. Record completion only at the defined validation boundary. A partial final block must not advertise uninitialized positions as valid just because its backing allocation has the same size as a full block.

For each required shard, track the largest contiguous verified prefix rather than the highest position seen or the total number of positions stored. Holes matter. The logical request frontier is bounded by the least-complete required shard under a compatible layout. Layer-wise pipelining may permit earlier work on ready layers, but that is a separately validated dependency schedule; it does not justify declaring the entire handoff complete. Keep sparse coverage metadata if the transfer can arrive out of order, and derive contiguous frontiers from it correctly.

fs=max⁡{k:[0,k) is verified in shard s},f=min⁡sfsf_s=\max\{k:[0,k)\text{ is verified in shard }s\},\qquad f=\min_s f_s
For a nonempty set of required shards and integer position counts, fs is each shard’s contiguous verified prefix and f a necessary common frontier. Coverage must refer to the same request generation and compatible manifest; this minimum alone does not validate dtype, model identity, ownership, or output readiness.↔ Scroll wide equations horizontally.

Retry the same identity without reviving stale attempts

An idempotent chunk operation should recognize repeated delivery of the same immutable content and return a compatible result without advancing positions twice or allocating an unbounded duplicate. A duplicate key with different content is a conflict to reject, not a successful retry. Retain deduplication state for the promised retry interval and failure scope; process-local memory does not provide crash-persistent idempotency by itself. If a status acknowledgment was lost, query validated coverage before choosing which pieces to resend, subject to the actual protocol’s guarantees.

Generation and handoff epochs prevent an old transfer from overwriting a new request attempt that reused the same physical block. Enforce the identity at the destination and coordinate registration, cancellation, and cleanup with in-flight device or network operations. Canceling a logical request does not synchronously stop every DMA reader or writer. Reusing its address too early can corrupt an unrelated request even if the stale message is later rejected by the scheduler. Version checks and memory-lifetime protection must meet at the actual write boundary.

K=(q,g,e,manifest,s,[a,b))K=(q,g,e,\text{manifest},s,[a,b))
This illustrative immutable retry key contains request q, generation g, handoff epoch e, manifest identity, shard s, and integer position interval with 0≤a<b. A key must also bind expected content and representation under the protocol; naming fields alone does not implement durable deduplication or transport safety.↔ Scroll wide equations horizontally.

Reconstruct correctness after ambiguous failures

Test a lost chunk, a lost acknowledgment, duplicate delivery, an out-of-order block, a conflicting duplicate, and a late completion after cancellation. Compare the receiver’s verified prefix and content with the sender’s manifest before allowing the appropriate decode operation. Retrying all bytes can be simpler but still needs the same identity, capacity, and lifetime rules. Report retransmitted wire volume separately from logical payload and validated cache coverage. A successful retry test under one failure does not establish arbitrary transport durability or seamless continuation after every process loss.

The modular lab offers a finite coverage analogy. Reset, explicitly set modulus eight and additive step three, predict the orbit period, then Play or Step. Compare a fresh Reset with modulus eight and step two explicitly set. The first visits all eight residues over its period, while the second visits only four. This deterministic enumeration contains no chunks, acknowledgments, holes, manifests, or device transfers. Use it to question incomplete enumeration, then validate actual contiguous coverage and content in the transfer fixture.

An out-of-order chunk does not close an earlier hole

A logical cache spans 64 positions divided into four complete all-shard chunks: [0,16), [16,32), [32,48), and [48,64). Each chunk is eight MiB. Only the first and third are verified initially, with identical manifests and generation.

  1. Sixteen MiB is verified in total, covering 32 positions, but the contiguous common prefix ends at position sixteen because [16,32) is missing. The highest verified endpoint 48 is not the usable prefix frontier.

  2. After successfully validating the missing second chunk, 24 MiB is verified and the prefix reaches 48. The already verified third chunk can contribute once the hole before it is closed; it need not be counted twice.

  3. Validating the fourth chunk reaches all 64 positions and 32 MiB of logical coverage. Actual wire bytes can exceed 32 MiB because of retries or failed partial sends; coverage and network traffic must remain separate counters.

The result

The prefix advances 16→48→64 despite initially holding a later chunk. A received-byte total alone cannot establish the required cache state.

Explore the model

Interactive lab

Interactive lab · modular

Coverage of an index set is only a transfer-planning analogy

Use independent Reset → explicitly set BOTH modulus and step → predict → Play/Step runs for (8,3), period eight, and (8,2), period four. The instrument has no transfer acknowledgments, contiguous-prefix tracking, data integrity, or generations.

Coverage of an index set is only a transfer-planning analogyAddition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Current readings: Residue r0: 0; Orbit period · m/gcd(m,a): 12; Distinct residues visited: 1 of 12. All parameters are adjustable with the labeled controls below.Repeated addition modulo 12: the current residue is 0, and the orbit period is 12.01234567891011+ 5mod 12An additive orbitn = 0rₙ = 0Period = 12Cycle starts at 0:0 → 5 → 10 → 3 →8 → 1 → 6 → 11 →4 → 9 → 2 → 7 → 0Exact residues · two additions per second
Additive orbitCurrent residue / next edge
Residue r0
0
Orbit period · m/gcd(m,a)
12
Distinct residues visited
1 of 12
t = 0 s

Addition modulo m starts at zero: rₙ = na mod m. Playback advances two additions per second. The orbit has period m/gcd(m,a), including period one for a = 0. This is not a cryptographic security test or a primality algorithm. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forVerify actual coverage and identity rather than infer completeness from bytes or a repeating index sequence.

Check your understanding

A cache receiver has verified positions [0,16) and [32,48) for every required shard. What is the contiguous prefix frontier?

Try it yourself

A 128-position cache uses four 32-position all-shard chunks and one KiB per logical position. Chunks zero, one, and three are verified. Calculate total verified KiB and contiguous prefix positions. What changes after validating chunk two?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Separate union size from the prefix beginning at zero; the last chunk cannot bridge an earlier gap.

Compare with the solution

Three chunks contain 96 KiB of verified logical data, but the prefix reaches only position 64 because [64,96) is missing. Validating chunk two gives 128 KiB and a prefix of 128 positions. This assumes all required shards, content, generation, and manifest checks agree. It does not prove ownership or visible-output commitment, and retransmitted bytes must be counted separately. A delayed packet from another generation must not fill the hole or write into a reused allocation.

Take it with you

  • Track contiguous verified coverage across every required shard.
  • Bind retries to immutable generation-scoped identity and content.
  • Protect in-flight memory and separate logical coverage from wire traffic.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.