KV transfer: count the actual cache representation

For a uniform dense KV layout, each retained token has a key and a value for every stored layer and KV head, each with the stated head dimension and dtype. Use KV heads, not the number of query heads. A heterogeneous model needs a sum over layers and representations rather than a single uniform product. Quantized caches can add scales, metadata, padding, or alignment; a one-byte element does not prove that complete wire payload is exactly half a two-byte layout. Record the concrete format and its version.

Map each actual source shard to destination-owned positions, layers, and heads. Equal head sharding can divide the logical payload only if those heads are genuinely partitioned without replication and the transfer layout matches. A tensor-parallel degree greater than the KV-head count can involve replication, and a different destination degree can require reshaping or redistribution. Prefix-cache reuse can remove known matching positions, but only after validating identity, access scope, and position semantics. Payload, allocated cache capacity, compressed bytes, and transmitted retries are different counters.

A transfer path can include reading and packing device memory, host staging, network serialization, destination unpacking, and completion notification. Its effective payload bandwidth can be limited by any shared hop rather than the link’s advertised rate. Define whether a time measurement ends at host receipt, device completion, per-layer readiness, or decoder usability. Enqueueing an asynchronous copy does not make its destination safe to read or its source safe to overwrite. The pinned connector’s layer waits and save completion hooks illustrate distinct lifetime boundaries.

If packing, transfer, and unpacking are fully sequential, add their durations and non-overlapped setup. If independent chunks can pipeline across stages, the ideal duration can approach the slowest stage plus unavoidable fill and drain costs. A maximum is then a lower-bound ingredient, not a promise that every operation overlaps perfectly. Layer-wise transfer can overlap some computation while still leaving the last dependency on the critical path. Measure that path and contention before subtracting an entire network duration from a latency budget.

Capture model and adapter digests, cache dtype, per-layer shapes, source and destination shard maps, token positions, and transferred checksums or other integrity evidence. Compare reconstructed cache state and resulting logits with a matched local computation under suitable numerical tolerances. Correct byte counts can still describe the wrong rotary positions, model revision, or adapter. Include a partial final block and an unequal shard case. If a transport compresses or converts state, evaluate both representation compatibility and output-quality effects rather than count only a smaller packet.

The roofline lab offers a bandwidth-ceiling analogy rather than a transfer simulator. Reset, explicitly set intensity eight and bandwidth 80, predict its ceiling, then Play or Step. Compare a fresh Reset with intensity eight and bandwidth 40 explicitly set; ceilings are 640 and 320 GFLOP/s. The instrument holds a synthetic compute roof and models no KV layout, serialization, packing, or network path. Use measured byte inventories and completion events for real transfer timing.