Decode capacity: token service and cache placement constrain scaling

For a fixed block size, a sequence generally occupies a rounded number of blocks, leaving unused positions in its final block. That internal waste differs from free blocks stranded across different devices or parallel groups. Shared prefixes and copy-on-write branches add reference accounting; do not charge one physical shared block once per logical sequence unless that is the intended conservative budget. Distinguish usable cache capacity from total device memory after weights, workspaces, staging, and runtime margins. A count of free bytes without these distinctions is not an admission proof.

Decode growth also requires a future policy. An estimated output length can guide packing but cannot guarantee that a request stays within the estimate. A hard generation cap, reserved extension blocks, explicit eviction/recomputation, or a defined failure response is needed for bounded resource claims. Variable arrivals and endings alter fragmentation over time. Test the allocation and release trace across EOS, cancellation, transfer, and branch termination rather than compute a single static occupancy percentage and assume it persists under continuous batching.

With request arrival rate and mean generated length, the mean output-token demand is their product under a consistently defined workload. A decode pool needs sufficient useful token service as well as enough resident state and acceptable iteration latency. Larger batches may improve aggregate throughput while worsening per-request gaps; small batches may underuse compute. Longer contexts can change per-token cost, so one fixed tokens-per-second number is not transferable across every length distribution. Preserve the prompt and output mix when comparing capacity measurements.

Adding a replica adds model residency and a potential place for new work; it does not automatically rebalance current sequences. Moving an active request needs a compatible KV and generation-state handoff, valid output ownership, transfer capacity, and a latency budget. A tensor-parallel decode instance also requires feasible state across its required shards, not merely aggregate free memory across the cluster. A request whose state fits in total can still fail because one required shard or chosen instance lacks its local obligation.

Report allocated blocks, logically used positions, unused final-block positions, shared references, uncommitted reservations, and per-instance free capacity. Include workload phases with long survivors and short arrivals, which can leave capacity distributed poorly even when the cluster average looks healthy. Check cleanup after aborted transfers and canceled streams. Scale-out tests should show when new instances become model-ready, how new requests are routed, and whether active-state movement is attempted. Count transfer and duplicated state during migration rather than treating it as a free rebalance.

The vector lab illustrates two normalized resource dimensions. Reset, explicitly set angle 45 degrees and magnitude two, predict both projections, then Play or Step. Compare a fresh Reset with angle zero and magnitude two explicitly set. The projections are approximately (1.414,1.414) versus (2,0) at the same magnitude. This geometric model contains no KV blocks, sequences, batch scheduling, or replicas. It only shows why one aggregate resource number cannot describe every memory-and-service demand profile.