Topology placement: shared cuts constrain aggregate handoff

Draw source and destination parallel groups and identify which layer, head, and position slices cross which links. Different stage parallelism degrees can require redistribution rather than a one-to-one shard copy. KV-head replication, packing, staging, and protocol metadata can increase actual traffic beyond a logical payload inventory. Include model-parallel communication and other tenants sharing the path. A placement that minimizes one cache transfer can worsen an intra-stage collective, so the complete stage-and-transfer topology must be evaluated together.

Use the actual endpoint path: GPU-to-NIC reachability, NUMA placement, PCIe switches, host copies where applicable, and inter-node or inter-rack links. An advertised aggregate NIC rate may be unavailable to one device or one flow. A local copy benchmark can avoid the shared uplink that dominates production. Record payload size, concurrent flows, source/destination placement, and all relevant competing traffic when reporting effective bandwidth. Without that context, a transfer-rate number cannot be converted safely into a per-request latency budget.

For one transfer, each required stage with finite bandwidth supplies a lower bound on completion time from its traffic volume. The maximum of those bounds is necessary even if stages overlap perfectly. Serial dependencies, setup, contention, and fill/drain can make actual time larger. Do not add independent-path bandwidths unless the transfer is genuinely striped across them with compatible routing and endpoints. Conversely, do not count the same logical payload twice simply because two counters observe it along the same path; distinguish traffic at each constrained resource.

At cluster scale, crossing only a fraction of requests over a scarce cut can reduce average load, but it changes placement and possibly cache reuse or resource balance. A local prefill-to-decode pair still needs capacity and compatible model state. Confining all transfers to one locality can create a decoder hotspot while remote resources sit idle. Compare the saved cut traffic against resulting stage queueing and failure domains. A cut below its mean capacity can still violate token deadlines under synchronized bursts.

Benchmark the intended number of concurrent transfers while both stages execute representative model work. Measure queueing, effective payload bandwidth, link and endpoint utilization, first-token and streaming gaps, and retransmission or staging costs. Move one stage or shard group to test a predicted bottleneck, then verify that another resource has not become limiting. Repeat under relevant failure and reroute conditions. A topology-aware policy should log why it chose a placement and what capacity estimate it relied on, so regressions can be traced to concrete path changes.

The roofline lab illustrates only a bandwidth ceiling. Reset, explicitly set intensity eight and bandwidth 80, predict the result, then Play or Step. Compare a fresh Reset with intensity eight and bandwidth 40 explicitly set; the ceilings are 640 and 320 GFLOP/s under its fixed compute roof. The instrument has no network graph, topology cut, concurrent flow, shard map, or cache transfer. Use it for the effect of a bandwidth parameter, while the actual path inventory and concurrency benchmark establish placement feasibility.