Stage asymmetry: separate work before separating hardware

Prefill processes many prompt positions and constructs the key/value state needed for subsequent generation. Matrix operations can exploit substantial prompt-level parallelism, and long prompts may make arithmetic throughput important. Decode repeatedly extends active sequences, reading model and cache state while producing relatively few new positions per request. At some batch sizes its bandwidth and cache-capacity demands dominate. These are tendencies, not fixed labels: short prompts, large decode batches, attention variants, quantization, parallelism, and kernel implementation can move either stage to another bottleneck.

Colocation allows shared weight residency and avoids a remote cache handoff, but prompt work can delay the next decode iteration and distort streaming latency. Chunked prefill and other scheduling policies can reduce that interference without physically separating stages, with their own overhead and tradeoffs. Disaggregation introduces separate copies or placements of model state, a transfer path, destination capacity requirements, and another failure boundary. Compare complete useful service under a controlled resource budget instead of inferring an architectural winner from one isolated stage benchmark.

Time to first token includes the chosen request-admission and output-delivery boundaries, not merely a prefill kernel duration. Some implementations sample and expose a first token from prefill; others gate output on successful decode handoff. Those designs place transfer cost differently relative to the first visible token and first streaming gap. State the convention and record both stage events and client timestamps. A faster prefill kernel can coexist with slower first response when destination admission or cache movement dominates.

Average time per output token and tail inter-token latency are also different metrics. A request can have an acceptable average while one long handoff or scheduling pause produces a visible stall. Measure output length, stop conditions, and the actual sequence of token arrival times. Streaming buffers and detokenization can coalesce arrivals, so server iteration timestamps alone do not describe the client experience. Count successful requests meeting the stated first-token and streaming objectives rather than promote total generated tokens per second to a complete service metric.

Build a workload matrix across prompt length, generated length, cache reuse, concurrency, and model configuration. Profile each stage independently and then together, including transfer and admission. The same request can demand high prefill work but little decode residence, or the opposite. Scaling the prefill pool according to request rate alone can overproduce cache state for a memory-bound decode pool. Preserve workload classes and deadlines when evaluating a new split, and report rejected work and cache recomputation rather than hide them from latency statistics.

The roofline lab illustrates a synthetic bottleneck change. Reset, explicitly set intensity eight and bandwidth 80, predict the ceiling, then Play or Step. Compare a fresh Reset with intensity sixteen and bandwidth 80 explicitly set. The fixed 1,000-GFLOP/s compute roof gives ceilings 640 and 1,000. These are hypothetical intensity choices, not measured prefill or decode phases. The instrument has no batching, cache handoff, queues, or model weights and cannot establish that either real stage always belongs to one resource regime.