Prefill, decoding, and phase disaggregation

During prefill, a causal Transformer processes known prompt positions with considerable parallelism and constructs the cache. During decoding, each sequence's next token depends on its preceding token choice, although many sequences can be batched together. Prefill is often more compute intensive and decoding more sensitive to memory movement at modest batch sizes. These descriptions are tendencies, not universal hardware laws: context length, model shape, attention kernels, and batching can change which resource limits either phase.

Measure time to first token from a declared boundary, such as request receipt. It can include queueing, tokenization, prefill, transfers, and the first decode operation. Streaming latency concerns subsequent intervals; averages can hide pauses, so report an appropriate percentile and define whether it is per request or per token. End-to-end completion includes output length. A system can meet a first-token target while delivering the rest of an answer too slowly, or improve streaming after making users wait longer initially.

A disaggregated design assigns prompt processing and ongoing decoding to different worker pools. Each pool can use a suitable batch policy, parallelism arrangement, and potentially hardware type. The prefill worker must convey compatible cache state and request metadata to the decoder. DistServe and Splitwise investigate this phase separation with explicit attention to resource allocation and transfer. Their measured improvements belong to their evaluated workloads and configurations; they are not a guarantee for every cluster or service.

Estimate transfer using bytes divided by effective payload bandwidth, then add serialization, coordination, and network queueing. A link's advertised line rate is not the application's sustained transfer rate. Tensor parallelism may place cache shards on several devices, and data movement can contend with model execution or other collectives. Transfer overlap helps only to the extent that dependencies permit it. For a short prompt on an already busy network, a colocated worker can easily be the better design.

Goodput counts useful completed work that satisfies declared service objectives, rather than every generated token. Specify both first-token and streaming thresholds and report the fraction of requests satisfying them. Aggressive batching can increase throughput while making interactive requests miss deadlines. Compare designs at matched hardware cost or power budgets and include the cost of spare capacity, transfers, and rejected or cancelled work in the result.

Replay joint prompt/output length distributions, including bursts. Size the pools so one phase does not simply push a persistent queue onto the other. Consider overload, a lost decoder, a failed transfer, and cancellation while cache state is in flight. Bound queued work and assign deadlines across both pools. Begin with traces from a colocated baseline; phase separation should solve a measured interference or provisioning problem rather than add a network boundary without evidence.