Name the computation and traffic behind each scheduling unit

For a dense decoder's linear layers, a rough weight-multiply term often scales with parameter count times processed tokens. Attention adds work depending on prompt or existing context length, and the exact factor depends on which projections and arithmetic are counted. Name the included FLOPs rather than calling a single parameter-based approximation the complete model. Speculative decoding, sparse experts, sliding windows, quantization, and multimodal inputs can require different terms. The chosen scheduler cost should match the served architecture and execution path.

Decode can reuse loaded weights across several active sequences, while each sequence also reads its relevant KV history. Increasing batch size can amortize some weight traffic without making every added sequence free. Longer contexts can increase KV reads and retained state, changing both latency and feasible batch size. Prefill is often more compute-intensive than small-batch decode, as the original Sarathi and DistServe analyses discuss, but neither phase is universally tied to one bottleneck on every configuration.

If prefill and decode run on different devices, the relevant KV state must become available to the decode side under the design's transfer protocol. That can reduce interference between phases but adds bandwidth, placement, coordination, and failure considerations. A model that counts only GPU compute can miss a transfer that dominates first-token delay. Conversely, shared devices avoid that transfer while coupling their scheduling and memory budgets. DistServe studies one such design space rather than proving one topology optimal for all workloads.

Distinguish work estimates from latency objectives. Time to first token includes relevant waiting and transport boundaries, not simply prefill kernel time. Inter-token gaps can include other requests' prefill chunks, scheduling, and delivery stalls. Record batch composition, context lengths, cache-hit policy, and the measured boundary when calibrating a cost model. An average per-token coefficient fitted to one workload should be checked again when the workload shifts or the scheduler changes the batch distribution.

Choose controlled prompt and output-length classes, vary batch size, and compare resource counters with synchronized iteration timings. Keep output semantics and cache state comparable. A growing token count with flat iteration time can indicate amortization within a range, but not unlimited capacity. A change in latency can also reflect admission or queueing instead of a slower kernel. The scheduler needs the cost distribution and relevant tails under its operating conditions, not one isolated peak tokens-per-second result.

The roofline lab isolates a synthetic resource ceiling. Reset, set both intensity I=4 FLOP/byte and bandwidth B=80 GB/s, predict 320 GFLOP/s, then Play or Step. Reset and set both I=16,B=80 for the capped 1,000 GFLOP/s comparison. Playback traces the selected roof without running a kernel. The instrument has no prompt length, KV cache, batching, network transfer, or request latency; it illustrates why a named compute/bytes assumption is necessary before declaring a phase compute-bound or memory-bound.