Choose prompt chunks at real interruption boundaries

Later prompt chunks depend on earlier prompt context. An implementation must retain or reconstruct the appropriate state and apply correct causal attention to the prefix. Treating chunks as unrelated documents would change the model computation. The scheduler's opportunity occurs where the engine returns control or supports an explicit boundary; choosing a small logical chunk does not imply a running GPU kernel can be interrupted at any instruction. Include in-flight work and synchronization when measuring how quickly a new priority can take effect.

Sarathi-Serve studies chunked prompt work and mixed batches as a way to balance throughput and latency. Pinned vLLM 0.10.2 documentation describes its own V1 decode-prioritized chunking policy and iteration token budget. Those designs motivate experiments but do not justify copying a reported gain or a default chunk size to another workload. Also distinguish a chunk's raw prompt-token count from its actual time when growing context changes attention work and when mixed decode work changes batch efficiency.

In a deliberately simple serial model, a prompt costs α milliseconds per token and h milliseconds of fixed overhead per chunk. Its total prompt work is αP+h·ceil(P/c). This model exposes repeated overhead but deliberately ignores changing attention cost, batching, transfer, and overlapping execution. It can compare a stated scheduling idea without pretending to predict an accelerator. Measurements should then replace its coefficients or reject the model where it fails.

If decode can be reconsidered between chunks, one chunk contributes at most its modeled nonpreemptive interval before that opportunity under the simple assumptions. That is not the full inter-token latency: waiting for other work, decode duration, host handling, and transport remain. Very small chunks can also postpone a new prompt's completion and time to first token through repeated overhead or deprioritization. A policy that protects existing streams indefinitely can starve new prompts unless admission and aging address that outcome.

Measure prompt completion, first-token delay, stream gaps, completed-request goodput, memory occupancy, and fairness under the same offered workload. Include short prompts, long prompts, and bursts, and preserve the exact chunking policy and serving version. A mean decode rate can hide a single long gap, while a favorable stream percentile can hide prompt starvation. Record which boundary timestamps the profiler and client use so a shorter kernel is not confused with a shorter user-visible response.

The queue lab adds only a fluid overload comparison. Reset, set both arrival λ=4 and service μ=3, predict backlog eight after four seconds, then Play or take eight Steps. Reset and set both λ=2,μ=3 for a separate run that drains its initial four units at four seconds. Control edits retain elapsed time, so these are fresh comparisons rather than live rate switches. This fluid model contains no chunks, kernels, priorities, latency distribution, or measured p99; it shows why chunk tuning alone cannot eliminate sustained excess offered work.