Schedule speculative work under shared latency and capacity budgets

Iteration-level or continuous batching can remove completed requests and admit or advance others between scheduling boundaries. Orca develops iteration-level scheduling for generative models, while later serving systems combine batching with other memory and execution mechanisms. Speculation adds variable-width work within a request: a long proposed block may consume many target query positions even if the first candidate fails. Define the backend's work unit explicitly. Some accounting includes a pending last token, prefix work, or a bonus prediction differently, so do not infer target input rows from committed token count alone.

Use a measured or supported estimate of verification cost and temporary storage before admitting work. A batch can contain requests with different lookahead lengths and acceptance histories, and padding or graph-shape constraints can waste some of the intended gain. Draft work may run on the same device, another device, or a shared stream, introducing communication and synchronization. Reserving room for every temporary branch can reduce the number of requests that fit. Separate a reduction in sequential target calls from a reduction in total device work or scheduler occupancy.

Long prompts can compete with decode and verification for compute and KV capacity. Chunking prefill or limiting speculative work may reduce an individual iteration's duration and improve opportunities for short requests, but the effect depends on the backend and workload. Measure time to first token, inter-token gaps, completion latency, throughput, and memory at the intended arrival process. A speculative round can emit several tokens at once followed by a longer pause; dividing its time by token count does not describe every user-visible gap. Preserve per-request traces rather than rely only on aggregate tokens per second.

Dynamic policies can reduce lookahead under high load, disable expensive drafts for poorly matched domains, or adapt to recent accepted-prefix statistics. Their correctness still requires the verifier to use actual proposal distributions and valid histories; changing scheduling should not introduce an unproved acceptance shortcut. Their performance must be compared as policies, including estimation and switching overhead. Pinned Transformers 4.56.2's documented assisted interface does not support batched inputs, while a separate serving backend has its own feature matrix. Do not combine those API capabilities into an unsupported universal implementation claim.

Replay or generate a declared arrival and prompt/output-length mix, and compare speculative and baseline configurations under matching target laws and completion criteria. Include warmup, admission, cancellation, and EOS behavior as relevant. Record completed and dropped requests so an apparent latency improvement is not explained by excluding difficult work. vLLM 0.10.2 documents version-specific speculative compatibility and numerical limits; its examples and warnings are not a promise that all hardware or parallelism modes support the same path. Profile the supported configuration rather than extrapolate an isolated benchmark.

The queue lab shows only a fixed-rate backlog effect. For each separate run, Reset, set both arrival λ=4 and service μ to 3 or 5, predict, then Play or Step. The first four-unit fluid backlog grows by one unit per second; the fresh second backlog drains in four seconds. These normalized units can illustrate offered work versus completion capacity, but the model has no batches, token trees, priority, admission, or random arrivals. It cannot certify tail latency or scheduler fairness. Actual serving decisions need the work and request-level evidence described above.