Stage routing: admit the complete path, not the emptiest prefill

A route estimate should include prompt length, expected generated length, eligible prefix-cache reuse, model and adapter placement, prefill queue work, transfer bytes, decode queue state, and required service objectives. Queue length in requests is insufficient when one queued prompt is much larger than another. Predictions are uncertain, so retain errors by workload class and define conservative bounds or rejection behavior where required. A routing score ranks candidates; it does not itself reserve their resources or guarantee that an estimate will still be accurate when the request arrives.

Stage compatibility is also a correctness filter. The receiving decoder must interpret cache state from the exact intended model, adapter, position scheme, dtype, and layout, with supported reshaping where needed. Identical tensor dimensions do not establish semantic compatibility. Prefix reuse must be keyed by the actual token and configuration identity and authorized within the intended tenant scope. A cache hit on an incompatible or unauthorized prefix is not a latency optimization. Preserve identity and provenance across routing, transfer metadata, and the output stream.

A prefill can consume expensive compute and create a large cache that no decoder can admit. One design reserves a bounded decode obligation before or during prefill, then expires or releases it through an explicit cancellation and ownership protocol. Count current allocations, unmaterialized reservations, and the new request in disjoint categories; double-counting a transferred block as both reserved and allocated can cause false rejection, while counting neither causes oversubscription. A reservation’s lifetime must include the handoff uncertainty it is intended to cover.

Coordinating multiple resources introduces hold-and-wait risks. A request holding source memory while waiting indefinitely for destination space can block other requests and retain stale transfer state. Bound reservation lifetimes, define retry and reroute conditions, and ensure stale owners cannot later spend an expired allocation. Estimated output lengths are not hard capacity guarantees; use a defined generation cap or an explicit extension/eviction policy for future growth. Resource availability snapshots and actual accepted reservations are different evidence, especially with concurrent routers.

Log candidate estimates, selected route, reservation identity, stage timestamps, actual lengths, transfer outcomes, and release events. Check rejected requests and expired reservations so a low completed-request latency cannot hide stranded work. Test concurrent admissions, decoder loss after prefill starts, cancellation during transfer, and a cache hit invalidated before use. A reroute needs a new compatible ownership transition rather than merely changing an address in an old transfer message. Preserve the client request and generation identity while distinguishing each internal attempt.

The queue lab isolates local average pressure. Reset, explicitly set arrival four and service three, predict growth from four items, then Play or Step. Compare a fresh Reset with arrival two and service three explicitly set before predicting drainage. Edits retain elapsed time and recompute from the initial backlog. This single fluid queue contains no second stage, routing score, reservations, model compatibility, or cancellation. Use it to motivate a local pressure measurement while validating end-to-end routing with the actual stage trace.