Query routing and reranker budgets

Include query processing, routing, each retrieval branch, fusion, reranking, and any later response work. If two branches execute in parallel and fusion waits for both, their contribution is the slower branch’s completion, not the sum of their isolated durations. Shared resources can still make both slower under load. A branch that is optional at a deadline needs an explicit fallback and quality measurement. Canceling a client wait does not automatically stop every already submitted backend operation or release its compute immediately.

Routing features can include query length, identifier patterns, language, and validated uncertainty signals, but the policy must be evaluated on its actual mistakes. A cheap route that misses relevant evidence on a minority query class can look attractive in an aggregate latency average. Keep candidate recall, final relevance quality, and latency outcomes by route and query class. When a route changes the candidate distribution, a reranker threshold or fusion weight tuned on another distribution may need fresh validation rather than being carried over unchanged.

A simple local reranker model uses a fixed setup cost plus a per-candidate term. It can help choose a preliminary cap, but batching, token lengths, padding, device occupancy, and queueing can make the real cost nonlinear. Fit and validate the model over representative operating conditions, and reserve slack for effects it does not capture. Candidate depth cannot exceed the eligible pool, and a cap of zero needs a defined skip-rerank result rather than an accidental empty response.

Tail-latency evaluation requires an arrival model and a load condition. Component percentiles cannot generally be added to obtain the end-to-end percentile, especially when queueing and shared contention correlate them. Measure complete request latency and its quality outcome, including timeouts and fallback results. A routing policy that simply drops difficult requests can improve latency among survivors while worsening user success. Report denominator and failure accounting so the apparent service improvement is not a consequence of excluding its expensive errors.

Compare a fixed baseline with the proposed routing policy on held-out traffic and labeled query classes. Log selected branches, candidate counts, token lengths, actual durations, and which relevant documents were absent before reranking. Keep a fallback that has a defined relevance and latency contract, and test whether its activation frequency shifts over time. Routing errors, retrieval misses, and reranker mistakes call for different corrections. Increasing reranker depth cannot recover a relevant document excluded by every selected branch.

The queue lab isolates one sustained-capacity effect. Reset, set arrival rate to 3 and service rate to 2, predict the backlog at one second, then Step twice: five units. Compare a fresh Reset with both arrival rate at 2 and service rate at 3 before two Steps: three units. This initial-four fluid queue has no query classes, branch routing, token-dependent batches, deadlines, or measured tail latency. It cannot certify a reranking service’s operating point.