Uncertainty-aware human review queues

Let total request intensity be lambda and model coverage be phi over the planning population. Average review demand is lambda times one minus phi when every rejected request is sent to review once. Account separately for retries, duplicate submissions, escalations, and cases requiring multiple reviewers. Use the same time unit for arrivals and completed reviews. A model threshold chosen only for accepted accuracy can change the arrival mix and review effort per case, so the review service rate may change with the policy too.

A long-run average below capacity does not guarantee short waits or immunity to bursts. Reviewer shifts, correlated incidents, and difficult case clusters can create queues even with spare average capacity. Measure service-time distributions and deadlines by workload rather than substituting one global mean. Admission control may need a bounded buffer, prioritization, or a safe fallback when the review budget is exhausted. Specify what happens at overload before claiming that every abstention eventually receives a timely expert answer.

A reviewer’s accuracy on randomly sampled easy cases may not transfer to the hard cases selected by a model. Measure review performance on the routed population, including available context, interfaces, and time pressure. Distinguish an expert’s independent judgment from a judgment influenced by seeing the model answer. If the model and reviewer share a failure mode, deferral can preserve the error rather than correct it. Learning-to-defer methods model this complementarity instead of assuming that human intervention is always superior.

For completed decisions, total error is a coverage-weighted combination of model errors on accepted cases and reviewer errors on rejected cases. Both rates need their own denominators and adequate labels. Add review fees, delay penalties, and unresolved or dropped cases to the cost analysis rather than hiding them outside the accuracy denominator. A release should also retain a policy for label acquisition on auto-accepted cases; otherwise only rejected cases receive scrutiny and the apparent model reliability can become unidentifiable from operational logs.

A case deserves review when the expected reduction in decision loss justifies review effort and delay under the capacity policy. Raw uncertainty alone cannot answer that question: a highly ambiguous case may remain equally hard for the reviewer, while a moderately uncertain case may be easy with a missing document the reviewer can obtain. Estimate these differences on representative data and validate queue behavior with arrival traces, staffing schedules, and failure scenarios. Record routing reasons and policy versions for later error analysis.

The queue lab isolates constant-rate flow. Reset, set arrival lambda to 3 and service mu to 2, predict backlog five after two Steps, then take those Steps. Compare a fresh Reset with both lambda at 2 and mu at 3 before two Steps: backlog three. Initial backlog is four and two Steps represent one second. Edits pause and preserve elapsed time, so reset before comparing. The instrument has no uncertainty model, review priorities, service-time variation, or human accuracy.