Judge recommendation quality, completed tail latency, and cost on the same cohorts

Use a relevance or user-outcome definition with an explicit horizon and measurement unit. Click-through rate is clicks per defined exposure opportunity, not a universal measure of utility. Satisfaction, retention, task completion, and harmful repetition can move differently. Offline ranking metrics also depend on the candidate universe and whether negatives were sampled. Preserve those choices and feedback knowledge windows so a changed score is not actually a changed denominator, easier candidate set, or use of later information unavailable to the compared decision.

Break results down by meaningful request and user cohorts, including sparse history, new items, region, device, route, and cache state where appropriate. A pooled rate is a weighted combination of cohort rates. Traffic composition can change that rate even when within-cohort behavior is unchanged or worse. Report fixed-population standardized comparisons when useful, but do not pretend reweighting alone creates causal identification. For causal claims, use an appropriate randomized or otherwise justified design with consistent eligible populations and outcome collection.

Measure request latency from a declared start to a usable response or defined failure, including readiness, feature fetches, retrieval, ranking, constraints, and serialization as appropriate. Parallel source retrieval follows its required join, not the sum of all branch times. Tail percentiles must come from the end-to-end distribution; summing stage p95 values or averaging route p95 values generally gives the wrong number. State how timeouts, cancellations, degraded fallbacks, and retries enter both latency and quality reports. A successful-only tail can improve simply because hard requests disappeared.

Track resource cost per stated work unit using a defined boundary: device time, host time, network bytes, storage, or money under a particular pricing scope. Allocated capacity and marginal utilization are different accounts. Include cold-cache or index-refresh regimes when they matter operationally. A larger candidate or embedding configuration should be evaluated on the same useful request mix and outcome requirement. Faster service that changes eligibility, omits work, or fails to return a meaningful slate is a different product behavior, not automatically an efficiency gain.

First verify a compatible model-index-state version, correct exposure and feedback joins, and candidate coverage. Then compare relevance or outcome measures, the required tail and failure rates, and resource cost on aligned cohorts. Preserve uncertainty, experiment assignment, and enough request lineage to investigate regressions. A policy that changes exploration or eligible actions can require new support for offline evaluation. A controlled online comparison should still monitor long-horizon outcomes and cohort harms rather than declaring victory from an early click proxy alone.

The queue lab gives a limited capacity-warning analogy. Reset, set both λ=2 and μ=3, predict stable M/M/1 mean residence 1 second, then Play or Step. Reset and set both λ=2.5 and μ=3 for 2 seconds. Its visible fluid backlog starts at four and follows max(0,4+(λ−μ)t); its separate mean assumes Poisson arrivals and exponential single-server service. There are no recommendation cohorts, tail percentiles, failures, money, or user outcomes. Use measured end-to-end request distributions and the defined cost account rather than interpreting this toy mean as a production budget.