Candidate coverage sets a ceiling that a downstream ranker cannot repair

For one request, define an eligible catalog snapshot and a reference set of relevant items within it. Relevance can come from a carefully designed judgment set or another explicitly justified target; logged clicks alone do not reveal every relevant unshown item. Let the candidate union include only distinct eligible identities. Filters applied after retrieval can remove otherwise useful items, so report coverage before and after important filters. A deleted item or an item unavailable in the request context should not inflate a metric whose target is eligible recommendations.

Candidate recall measures how much of the reference relevant set reached the union. A final slate drawn from that union cannot have more relevant identities than the union itself. The bound assumes the downstream system adds no new candidates; if another retrieval branch runs later, include it in the defined union. A final slate limit can impose an additional ceiling. Specify whether the evaluation averages per-request recall or pools counts across requests, because large relevance sets weight those two summaries differently.

Two candidate sources can return many of the same relevant items. Adding their individual relevant counts double counts the intersection and exaggerates union coverage. Preserve source membership so marginal coverage and replacement effects can be measured. A source with fewer standalone hits can still contribute valuable unique items or serve a poorly covered cohort. Candidate count, latency, freshness, and item diversity belong beside relevance coverage when deciding how much budget each source receives. Increasing a list size may improve one cohort while wasting work on redundant items in another.

For an eligible relevant item, the probability of being shown factors into reaching the candidate stage and surviving conditional on having reached it. This is a conditional-probability identity, not an independence assumption between retrieval and ranking. Approximate nearest-neighbor recall is another metric: it compares an index's returned neighbors with an exact search under a specified vector score. High index recall does not prove that the learned score captures user relevance. Conversely, a good relevance model can lose coverage through index approximation or stale item representations.

Replay representative requests against a versioned eligible catalog, retaining query features, retriever versions, index snapshot, source membership, and filter reasons. Compare exact-score neighbors with approximate retrieval separately from relevance judgments. Inspect new items, sparse-history users, language, geography, and freshness cohorts rather than relying on one pooled result. Trace whether a missed relevant item was absent from the index, retrieved but filtered, or retained but ranked below the slate. Each failure points to a different intervention and a different latency or storage cost.

The probability lab can show a deliberately independent count model, not retrieval quality. Reset, set both p=0.6 and n=10, predict mean six and variance 2.4, then Play or Step. Reset and set both p=0.8 and n=10 for mean eight and variance 1.6. The bars are exact binomial probabilities for equal-probability independent trials; playback highlights outcomes rather than sampling. Actual candidates share sources and relevance structure, so their outcomes need not satisfy that model. Use the worked set calculation to establish the real union ceiling.