Ranking error analysis by query class

Choose useful classes from the task: exact identifiers, compositional questions, temporal constraints, languages, ambiguous intents, or document formats. Record how queries enter a class and allow overlapping categories when the analysis requires them, while avoiding double counting in a mutually exclusive aggregate. Include class sizes and the traffic distribution. A class mean based on a few handpicked failures is a diagnostic illustration, not a population estimate. Keep discovered examples separate from the held-out set used to assess the final correction.

Inspect the full path for each error. Was the relevant content indexed and eligible? Did analysis preserve the identifier or negation? Did either retriever return it? Did fusion retain it at the reranker cutoff? Did the reranker see the decisive passage, and did diversity selection discard it? Preserve candidate identities, scores, ranks, and visible text so these questions have evidence. A wrong final answer can arise even with good retrieval, while a reranker cannot repair content absent from every candidate list.

A paired comparison uses the same queries, corpus snapshot, candidate eligibility, and judgment policy for both systems. Report the distribution of per-query differences as well as the mean: a small gain on common easy queries can offset large losses elsewhere. Query-level resampling can estimate uncertainty under a declared sampling model, but related queries, repeated users, or time clusters may require grouped treatment. Repeatedly searching many slices and reporting only the most favorable one creates selection bias. Reserve fresh evaluation for changes motivated by exploratory findings.

Ground-truth uncertainty deserves its own category. An unjudged passage, disputed interpretation, outdated annotation, or partial-support document can make a metric difference ambiguous. Collect blinded supplementary judgments where they affect the comparison, retaining assessor disagreement and provenance. Do not force every inspected case into either retriever failure or success when the evidence is unresolved. Separate observed errors, suspected label errors, and unknowns, then explain which conclusion would change under plausible alternate judgments. Confidence in a stage diagnosis should follow the available evidence.

Change the component implicated by the trace. A tokenizer fix, broader candidate branch, new reviewed hard negatives, reranker input repair, or different diversity policy addresses different failure mechanisms. Compare the corrected system against the same baseline, checking neighboring classes and latency costs for regressions. Keep uncertainty and sample size visible when deciding whether a change is supported. A dramatic example can justify an investigation, but a deployment claim needs a stable evaluation protocol and representative evidence.

The probability lab illustrates sample-size sensitivity for a hypothetical independent binary hit event. Reset, set p to 0.8 and trials to 10, predict mean count, then Step: eight. Compare a fresh Reset with both p at 0.8 and trials at 30 before Step: twenty-four. Dividing count variance by the squared trial count gives different mean variances. This exact binomial instrument is not the distribution of NDCG, does not resample real queries, and cannot resolve uncertain relevance judgments or estimate a class’s unknown true rate.