Measure a frontier instead of selecting one flattering number

For each query, build a deterministic exact top-k reference using the same metric, normalization, eligible population, and document versions as the approximate search. Count unique logical IDs after deduplication. A returned document with a stale version is not automatically the current reference item. If equal scores straddle the cutoff, either fix a shared total tie order or report a carefully defined tie-aware alternative alongside the strict measure. Do not switch definitions only for difficult examples.

Recall@k measures overlap with geometric neighbors, not human relevance or successful downstream answers. Evaluate those separately. Also keep one-nearest-neighbor-found-in-top-R distinct from intersection recall at R: they answer different questions even when both are called recall informally. Include empty populations, short lists, and timeouts under a declared policy. Removing failures from the denominator can improve the reported average while the application actually receives fewer useful responses.

Use training data to learn coarse and PQ codebooks, a validation workload to select search parameters, and a held-out test workload to estimate the chosen configuration’s behavior. Corpus training and query evaluation play different roles, so record their sampling and any overlap deliberately. Parameter selection on the final test queries leaks information into the reported estimate even if no encoder weights changed. After extensive tuning, gather another independent query set or disclose the selection procedure.

Uncertainty should respect the sampling unit. Treating k neighbors from one query as k independent Bernoulli trials usually exaggerates the amount of evidence because those successes share a query and traversal. Independent query-level recalls lie in [0,1], allowing a conservative bounded-variable concentration bound for their mean. Real correlated sessions weaken that assumption; block or session-level analysis may be more appropriate. No interval fixes a biased workload that excludes the users or filters that fail.

A point is dominated if another measured configuration has at least its recall with no greater latency or memory, with at least one strict improvement, under the same workload and measurement boundaries. Specify which latency statistic is used; mean latency and p99 are not interchangeable objectives. Count resident vectors, codes, graph links, IDs, temporary buffers, and required reranking storage consistently. Build time and ingestion cost can be additional dimensions rather than costs quietly excluded from the decision.

Run load experiments at controlled concurrency and arrival patterns, record warm-up and cache conditions, and retain timeout counts. A query that is fast in isolation may queue badly under saturation. Repeat measurements enough to distinguish noise from a practical improvement, while avoiding a search for the luckiest run. Store the exact query list, configuration, machine description, and artifact hashes so an apparent regression can be traced to a changed workload, changed index, or changed measurement setup.