Retrieval-augmented generation

Begin with a corpus whose documents have stable identities, versions, timestamps, and access rules. Chunking decides which units can be retrieved and which surrounding context remains available. A chunk boundary can separate a rule from its exception; overlap can preserve context while increasing duplicated material. Dense embeddings retrieve by a learned similarity geometry, while lexical methods can be strong for names and exact identifiers. Choose and evaluate the retrieval design using the intended question distribution rather than assuming one index is universally best.

Query rewriting, candidate retrieval, reranking, and context selection are separate operations with their own failure modes and costs. Enforce authorization before exposing document content to a user or a downstream model. A generator instruction is not an access-control boundary. Preserve provenance through each transformation so a selected chunk can be traced to an exact source. Updating an index does not automatically update cached answers, replicas, or previously created embeddings; define a freshness and deletion policy for the complete path.

Context selection should prefer answer-bearing evidence, remove avoidable duplication, and retain enough surrounding qualification to prevent misreading. Count final serialized tokens, including source labels and instructions, and reserve output capacity. More retrieved tokens can raise prefill cost and obscure the relevant passage. Lost in the Middle studies sensitivity to where relevant material appears in long contexts; its findings motivate position-aware evaluation, not a universal claim that the middle is unusable for every model and task.

The original RAG formulation combines a retrieval distribution with a generation model. A retrieve-then-prompt pipeline need not implement that exact training or marginalization. Be explicit about which architecture is used. Ask the generator to cite actual source identifiers and distinguish retrieved facts from inference. If required evidence is absent or conflicting, abstention or a targeted follow-up can be the correct result. Retrieval improves access to information; it does not confer truth on every indexed document.

Maintain questions with labelled supporting passages, answer criteria, and cases where the corpus has no answer. Evaluate retrieval recall and ranking separately from supported-answer accuracy and citation entailment. If evidence never reached the prompt, modifying the generator can miss the problem. If evidence was present but the answer contradicted it, improving retrieval alone is unlikely to solve the failure. Include updated documents, conflicting versions, inaccessible content, and questions requiring evidence across more than one passage.

Compare to a no-retrieval baseline and to an oracle context containing verified evidence. The gap to oracle context helps locate retrieval limitations, while remaining oracle errors expose generation or task ambiguity. Track latency and cost for embedding, search, reranking, prefill, and decoding separately. Treat retrieved text as untrusted data, including instructions embedded within documents. Validate citations against the selected source versions and avoid publishing unsupported claims merely because the output is syntactically well formed.