Retrieval versus direct context selection

An exact lookup can need one short passage, while a global comparison or synthesis may require evidence distributed across many documents. A retriever optimized for local relevance can miss a low-overlap source needed for a multi-document constraint. Direct context can include that source but still suffer from distractors, truncation, or weak positional use. Define the required evidence and evaluate whether it is present, visible, and actually sufficient for the answer. Candidate recall and generator reasoning are separate parts of that diagnosis.

Keep corpus eligibility, source versions, and task definitions consistent across the compared paths. Retrieval over a newer index than the direct-context snapshot is not an isolated comparison of context strategy. Chunk boundaries, duplicates, passage ordering, and metadata can alter what the generator sees. Log exact retrieved identities and text, including separators and instructions. If a hybrid fallback expands context when retrieval is uncertain, evaluate its missed activations as well as successful examples instead of assuming a confidence signal identifies every retrieval failure.

For a fixed-width dense-attention prefill, a squared-token proxy illustrates how prompt reduction can lower the attention-pair component. It does not model every projection, feed-forward layer, kernel, device, or cached prefix. Retrieval has its own embedding, search, fusion, reranking, and transfer costs; a query-aware compressor adds another inference stage. Measure complete time to first token and generation under representative arrivals rather than claiming a runtime speedup from token ratio alone. Repeated queries and prefix reuse can change the comparison substantially.

Both strategies must fit the actual serving limit with instructions and output reserve accounted for. Reducing the prompt can leave more output capacity, while a huge direct prompt can constrain completion length. A short retrieved set may reduce generation cost yet lower answer completeness. Report those quality and resource outcomes together. A routing policy can choose among paths, but its thresholds need held-out evaluation by task and query class; easy-query averages can hide expensive failures on the cases that needed broader evidence.

Compare direct context with a retrieved path and an oracle-evidence condition where the necessary sources are supplied deliberately. If the oracle succeeds but retrieval fails, inspect candidate coverage. If direct context fails despite containing the same evidence, test ordering, distractors, and reasoning separately. Use fresh source-dependent values to reduce shortcuts through model memory, while preserving realistic tasks as another test set. A single synthetic lookup result cannot settle the tradeoff for summaries, code repositories, or multi-document reconciliation.

The Roofline lab illustrates only a generic assumed traffic effect. Reset, set operational intensity to 8 and bandwidth to 80, predict the synthetic ceiling, then Step: 640 GFLOP/s. Compare a fresh Reset with both intensity at 2 and bandwidth at 80 before Step: 160 GFLOP/s. Its fixed compute ceiling is 1000. It does not retrieve documents, execute attention, count prompt tokens, or measure an LLM request; the plotted ceilings cannot establish which context path answers better or faster.