Retrieve for relevance only after checking scope, validity, and revocation
Separate retrieval ranking from evidence eligibility and revalidate source-dependent memories before their use in a current decision.
On this page
The question for today
What you’ll understand
- Filter stale, revoked, and out-of-scope records before a top-k result is used.
- Interpret a weighted retrieval score as a heuristic rather than a truth probability.
- Trace invalidation through source-linked summaries and caches.
A memory can be highly similar to the current question and still be invalid: its source changed, its scope differs, its retention ended, or its authority was revoked. Vector ranking answers a similarity question, not all of those validity questions. A reliable retrieval path makes eligibility explicit, preserves the source version, and can backfill when high-ranked records are rejected. It also checks whether evidence remained applicable between retrieval and an effectful action. A fast cache hit does not establish current correctness.
Eligibility and ranking are different stages
Attach scope, provenance, source revision, observation time, and validity status to records. Before use, apply the relevant access and task restrictions and check current revocation or supersession state. A record can be useful historical evidence while being ineligible as a current instruction or fact. If an index returns stale candidates, a final authoritative filter can reject them and retrieve more. Measure how often that happens, because an apparently full top-k response may shrink to very little usable evidence after the actual validity checks.
A time-to-live is one freshness policy, not proof that a record remains true throughout its lifetime. A source may change or be revoked earlier. Conversely, an old mathematical identity may remain applicable even after a generic recency score decays. Define freshness by record type and the action that will consume it. A derived memory needs its source dependencies reconsidered when they change; removing the original row alone does not remove an unsupported summary elsewhere. Provenance and invalidation systems model these relations explicitly rather than relying on similarity to discover them later.
A retrieval score is not a calibrated probability
A system can rank eligible records using relevance, recency, and importance, as the original Generative Agents explores in its simulated environment. Weights and feature normalization determine that heuristic. An embedding similarity near one does not mean the claim has ninety-nine-percent probability of being true, and adding a recency term does not turn the total into a Bayesian belief. Keep score components and their purpose visible. A different query, normalization population, or stored corpus can change rankings without changing the underlying factual evidence.
Retrieval also needs diversity and source dependence checks. Several near-identical passages can crowd out a contradictory update or an exception required by the task. Deduplicate logical sources and inspect the actual context assembled for the model. If a source cannot be resolved, preserve that failure rather than presenting its cached summary as equally current. Immediately before an effect, recheck required revisions or use an atomic precondition where supported. This closes a particular check-to-use gap; it does not promise that all external state remains fixed afterward.
Evaluate stale hits and revocation after retrieval
Test a record revoked while it remains in an approximate index, a source changed after retrieval, a conflicting newer record, and an inaccessible original source. Verify that summaries, prompt caches, and action preconditions follow the declared invalidation policy. Measure usable coverage, backfill cost, and downstream errors by record age and scope. A system that only tests fresh successful reads can miss the most consequential failure path. Research retrieval mechanisms should be compared under these controlled conditions rather than assumed correct because memory improved an unrelated benchmark.
The cache lab isolates reuse without any validity metadata. Reset, set both C=4 and s=4, predict four misses and twelve hits at the initial sixteen requests, then inspect or Play. Reset and set both C=2 and s=4 for sixteen misses. The empty fully associative LRU model accesses(i×s) mod 16, adds two requests per second, and caps at 4096. It has no source revision, query scope, TTL, revocation, or corrective update. Its hit count therefore cannot establish that a real retrieved memory remained usable.
Work it through
A top-five response can contain only two usable records
A query requests five usable memories. The index's first five candidates contain two revoked records, one expired record, and two eligible records. The next four candidates contain three eligible records and one out-of-scope record. These categories are disjoint and checked against authoritative status.
Filtering the first five leaves only two usable records. Reporting five successful memory hits would confuse index retrieval with eligible evidence coverage.
Checking the next four supplies three more eligible records, giving five usable memories after examining nine candidates. Four examined candidates were rejected in total, so usable yield is 5/9 rather than five out of five.
A separate record has TTL until 10:30 but was revoked at 10:15. At a 10:20 query it must fail the stated revocation policy despite its remaining ten minutes of TTL. A later effect also needs its required source-version precondition checked.
Ranking, validity, backfill, and check-to-use behavior have distinct costs and guarantees. A top-k similarity result alone does not complete the retrieval contract.
The living notebook
Explore the model
Interactive lab · cache
A cache hit contains no built-in validity claim
Reset, set both C=4 and s=4, predict four misses and twelve hits at initial N=16, then inspect or Play. Reset and re-enter both C=2 and s=4 for sixteen misses. The empty LRU model accesses(i×s) mod 16 and uses N=16+floor(2t), capped at 4096. Entries have no TTL, source revision, scope, revoked status, or semantic evidence.
- Hits / requests
- 0 / 16
- Hit rate
- 0%
- Address-stream period
- 16 addresses
A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forMeasure usable current evidence, not only the number of entries returned from storage.
Pause & reason
Check your understanding
Your turn
Try it yourself
A query needs four usable records. Six retrieved candidates contain two stale, one revoked, and three usable entries. A backfill adds two candidates, one usable and one out of scope. Compute examined count, usable result count, and usable yield under these checks.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Count index candidates separately from records that survive eligibility.
Compare with the solution
Eight candidates were examined and four are usable, giving a 50% usable yield. The result satisfies the four-record count under the stated checks, but does not by itself establish truth, sufficient task coverage, or unchanged source status at a later effect.
Keep the idea
Take it with you
- Apply scope, source, and validity checks independently of similarity ranking.
- TTL and heuristic scores do not establish current truth or authorization.
- Test backfill, revocation, source changes, and derived-record invalidation through actual use.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.