greatitself.
Continue
advanced25 min studyLesson 675 in Engineering

Context packing: budget the rendered evidence, not a pile of hits

Deduplicate by source-aware identity and choose a bounded evidence portfolio with explicit tokenization, ordering, and coverage assumptions.

Before you begin
On this page

The question for today

What you’ll understand

  • Calculate the evidence budget after prompt and output reserves.
  • Distinguish exact duplicates, overlapping spans, and meaningful version differences.
  • Evaluate packing and ordering without assuming more context always helps.

Retrieval produces candidates, while generation consumes a specific rendered input. Deduplication and packing connect those stages. Their job is to retain authorized evidence needed for the question within a real model budget, including labels, quotations, separators, and other prompt material. A high retrieval score is only one input to that decision; it does not make redundant or contradictory passages free.

Deduplicate without erasing a meaningful distinction

Exact duplicate hits from the same document version and source span can usually share one evidence block while retaining all query-hit lineage. Overlapping chunks may be merged into a larger authorized span if the mapping is valid and the final rendering remains bounded. Identical text in different documents can have different authority, dates, or permissions. A near-duplicate policy can accidentally delete a changed number, an exception, or a newer revision, so compare the attributes relevant to the task before treating similarity as interchangeability.

Keep deduplication decisions observable: record which candidates were merged, suppressed, or retained as distinct versions. Repeated copies from a syndicated source do not necessarily provide independent confirmation. Conversely, two genuinely independent records that state the same fact can matter for corroboration. Avoid an unconditional text-hash rule across principals or source identities. Deduplication reduces redundant input; it does not decide whether a source is correct, current, authorized, or sufficient for a multi-source claim.

Bevidence=W−Bfixed−Boutput−Bother,τ(Pfinal)+Boutput+Bother≤WB_{\rm evidence}=W-B_{\rm fixed}-B_{\rm output}-B_{\rm other},\qquad \tau(P_{\rm final})+B_{\rm output}+B_{\rm other}\leq W
W is the model's applicable total token budget under the stated interface, τ tokenizes the final rendered input, and the other terms are explicit nonnegative reserves. Some architectures or APIs expose separate limits, requiring their actual contract instead of this teaching combined-budget model.↔ Scroll wide equations horizontally.

Packing is a constrained evidence-selection problem

Count the generation tokenizer's tokens after rendering the question, history, system instructions, evidence labels, source links, and formatting. Separate input and output limits when the serving interface does. Individual block counts are useful for planning, but concatenation can change tokenization around boundaries, so validate the completed prompt. Truncation is a policy choice: chopping the end can remove a qualification or citation locator. Prefer a measured bounded unit or explicit extraction whose source span remains traceable.

A simple packing model can assign each candidate a task-specific value and cost, but relevance scores are not automatically calibrated evidence value. Multi-hop questions may require a pair of passages whose joint value exceeds either alone. Diversity, contradiction coverage, authority, and provenance can impose additional constraints. A greedy order is an implementation choice rather than a proof of optimal support. Keep the chosen order and omitted-evidence reason in the trace so a missing fact can be attributed to retrieval, deduplication, or packing.

max⁡xi∈{0,1}∑ivixisubject to∑icixi≤Bevidence\max_{x_i\in\{0,1\}}\sum_i v_i x_i\quad\text{subject to}\quad\sum_i c_i x_i\leq B_{\rm evidence}
This simplified additive selection model uses candidate token costs c_i and declared heuristic values v_i. It excludes pairwise evidence dependencies and final-render tokenization effects unless added explicitly. Solving it does not prove factual support or optimal generation quality.↔ Scroll wide equations horizontally.

Ordering and long context require empirical tests

Lost in the Middle reports position-sensitive performance in its tested models and tasks; it does not establish a permanent law for every model or prompt. Test evidence at different positions and with realistic distractors under the current configuration. Compare answer quality, citation support, and omitted claims at matched token budgets. Extra passages can introduce noise or consume space needed by a decisive exception. A model advertising a large input limit does not guarantee reliable use of every included fact.

The cache lab isolates the distinct working set in a repeated lookup stream. Reset, set C=4 and s=2, predict zero hits among sixteen initial requests, then Play or Step. Separately Reset, set C=4 and s=4, predict twelve initial hits, then Play or Step. The first stream cycles through eight addresses and the second through four. This is not text deduplication, token packing, or attention. Use it to distinguish raw lookup count from distinct retained identities, then measure the actual rendered evidence budget and support coverage.

Work it through

Pack distinct evidence with explicit reserves

Use a teaching total budget W=4,096 tokens, fixed input 480, output reserve 512, and no other reserve. After source-aware deduplication, prioritized blocks A,B,C,D cost [900,850,700,600] tokens. Assume these additive costs have been checked against the final rendering.

  1. The evidence budget is 4,096−480−512=3,104 tokens. Output reserve is not available for evidence.

  2. A repeated hit for A contributes no second block under the stated exact-identity deduplication. The four distinct blocks cost 900+850+700+600=3,050.

  3. All four fit, leaving 54 tokens of headroom. The complete budget is 480+3,050+512=4,042, so the final rendering remains below 4,096 under the stated assumptions.

The result

The distinct evidence set fits with 54 tokens to spare. This establishes size feasibility, not relevance, authority, or faithful use by the generator.

The living notebook

Explore the model

Interactive lab

Interactive lab · cache

Raw requests and distinct retained identities differ

Reset, set C=4 and s=2, predict zero initial hits in 16 requests, then Play or Step. Separately Reset, set C=4 and s=4, predict 12 initial hits, then Play or Step. The modulo-16 LRU stream has no text, tokenizer, evidence scores, or context window.

Raw requests and distinct retained identities differA fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Current readings: Hits / requests: 0 / 16; Hit rate: 0%; Address-stream period: 16 addresses. All parameters are adjustable with the labeled controls below.Fully associative LRU cache after 16 deterministic requests: 0 hits and 16 misses.Most recent 16 requestsaddress = i × 3 mod 160M3M6M9M12M15M2M5M8M11M14M1M4M7M10M13MCurrent cache order · oldest → newest471013Latest address 13 · miss: inserted as newestToy address stream · period 16 · starts empty
Cached addressMiss / evictionMost recent request
Hits / requests
0 / 16
Hit rate
0%
Address-stream period
16 addresses
t = 0 s

A fully associative LRU cache starts empty and requests address (i·s) mod 16. Hits move an address to the newest position; misses evict the oldest when full. Playback starts with 16 requests and adds two per second, capped at 4096 explicitly simulated requests. This toy stream is not a processor-cache benchmark. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forDeduplicate with source semantics and budget the actual final rendering.

Pause & reason

Check your understanding

Why can aggressive near-duplicate removal reduce answer quality?

Your turn

Try it yourself

A combined teaching budget is 2,048 tokens, with fixed input 400 and output reserve 256. Two evidence blocks cost 800 and 650 tokens. Can both fit, and how much remains if the priority policy keeps the first only?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Compute available evidence tokens before summing candidate costs.

Compare with the solution

Evidence budget is 2,048−400−256=1,392. Both blocks total 1,450, exceeding it by 58. Keeping the 800-token first block leaves 592 evidence tokens. Another packing policy could choose differently, but every policy must recount the actual final prompt and report omitted needed evidence.

Keep the idea

Take it with you

  • Deduplication must preserve meaningful source, version, and permission differences.
  • Token budgets apply to rendered inputs under the actual model interface.
  • Packing feasibility and empirical evidence use are separate properties.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.