Engineering / research
Long context under memory and quality constraints
Connect positional behavior, context retention, cache policies, retrieval, and adversarial evaluation to the actual usefulness of longer inputs.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Long context under memory and quality constraints
Connect positional behavior, context retention, cache policies, retrieval, and adversarial evaluation to the actual usefulness of longer inputs.
- 01
Positional extrapolation and scaling tests
Separate algebraic position encodings and scaling rules from empirically demonstrated long-distance use and short-context retention.
29 min - 02
Long-context training mixture design
Measure sequence, token, and compute mixtures separately while controlling packing, supervision, and benchmark leakage.
29 min - 03
Context-window retention and eviction
Distinguish logical attention history from reusable physical cache storage and avoid treating surviving state as an exact archive.
29 min - 04
Attention-sink and recency policies
Explain normalization-sensitive sink retention while separating stable streaming from access to omitted historical evidence.
29 min - 05
Context compression and summary loss
Account for token reduction while preserving evidence provenance and testing which task-relevant distinctions a summary discards.
29 min - 06
Retrieval versus direct context selection
Choose an evidence path using task coverage and measured complete costs rather than assuming a larger prompt or a smaller retrieved set always wins.
30 min - 07
Document order and Lost-in-the-Middle tests
Vary evidence placement with matched content and actual token positions while controlling ordering semantics, truncation, and task shortcuts.
30 min - 08
Long-context citation and attribution tests
Verify claim-level source support and probe evidence use without mistaking citation syntax or a single perturbation for causal proof.
30 min - 09
Long-sequence cost and capacity models
Budget named KV, prefill attention work, and decode IO components while separating mathematical bounds from measured service behavior.
30 min - 10
Long-context release regression suite
Gate changes with matched length, position, language, workload, provenance, and resource tests rather than one advertised context limit.
30 min