Engineering / advanced
Manage memory across generation requests
Model parameter, workspace, and key-value allocations through paging, sharing, eviction, and reclamation under concurrent generation.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Manage memory across generation requests
Model parameter, workspace, and key-value allocations through paging, sharing, eviction, and reclamation under concurrent generation.
- 01
Account for weights, workspace, and the KV state actually resident on each rank
Calculate layer, head, token, and dtype contributions while separating GQA savings, tensor-parallel partitioning, replication, and non-KV memory peaks.
28 min - 02
A KV block table maps logical token ranges to a managed device pool
Translate token positions into block slots, quantify tail slack, and distinguish application-managed KV paging from operating-system pages or automatic host swapping.
27 min - 03
Prefix reuse needs identical state, explicit ownership, and a trust boundary
Construct a cache identity that covers the effective prefix, count shared physical blocks once, and distinguish active references from evictable retained cache entries.
29 min - 04
Diverging generation branches copy the shared writable tail
Track partial-tail ownership through parallel generation, distinguish immutable full-prefix sharing from copy-on-write, and release pruned branches after outstanding accesses finish.
28 min - 05
Eviction releases reusable storage only when the required state remains recoverable
Separate idle-prefix eviction from active-request preemption, identify replay dependencies, and compare recovery costs without changing the model's attention semantics.
28 min - 06
Variable lengths create several different forms of unusable capacity
Quantify fixed-block tail slack and contiguous-hole failures, then separate KV-pool occupancy from framework allocator fragmentation and workload reservation policy.
27 min - 07
Host offload exchanges device capacity for a dependency-bound transfer path
Account for transfer bytes, bandwidth, fixed overhead, staging, and synchronization while distinguishing possible overlap from guaranteed latency hiding.
28 min - 08
KV compression saves representation bytes under a separately tested quality contract
Count packed values and metadata, distinguish quantization from lossless state recovery, and evaluate attention and generation quality across contexts and workloads.
29 min - 09
Admission reserves future obligations instead of trusting an average output length
Separate live physical blocks from unmaterialized promises, account for sharing and speculative state, and release canceled work only at safe lifetime boundaries.
29 min - 10
Attribute memory growth at the tensor, allocator, pool, and device boundaries
Distinguish retained tensors, cached reservation, logical pool leaks, external allocations, and transient peaks with synchronized workload evidence and versioned diagnostic tools.
29 min