Engineering / research
Separate inference stages safely
Study placement, cache transfer, request routing, handoff, and recovery when prompt processing and token generation use different resources.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Separate inference stages safely
Study placement, cache transfer, request routing, handoff, and recovery when prompt processing and token generation use different resources.
- 01
Stage asymmetry: separate work before separating hardware
Analyze prompt processing and iterative generation with explicit client-visible latency boundaries and workload-dependent resource assumptions.
29 min - 02
KV transfer: count the actual cache representation
Inventory layers, KV heads, positions, dtype, and shards before deriving serialization and bandwidth bounds.
29 min - 03
Stage routing: admit the complete path, not the emptiest prefill
Combine stage queues, transfer placement, destination capacity, and request identity in an explicit admission decision.
29 min - 04
Cache handoff: separate copy completion from ownership commitment
Design an explicit acknowledged handoff state machine with source lifetimes, receiver validation, fenced ownership, and scoped durability assumptions.
29 min - 05
Partial retries: a byte count is not a valid cache prefix
Track generation-scoped chunk identity, contiguous coverage, integrity, and idempotent acknowledgments before resuming a transfer.
29 min - 06
Prefill queues: service-time variation matters before saturation
Use a scoped M/G/1 calculation to separate mean utilization, service-time variance, and latency-aware admission in a real prefill pool.
29 min - 07
Decode capacity: token service and cache placement constrain scaling
Account for rounded KV blocks, per-instance feasibility, future growth, and token-rate demand before adding decode replicas.
29 min - 08
Topology placement: shared cuts constrain aggregate handoff
Calculate cross-cut load and bottleneck bounds while preserving actual shard redistribution, device staging, and parallelism traffic.
29 min - 09
Compare architectures with matched workloads and useful outcomes
Design controlled serving experiments that preserve resource budgets, request distributions, output semantics, and failure accounting.
29 min - 10
Stage recovery: preserve the visible token prefix under explicit assumptions
Separate cache recomputation from output replay, generation ownership, durable token records, and client deduplication.
29 min