Engineering / advanced
Schedule generation against latency targets
Analyze mixed prompt and generation work through admission, batching, fairness, and feedback while measuring both token throughput and request latency.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 advanced
Schedule generation against latency targets
Analyze mixed prompt and generation work through admission, batching, fairness, and feedback while measuring both token throughput and request latency.
- 01
Name the computation and traffic behind each scheduling unit
Separate prompt processing, autoregressive decode, weight traffic, KV reads, and transfer overhead before using token counts as a scheduling cost.
28 min - 02
Schedule requests through explicit iteration-boundary states
Track waiting, prefill progress, decode eligibility, completion, preemption, and cancellation without treating an active batch as an immutable set.
28 min - 03
Reserve memory for the work an admitted request may still create
Distinguish iteration token budgets, resident KV allocation, enforceable output caps, and conservative reserve admission with explicit accounting scope.
28 min - 04
Choose prompt chunks at real interruption boundaries
Quantify a fictional chunk-size latency tradeoff while preserving prompt dependencies, decode service, and implementation-specific scheduling boundaries.
28 min - 05
Admit work only when a declared deadline model can support it
Use remaining-work estimates and slack while distinguishing a feasible toy schedule, uncertain online admission, and resource-safe execution.
28 min - 06
Estimate remaining generation without discarding the long requests
Distinguish output caps, realized lengths, censored observations, conditional remaining work, and predictor calibration across request cohorts.
28 min - 07
Choose a service unit before claiming tenant fairness
Account for variable prompt/decode costs, weighted service, bounded quanta, and backlog eligibility instead of equating equal request counts with fair resource shares.
28 min - 08
Carry flow control and cancellation all the way to resource owners
Bound generated-but-undelivered output, propagate aborts to scheduled work, and verify safe KV release without confusing socket closure with completed cleanup.
28 min - 09
Count requests that satisfy the whole latency contract
Define arrival cohorts and joint TTFT, inter-token, and completion thresholds while keeping raw token throughput, average TPOT, and qualified request goodput distinct.
28 min - 10
Replay offered arrivals without hiding overload
Preserve arrival timing, length correlations, client dispatch delays, warm-up state, and incomplete outcomes while measuring latency without coordinated omission.
28 min