Engineering / research
Route and train sparse expert models
Examine routing, capacity, token movement, load balance, and numerical checks in sparse mixture-of-experts training without assuming sparsity removes communication costs.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Route and train sparse expert models
Examine routing, capacity, token movement, load balance, and numerical checks in sparse mixture-of-experts training without assuming sparsity removes communication costs.
- 01
Top-k routing: separate a discrete choice from its differentiable weights
Derive router gradients inside a fixed selection region, distinguish gate normalization conventions, and test what changes when expert membership changes.
26 min - 02
Expert capacity: distinguish a dropped branch from a rerouted token
Calculate assignment capacity and overflow while separating fixed buffers, padding, branch admission, rerouting, and dropless execution.
26 min - 03
Load-balancing losses: write the denominator before tuning the coefficient
Derive a stated Switch-style auxiliary objective and distinguish hard counts, soft probabilities, top-k normalization, and local versus global reduction scope.
27 min - 04
Dispatch and combine: preserve token identity across expert ranks
Follow routing metadata, packing, variable peer splits, expert execution, and inverse weighted combination through a defined expert-parallel communication path.
27 min - 05
Grouped expert GEMM: schedule irregular rows without changing the experts
Count useful expert arithmetic, distinguish grouped and block-sparse execution, and inspect how row counts, padding, and precision constrain kernel efficiency.
26 min - 06
Expert placement: optimize the traffic matrix without hiding routing changes
Connect token demand, rank placement, network cuts, and per-rank computation while distinguishing relocation from topology-constrained expert selection.
27 min - 07
Router precision: protect the selection margin as well as softmax
Separate stable probability evaluation from robust expert ordering, and evaluate selective precision, logit regularization, and near-tie sensitivity.
27 min - 08
Shared experts: keep common computation separate from routed choice
Account for always-active shared experts and fine-grained routed experts while distinguishing architecture changes from equivalent parameter or FLOP budgets.
27 min - 09
Load skew: name the population behind every expert counter
Measure requested, accepted, padded, and executed work by expert and rank without confusing normalized demand, overflow, or aggregate throughput.
26 min - 10
Sparse reproducibility: compare routing identities before aggregate loss
Build layered tests for gate arithmetic, selection, capacity admission, permutation, gradients, and repeatability under a precisely recorded sparse-training configuration.
28 min