Expert placement: optimize the traffic matrix without hiding routing changes
Connect token demand, rank placement, network cuts, and per-rank computation while distinguishing relocation from topology-constrained expert selection.
On this page
The question for today
What you’ll understand
- Calculate remote payload from a stated demand and placement matrix.
- Distinguish expert relocation, replication, and group-limited routing.
- Evaluate critical-path and communication constraints with matched semantics.
Two placements of the same expert weights can produce different communication patterns for the same routing decisions. A routing rule constrained to nearby experts goes further: it can change which computation is selected. Separate these interventions before comparing quality and speed. The useful object is a demand matrix tied to token origins, expert identities, and actual device topology, not only an average fraction of remote tokens.
Placement maps a logical assignment onto a physical path
A token starts on an origin rank and requests an expert whose parameters have an owning rank or shard group. Placement determines whether that branch is local, crosses a fast device fabric, or crosses a node boundary. Build a count matrix of accepted assignments from origins to destinations, and translate it into bytes using the actual payload and duplication policy. The same global expert counts can produce different cut traffic because token origins differ. Include return paths and other parallel dimensions when profiling the complete layer.
Placing frequently co-selected experts together can reduce some destination fanout or communication, but it can also put their combined compute and memory demand on one rank. A rank hosting several popular experts can become a straggler even when network bytes fall. Expert relocation must move the associated weights and optimizer state consistently. Replicating a hot expert introduces another placement and synchronization design: replicas need a defined parameter-update policy and do not create free memory or bandwidth. Rebalancing decisions should include their migration cost and workload stability.
A nearby-expert constraint changes more than placement
Group-limited routing first selects eligible expert groups and then chooses experts within them under a stated scoring rule. Megatron-Core 0.15.0 documents device- and node-limited variants with explicit group scoring and selection parameters. This restricts the candidate set relative to unrestricted global top-k, so output identities and learning signals can change. A faster constrained run should be evaluated as a model-and-system variant, not as a transparent relocation of an unchanged forward graph. Match the intended expert activation budget and report the constraint.
Network topology also sets shared bottlenecks. Divide bytes by an effective rate for the matching direction and communication phase, and consider simultaneous sends, receives, contention, protocol startup, and dependency order. An ideal full-duplex link can carry opposing directions concurrently under suitable conditions; summing both directions and dividing by one directional rate can misstate the bound. Conversely, an aggregate cluster bandwidth number cannot remove a congested cut. Overlap only hides work that is actually concurrent and outside the remaining dependency chain.
Compare placements with a preserved assignment trace
Replay the same accepted token-expert trace to study placement independently of router learning, then run an end-to-end training comparison if the placement influences batching or selection. Collect per-peer volumes, per-rank expert rows, compute durations, exchange durations, and exposed critical-path time. A lower remote fraction can coexist with worse latency if it concentrates work or adds many tiny messages. Repeat across representative batches and data mixtures; an arrangement optimized for one transient hot expert may perform poorly after demand shifts.
The queue lab demonstrates only a single fluid bottleneck. Reset, set λ=4 and μ=3, predict q(2)=6, then use four Steps. Separately Reset, set λ=4 and μ=5, predict q(2)=2, then use four Steps. Both start with four work units, and the controls are not link bandwidth or expert placement. The separate M/M/1 residence-time readout applies only under its own stochastic assumptions and stable rates. Use the backlog arithmetic to identify demand exceeding service, then obtain the actual traffic matrix and topology for placement.
Work it through
Read remote work from a two-node demand matrix
Eight top-one accepted assignments originate four per node. Destination-node counts are [[3,1],[2,2]], with rows as origins A,B and columns as destinations A,B. Each dispatch and return payload has 1,024 two-byte elements, without deduplication.
Off-diagonal entries are one from A to B and two from B to A, giving three remote assignments. Diagonal entries require no cross-node payload under this count.
Each remote assignment has 1,024×2=2,048 bytes per direction, so aggregate dispatch payload is 6,144 bytes and equal-width return adds another 6,144.
Total dispatch-plus-return cross-node payload is 12,288 bytes, or 12 KiB. Its direction and phase distribution still matter; this total cannot be divided by an unrelated cluster bandwidth to infer latency.
The placement-specific matrix gives three remote branches and 12 KiB of stated forward payload, while compute balance and physical path costs remain separate measurements.
The living notebook
Explore the model
Interactive lab · queue
A bottleneck responds to matching demand and service units
Reset, set λ=4 and μ=3, predict q(2)=6, then use four Steps. Separately Reset, set λ=4 and μ=5, predict q(2)=2, then use four Steps. This initial-four fluid backlog has no topology, communication cuts, or placement controls.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA placement change must address the measured constrained resource rather than an unrelated capacity total.
Pause & reason
Check your understanding
Your turn
Try it yourself
A stated dispatch phase must inject 12 MB through a particular one-direction cut sustaining 6 GB/s, using decimal units. Find its ideal bandwidth-time lower bound. Why is this not the whole layer time?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Match bytes and bytes per second before considering other phases or work.
Compare with the solution
The bound is 12×10⁶/(6×10⁹)=0.002 seconds, or two milliseconds. Startup, contention, return communication, expert compute, other cuts, and dependencies can add cost. Only measured overlap under the actual schedule can determine how much of that phase remains exposed.
Keep the idea
Take it with you
- Origins and destinations determine communication; expert totals alone do not.
- Relocation and candidate-restricted routing are different interventions.
- Use matching cut, direction, phase, and critical-path measurements.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.