Dispatch and combine: preserve token identity across expert ranks

Start with the accepted token-expert assignment list after the chosen capacity policy. Each entry needs enough information to associate a token representation with a destination expert and later restore its contribution to the original token. Top-k can create several entries from one token. A local permutation groups entries by destination rank, and another arrangement may group arrivals by local expert. Keep the inverse mapping or equivalent indices; expert output order is not automatically the original sequence order.

A variable-split all-to-all exchanges the packed representations according to an agreed peer layout. Send and receive counts, dtypes, ordering, communicator membership, and matching collective sequence must be compatible. A rank with zero useful assignments still participates according to the communication protocol. PyTorch 2.14 documents split-size forms for all_to_all_single, but the exact runtime and backend support must be checked. Megatron-Core 0.15.0 also offers dispatcher variants, so all-to-all is a selected implementation path rather than the definition of every sparse layer.

Under a simple path that sends one representation per remote accepted assignment and returns one same-width expert output, count each payload once in each direction. Local expert work contributes no network payload at this boundary. A remote fraction can convert assignment count into remote count only when its denominator is explicit. Padding, routing metadata, collective protocol overhead, repeated link traversals, tensor-parallel traffic, and backward exchanges are additional quantities. Do not call this payload count total network bytes without stating what has been omitted.

If several selected experts share a destination rank, a dispatcher might reduce duplicate transfer of the same token and expand it locally, while another implementation sends each branch separately. Such deduplication changes the simple per-assignment count. Activation compression or differing output widths also changes it. Measure peer volumes and bytes using the actual packing protocol. A rank's aggregate sends do not reveal the busiest link or simultaneous contention, and dividing cluster-total bytes by one link's bandwidth usually produces an invalid timing estimate.

Gradient flow follows the actual accepted and weighted branches. Expert gradients need the token activations assigned to that expert, while token-input gradients combine contributions from its surviving branches. Router gradients depend on where weighting occurred and which gate convention was used. Expert replicas in a data-parallel dimension may need gradient synchronization over their own group rather than an unrelated world group. Check these axes explicitly; a forward path that preserves values can still train incorrectly if backward uses a stale permutation or wrong reduction group.

The modular lab offers only a small ordering exercise. Reset, set m=4 and a=1, predict period four visiting all four residues, then Play or Step. Separately Reset, set m=4 and a=2, predict period two, then Play or Step. The second orbit misses half the residues, illustrating why an assumed inverse map needs a real bijection over its intended entries. This is not an all-to-all or dispatcher simulator: it has no peer buffers, replicated token branches, communication time, or gradient storage. Verify actual pack/unpack identities with labeled tensors.