Sparse mixtures of experts

Suppose a layer has E feed-forward experts, each accepting a d-dimensional token and returning the same width. A router computes E scores per token. A top-k rule selects k experts and combines their outputs using specified gate weights. Some implementations normalize weights over the chosen experts, while others retain probabilities from the full router softmax. State that convention when reproducing an architecture because it changes output scale and gradients.

If expert architectures match, expert parameter storage grows roughly with E while active expert arithmetic per token grows roughly with k. Shared attention, router computation, embeddings, and other dense components still execute. Expert choice is usually token-dependent rather than a permanent assignment of semantic categories. A router selecting an expert for a token does not demonstrate that the expert has learned a clean human-interpretable specialty.

For T tokens with top-k routing, the mean number of expert assignments is Tk/E per expert. Real routing is imbalanced, especially early in training or under distribution shift. A capacity factor allocates headroom beyond that mean. A finite-capacity design must define overflow behavior: drop a branch, reroute, pad to a buffer, or allow variable work. These policies can affect output quality, fairness across tokens, and reproducibility.

Load-balancing losses encourage useful utilization, but equal counts alone are not the entire objective. Experts with different sequence distributions can have different execution costs, and padding can waste compute even when average routing looks balanced. Record counts per expert, overflow rates, router entropy, and performance across data groups. A routing policy can appear efficient on one training distribution while concentrating a new tenant's traffic onto one device.

When experts reside on different accelerators, dispatch and return commonly require all-to-all communication. Token representations and routing metadata move even though only a small subset of experts runs. Network bandwidth, topology, batch size, and overlap determine whether arithmetic savings become throughput gains. A very small local token batch may create tiny inefficient expert matrix multiplications. The slowest busy expert can also determine the layer's completion time.

Compare dense and sparse models at declared quality and serving budgets. Report total parameters, active parameters, peak memory, token throughput, and tail latency separately. Keeping all expert weights resident may exceed one accelerator even when active compute is modest. Loading experts on demand changes the memory-latency tradeoff. Evaluation must include router stability and overload behavior; a low average loss is insufficient evidence that every deployed batch fits its buffers.