Shared experts: keep common computation separate from routed choice

In the teaching variant here, sum shared expert outputs with a gate-weighted selection of routed outputs and a residual input. Shared experts have no top-k membership decision for each token. Their outputs are not automatically the mean of routed experts, and they need not share parameters with them. Some architectures add gates or other transformations to a shared path, so write its coefficient explicitly. DeepSeekMoE's original formulation uses deterministic shared participation while its routed candidates retain their defined score and selection rule.

Always-active shared computation changes both modeling and systems behavior. Every token contributes to its gradient under the applicable loss, while each routed expert sees its selected population. This does not prove an expert learned general knowledge or that routed experts learned disjoint concepts; those are empirical hypotheses requiring ablations. A shared branch can also become a substantial compute cost. Replication, sharding, optimizer-state storage, and overlap depend on the training layout rather than on the word shared alone.

Splitting one feed-forward expert into smaller-width experts increases the number of combinations a router can select. If widths scale inversely with the split factor and the number of active experts scales with it, a simple matrix-weight and leading-FLOP budget can remain comparable. That accounting does not establish functional equivalence: different nonlinearities, parameter partitioning, and selection combinations define another model. Router cost, permutation metadata, small-group efficiency, and communication can also change even when the dominant expert arithmetic is matched.

DeepSeekMoE isolates some of the smaller experts as shared and reduces the number of active routed experts to retain its stated activation budget. Other systems may add shared computation on top of an existing routed budget instead, increasing active work. Distinguish those choices. Count shared and routed widths, their parameter storage, and actual accepted token rows. Do not compare only the number of expert objects because a narrow expert and a wide expert do not carry the same weight or matrix-work budget.

Compare no-shared, additive-shared, and budget-reallocated shared variants with explicit stored parameters, active arithmetic, token budget, and training objective. Disable or replace a branch only under a clearly described intervention; extra routed experts are not guaranteed to reproduce a trained shared branch. Inspect task quality, expert specialization metrics, accepted routing counts, and per-path time. Megatron-Core 0.15.0 documents an optional overlap path for shared experts with specific dispatcher and runtime requirements. Its availability does not guarantee that all shared compute is hidden.

The cache lab illustrates a repeated working-set pattern only. Reset, set C=4 and s=1, predict zero hits among the initial sixteen requests, then Play or Step. Separately Reset, set C=4 and s=4, predict twelve initial hits, then Play or Step. The second modulo-sixteen stream revisits four addresses and fits the toy LRU cache. It does not represent shared experts, weight residency, gradients, or hardware caches. Use it to ask what data is actually reused, then measure the real shared and routed weight traffic rather than inferring reuse from an architecture label.