Sparse reproducibility: compare routing identities before aggregate loss

Begin with a tiny reference using labeled tokens and known expert functions. Compare logits, full probabilities, selected identities, retained weights, capacity survivors, and final outputs as separate artifacts. Add exact ties and near ties with an explicitly supported tie rule. PyTorch 2.14 warns that tied top-k indices are not guaranteed stable, so an unstated index preference is not a portable reference. A deterministic selection requirement may need a deliberately implemented rule, whose numerical and performance cost should be evaluated.

Admission order can matter even when selection and total capacity are fixed. Position-based dropping depends on token order; score-based admission needs tie handling; randomized admission needs controlled random state. A token permutation may preserve a no-drop pointwise expert function but change a capacity-limited position policy. Do not demand permutation invariance from a policy that explicitly depends on position. Instead test its stated behavior and compare matched identities. The same distinction applies when changing routing groups or microbatch boundaries alters the capacity population.

Record data order, token masking, global expert identities, router parameters and dtype, gate normalization, capacity factors, admission policy, group membership, software versions, device architecture, and backend flags. Save optimizer and scaler state plus the relevant per-rank random-generator states for continuation. Dropout, router jitter, stochastic second choices, and randomized capacity selection may consume different random streams. A seed alone does not describe which operation consumed each value, particularly after graph or parallel-layout changes.

PyTorch's reproducibility guidance scopes deterministic settings to supported operations and environments rather than guaranteeing identical results across every version and platform. GPU reductions, atomics, fused kernels, and collective scheduling can alter floating-point order. Compare finite outputs and gradients with declared tolerances, while treating NaN and infinity under a separate policy. A small output difference can later change a near-tie selection, so track when routing first diverges. Distinguish repeatability within one configuration from equivalence between two changed algorithms.

Test pack and inverse-pack operations using unique assignment labels, including zero-row experts and empty peer splits. Validate a fixed-routing expert forward and backward path before enabling live routing. Check gate gradients with small perturbations that stay away from membership and admission boundaries, then test boundary behavior as discrete events. Compare checkpoint resume against an uninterrupted run under the same recorded conditions. When parallel layout changes, verify the intended mathematical policy first; local capacity or nondeterministic arithmetic can legitimately prevent exact identity replay.

The modular lab supplies two deterministic orderings over four labels. Reset, set m=4 and a=1, predict residues 0,1,2,3 before returning to zero, then Play or Step. Separately Reset, set m=4 and a=3, predict 0,3,2,1, then Play or Step. Both have period four, but their order differs. The instrument has no top-k, random generator, or capacity admission. Use the ordering contrast to construct a paper test where a two-slot position policy admits different labels, showing why equal counts alone do not prove the same executed computation.