Grouped expert GEMM: schedule irregular rows without changing the experts

For a simple two-linear-layer expert with width d, hidden width h, and n_e accepted rows, conventional matrix work is four times n_e d h. The count includes two multiply-add matrix products under the usual two-FLOP convention and excludes nonlinearities, biases, routing, and backward work. A gated expert with two input projections and one output projection has a different count, typically six times n_e d h under matching dimensions. Name the architecture instead of assigning one universal FLOP formula to every feed-forward expert.

Experts keep distinct parameters even if a scheduler groups their matrix operations. Grouping is not averaging their weights or concatenating token rows into one shared dense expert. Each row must be multiplied by its assigned expert matrices, with the appropriate leading dimensions and output mapping. A zero-row expert requires no useful matrix arithmetic but may still appear in metadata or a grouped API. Empty groups, odd widths, tail rows, and mixed local row counts need explicit support rather than relying on a balanced benchmark.

A grouped matrix interface presents several independent products to a scheduling mechanism so it can use resources more effectively than an unfavorable sequence of tiny launches. Backend implementations differ: kernel grouping, concurrent streams, persistent scheduling, or other strategies may be involved. Megatron-Core 0.15.0 documents its Transformer Engine grouped path for multiple local experts. Inspect the selected backend and version before claiming one launch shape or universal speedup. Row skew can leave a few long groups dominating completion even when launch overhead falls.

MegaBlocks instead reformulates expert computation using block-sparse operations that accommodate uneven assignment while avoiding token dropping. That mechanism is not identical to every grouped-GEMM library. Sparse block granularity, internal padding, metadata, and permutation still cost resources. Fixed-capacity dense padding and block-rounded sparse work have different wasted-row patterns. Compare useful rows against actually scheduled rows, and include temporary storage and preparation. Dropless refers to preserving assignments, not to the absence of all padding, zero work, or memory limits.

Record the distribution of rows per expert, number of local experts, hidden widths, dtypes, and selected precision path. Tiny groups may be dominated by overhead, while larger groups can reach a different compute or bandwidth regime. Quantized or specialized matrix paths can impose alignment and padding requirements that change the scheduled work. Check output and gradient agreement after reordering groups or changing accumulation. A speed comparison must include dispatch permutations and any required weight or activation conversion at its declared boundary.

The roofline lab isolates only a conditional reuse argument. Reset, set I=4 and B=80, predict ceiling 320 GFLOP/s, then Play or Step. Separately Reset, set I=8 and B=80, predict 640 GFLOP/s, then Play or Step. Its compute peak stays at 1,000 GFLOP/s. The instrument has no expert row distribution, grouped scheduler, padding, or sparse matrix representation. Use it to check the units of an intensity hypothesis, then test that hypothesis on actual irregular shapes rather than treating the curve as a grouped-kernel result.