Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

Ma B, Eitzinger J, Köstler H, Wellein G (2026)

Publication Language: English

Publication Type: Conference contribution, Original article

Publication year: 2026

Event location: Virtual

Abstract

AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism, and the interconnect community has responded with four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology. All four rest on two assumptions about the workload: that routing imbalance is correctable by the system layer, and that the mock-token benchmarks evaluating them faithfully represent production routing. We introduce DODOCO to test both, instrumenting five open MoE checkpoints that span today's sequence-mixer designs (MHA, MLA, GQA, Gated DeltaNet and Mamba-2 SSM) under a factorial grid of six data conditions and a matched expert-parallelism scan on H100 clusters. Both assumptions fail. Scaling EP leaves per-expert load concentration essentially unchanged: the straggler is intrinsic to the routing decision the model makes, not to how its experts land on ranks. Mock tokens overestimate routing imbalance by up to a factor of 2.35, and the error is a level offset rather than a trend: it stays flat across a 32× batch-size sweep. Skewing the synthetic distribution toward realism (Zipf) widens the gap instead of closing it. A third pattern organizes the results: the architectures separate into a data-resilient band (MHA, Mamba-2), whose routing approaches uniform on real text, and a persistently concentrated band (MLA, GDN), with GQA intermediate. These bands, not the EP degree or the mock-data profile, are the right workload input to AlltoAll-aware interconnect and dispatch design.

Authors with CRIS profile

How to cite

APA:

Ma, B., Eitzinger, J., Köstler, H., & Wellein, G. (2026). Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory. In Proceedings of the IEEE Hot Interconnects Symposium 2026. Virtual.

MLA:

Ma, Bole, et al. "Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory." Proceedings of the IEEE Hot Interconnects Symposium 2026, Virtual 2026.

BibTeX: Download