Mixture-of-Experts kernel engineering on Amazon SageMaker AI
AWS details kernel-level MoE optimizations on SageMaker targeting Hopper GPU decode-phase bottlenecks
“A fused MoE system merges token routing, data shuffling, and specialized network calculations into a single batched operation on the GPU.”
AWS describes low-level GPU kernel optimizations for Mixture-of-Experts models on SageMaker, focusing on a Swap-AB technique to better utilize NVIDIA Hopper's WGMMA instruction during the decode phase. The core problem is that MoE decode processes one token at a time, leaving Tensor Core capacity underutilized due to a hardware-enforced M=64 minimum dimension constraint. This is useful infrastructure signal for teams running MoE models at scale but is a niche engineering post rather than a broad industry announcement.