How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song
MiniMax M3 is the first open-scale model combining native multimodality, 1M token context, and sparse attention
“decoding 15 times faster compared to full attention”
MiniMax released M3, an open-scale model trained natively multimodal from scratch, featuring a 1 million token context window and a proprietary sparse attention mechanism (MSA) that achieves 15x faster decoding than full attention. The architecture uses a two-branch design—a lightweight index branch to identify high-attention tokens, and a main sparse branch that only attends to that selected subset. While technically interesting, MiniMax is a smaller lab and the claims (e.g., replacing 90% of code work) are marketing-inflected; the sparse attention efficiency story is the most credible engineering signal here.