Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI
Together AI details research scaling long-context training to 5 million token sequence lengths via context parallelism.
Max Ryabinin, VP of R&D at Together AI, presents a research project on training models with context lengths up to 5 million tokens by breaking GPU memory barriers through context parallelism. The work is motivated by the rise of agents and video generation, which demand far longer contexts and strong temporal consistency at training time. It is a useful engineering-infrastructure signal but not a top-tier industry announcement.