Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face releases 2.8 trillion token Indic pretraining dataset with open data generation pipeline
Hugging Face has published a large-scale Indic language pretraining initiative comprising 2.8 trillion tokens alongside an open data generation pipeline, representing a major push for multilingual AI development in South Asian languages. This matters because Indic languages are historically underrepresented in large language model training data, and open pipelines at this scale can meaningfully shift the accessibility of high-quality non-English AI. The open pipeline component is particularly significant as it enables the broader research community to replicate or extend the work.