2.8 trillion Indic tokens arrive with the recipe: 22 languages, 34 script setups, open pipeline
Synopsis
NVIDIA and IIT Madras' AI4Bharat lab released Nemotron-pretraining-Indic-v1, about 2.8 trillion tokens across 22 Indian languages, together with the open GPU-accelerated pipeline that generated it. Open synthetic Indic data had been limited, while these 22 constitutionally listed languages are first languages for 97% of Indians. The corpus holds native-script data for all 22 languages and romanized data for twelve, 34 language-script configurations, with every record traceable to a source document and translations scored out of 100, repaired and rescored below 75.
Interpretation
Open synthetic pretraining data for 22 Indian languages now reaches roughly 2.8 trillion tokens, shipped with a reusable generation recipe. Open synthetic Indic training data had been limited, and AI4Bharat head Mitesh Khapra attributed Indian-language technology lagging English to not having enough data for Indian languages. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer, and the counts describe the retained corpus rather than a training run's token budget.
The generation workflow chains translation, quality assessment and targeted repair into one traceable pipeline. It draws on documents from 15 Nemotron-CC-v2 subsets covering web text, question-answer material, mathematics and code, orchestrated with NVIDIA's open-source NeMo Data Designer framework. Twelve languages are translated in full by Gemma-4-31B-it and scored by the same model against an MQM-inspired rubric, keeping only outputs at 75 or above, with lower scores repaired from assessment feedback and reassessed, recovering 438.4 billion tokens.
The other ten languages switch to Indic Translate, a document-level translation model, with adjustments for long documents and script stability. These ten languages produce native-script data only, without feedback-based rewriting. Eight keep the whole-document approach, Santali translates overlapping windows and reconstructs the document, and Manipuri is translated into Bengali script before conversion to Meitei Mayek.
On Nemotron 3.5 Super VL, adding Indic continued pretraining raised scores on all five reported Indic knowledge and reasoning benchmarks. The stage's configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mixture of synthetic and natural Indic text and the remaining 90% keeping the preceding mixture's relative proportions. The before-and-after checkpoints show gains from 4.98 to 9.82 percentage points, but the comparison covers a stage combining synthetic and natural Indic data rather than isolating the synthetic corpus's contribution.
Perspective
The corpus targets model builders who need Indic pretraining data, covering native-script data for 22 languages and romanized data for twelve, 34 language-script configurations tagged by language, script and region. Translation broadens access to the source material but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture, so it is meant to be combined with appropriately sourced natural Indic text and evaluated for the intended use case.
Quality assessment is primarily automated, complemented by sampled human review whose coverage varies across languages, and in the twelve-language workflow the translator also serves as the judge. Retained text can still contain errors, and the ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates, not semantic duplicates. The CPT gains support the combined synthetic-and-natural recipe, while mixture choices for other models and languages remain for further study.