Better Indic Models Need Better Benchmarks

An important factor driving the progress in building LLMs has been the availability of high-quality evaluation benchmarks that allow researchers (and model builders) to systematically measure model capabilities, compare different approaches, and track progress over time. However, this progress has been heavily concentrated in English and other high-resource languages. For Indian languages, the situation is quite different. Although several promising Indic and multilingual LLMs have emerged in recent years, the resources available to comprehensively evaluate their capabilities remain relatively limited.

Consider a model that performs exceptionally well on mathematical reasoning benchmarks in English. Would it demonstrate the same reasoning ability when asked those questions in Hindi or Tamil? Or what about a model that scores highly on general knowledge benchmarks but struggles with questions about Indian history, geography, or culture?

And there's another challenge. The world of LLM evaluation is moving incredibly fast. As models become more capable, benchmarks have evolved from traditional knowledge and reasoning tasks to complex agentic and real-world evaluations. But this progress has been far from uniform across languages. Yet, for Indian languages, model builders often still have to rely on older generations of benchmarks, leaving a growing gap between what models can potentially do and what we can meaningfully measure.

We cannot meaningfully improve what we cannot reliably measure.

At AI4Bharat, we've long believed that building better language models requires not just better training data, but equally strong evaluation resources. The data side of this effort is already open: our companion release, Nemotron-pretraining-Indic-v1, provides roughly 2.8 trillion tokens of open Indic pretraining data across 22 languages (see the companion blog, Scaling Indic Pretraining: 2.8 Trillion Tokens). After all, benchmarks help us understand what our models can do, where they struggle, and what we should work on next. That's why, together with NVIDIA, we're introducing Indic-LLM-Eval-Suite, an open, evolving collection of offline benchmarks for evaluating LLMs across Indian languages.

Our focus is on offline, benchmark-based evaluation. We release benchmark datasets, evaluation implementations, and associated methodologies so that researchers and model developers can independently evaluate their models, reproduce results, and track improvements over time. By making these resources openly available, we aim to establish a common set of reliable benchmarks that can be used to systematically measure and compare LLM capabilities across Indian languages.

As part of our joint effort with NVIDIA, we're releasing the first version of Indic-LLM-Eval-Suite, along with the benchmark construction and validation methodology used to develop it.

And importantly, we don't want to build this alone. We hope this becomes a shared effort that the broader research community contributes to, improves, and helps keep up to date as models continue to evolve.

What's Missing in Indic LLM Evaluation?

Evaluation benchmarks have played a central role in the rapid progress of LLMs. From MMLU and GSM8K to GPQA and SWE-bench, benchmarks have helped researchers systematically measure model capabilities and identify areas for improvement. More recently, evaluations such as Terminal-Bench and GDPval have pushed toward measuring increasingly complex, real-world capabilities.

But when it comes to Indian languages, several challenges remain.

1Our Indic language benchmarks aren't keeping pace with our models

Most widely adopted LLM benchmarks have been developed primarily for English. As a result, even when models support Indian languages, much of their reported performance is still based on English evaluations. And strong English performance doesn't necessarily translate to equally strong performance in Indian languages. A model might solve a complex reasoning problem in English but struggle with the same task in Bengali, Kannada, or Odia.

The gap also extends beyond languages to the capabilities we evaluate. As the broader LLM community moves toward harder reasoning tasks, tool use, agentic workflows, and real-world applications, evaluations for Indian languages need to keep up. Otherwise, we risk evaluating increasingly capable models using benchmarks that no longer tell us enough about what they can do.

If our benchmarks stay the same while our models improve, how do we know what progress we're making?

2Great Indic benchmarks exist. They're just scattered.

This isn't to say that the Indic research community hasn't made progress on evaluation. Over the years, researchers have developed valuable benchmarks addressing different aspects of Indian language modeling. IndicGenBench evaluates generation capabilities, MILU focuses on India-centric knowledge and reasoning, and IndicIFEval tests instruction following, to name just a few.

But no single benchmark can tell us everything about a model. A model that excels at factual knowledge might struggle with mathematical reasoning. Another might be good at multilingual generation but poor at following precise instructions.

To get a meaningful picture of model capabilities, we need evaluations across different tasks, languages, and settings. Today, these benchmarks often come with different language coverage, evaluation procedures, and implementations. For a model developer, putting together a comprehensive evaluation can mean discovering, downloading, configuring, and running several independent benchmark suites.

What if we could bring these efforts together into a common, reproducible collection, while preserving their original methodologies and giving credit to their creators?

That's one of the key motivations behind Indic-LLM-Eval-Suite.

3Can we just translate English benchmarks?

One seemingly straightforward approach to addressing the shortage of Indic evaluations is to translate popular English benchmarks into Indian languages.

Indeed, translation is a powerful mechanism for extending benchmarks to multiple languages at scale [1, 4]. For tasks such as mathematical reasoning, translating an existing question into another language allows us to assess whether the model can perform the same underlying reasoning task in that language.

However, translation is not without challenges. Consider a mathematical problem where a particular condition is incorrectly translated. The resulting question might now have a completely different answer, even though the original answer key remains unchanged. Similarly, an instruction-following benchmark might contain constraints involving specific words, characters, or formatting requirements that cannot be directly translated without changing the intended task.

These errors can lead to misleading evaluation scores, where a model is penalized not because it lacks the required capability, but because the underlying benchmark is incorrect. Importantly, the problem is not limited to translated benchmarks. Even naturally sourced evaluations can contain incomplete questions, missing figures, incorrect answer keys, or other artifacts. Recent research has highlighted these issues in widely used English and multilingual benchmarks, reinforcing the need for careful auditing and validation [2, 3].

4Indian languages need more than translated benchmarks

There's another important piece to the puzzle. Translating an English benchmark into Telugu doesn't automatically make it a benchmark of knowledge about Andhra Pradesh or Telangana. Similarly, understanding Indian cultural practices, regional expressions, local institutions, or the many ways people mix languages in everyday conversations requires evaluations designed with these contexts in mind.

This gives us two complementary goals:

Both are important and building strong evaluations for both requires more than simply translating existing English benchmarks.

Introducing Indic-LLM-Eval-Suite

With these challenges in mind, we're excited to introduce Indic-LLM-Eval-Suite, developed by AI4Bharat at IIT Madras in collaboration with NVIDIA as part of our joint effort. Rather than creating yet another standalone benchmark, our goal is to build an open and continuously evolving collection of high-quality offline evaluations for Indian languages.

Our approach is guided by three broad principles:

What Are We Releasing Today?

For our initial release, we've assembled existing Indic benchmarks, refreshed some existing ones with improvements (MILU-1.5, IndicIFEval-1.5, …) and carefully adapted established English benchmarks spanning eight broad capability categories.

Our selection includes two broad categories of benchmarks.

• Existing Indic benchmarks. We incorporate benchmarks developed specifically for Indian languages and contexts, such as MILU, IndicIFEval, IndiVibe, and BhashaBench, among others.
• Translated English benchmarks. For capabilities that can be meaningfully evaluated across languages using equivalent tasks, such as mathematical reasoning and scientific question answering, we carefully translate established English benchmarks into Indian languages.

We've integrated these evaluations with NVIDIA NeMo-Gym, providing a common evaluation interface while retaining benchmark-specific scoring procedures. This allows model developers to quickly run standardized offline evaluations under the same configuration and reproduce results across multiple languages and capabilities. This mirrors the data side of the program: the companion corpus is generated with NVIDIA NeMo Data Designer, so a single NVIDIA tooling stack spans both data generation (NeMo Data Designer) and evaluation (NeMo-Gym).

Results snapshot

To demonstrate the utility of the suite, we evaluated several contemporary open-weight Indic and multilingual LLMs across these capabilities. The results reveal that no single model consistently performs best across all tasks. While some models demonstrate strong mathematical reasoning and coding capabilities, others perform better on translation, instruction following, or conversational quality. The table below presents a snapshot of these evaluations as of October 2026. For a more detailed analysis of model performance across individual languages and capabilities, we refer readers to our technical report.

Of course, this first release is only a starting point. Several important capabilities remain underrepresented, and expanding this coverage forms a central part of our future work. As we continue to add new benchmarks and incorporate contributions from the research community, we also plan to periodically refresh these results to track the progress of Indic LLMs over time.

Building Benchmarks We Can Trust

Bringing benchmarks together is one challenge. Making sure they're reliable is another.

Our initial investigations highlighted that simply translating existing benchmarks using state-of-the-art machine translation systems does not necessarily produce reliable multilingual evaluations. Translation errors can alter the meaning of questions, introduce inconsistencies between questions and their answers, or violate task-specific requirements.

To address these challenges, we developed a benchmark-aware, human-in-the-loop pipeline for constructing and validating multilingual evaluations. Our methodology consists of the following stages:

1. 01

   Benchmark-aware Translation

   Rather than using a generic translation prompt, we tailor the translation instructions to the particular benchmark, paying special attention to preserving the underlying task and its evaluation criteria.

2. 02

   Automatic Quality Assessment

   We evaluate the translated instances to identify potential semantic inconsistencies, linguistic errors, and violations of benchmark-specific constraints.

3. 03

   Targeted Correction

   Instead of discarding every problematic instance, we use the feedback from the assessment stage to selectively correct errors, followed by another round of evaluation.

4. 04

   Native-speaker Verification

   Finally, native-language speakers independently review sampled instances to verify their correctness and identify errors that may have escaped automated quality checks.

An interesting observation from our experiments was that even translations that passed automated quality checks could still contain errors. More importantly, we found that even existing Indic benchmarks were not immune to such issues. For instance, our audit of MILU revealed several incomplete questions, missing figures, and incorrect translations, which we systematically addressed in a revised release, MILU-1.5. Similarly, we identified and fixed several issues in the evaluation code of IndicIFEval, where certain constraints were incorrectly checked or silently ignored. These findings highlight the importance of not just creating new benchmarks, but also continuously auditing and improving existing ones.

These observations reinforce our belief that benchmark construction must emphasize quality as much as scale. As part of this release, we are also making our construction methodology and code available to the community, enabling researchers to reproduce, audit, and improve our process for developing multilingual benchmarks. For more details on the methodology and findings, we encourage you to check out our accompanying technical report.

An Ongoing, Community-Driven Collection

Here's something we feel strongly about: building comprehensive evaluations for Indian languages is too big a task for any one research group. India's linguistic and cultural diversity is enormous. And as LLMs become more capable, the range of things we need to evaluate will only grow.

We don't want Indic-LLM-Eval-Suite to become a static collection of benchmarks that gets released once and slowly becomes outdated. Over the past few years, several researchers and organizations have developed valuable benchmarks addressing different aspects of Indian language understanding and generation. We believe these efforts should be brought together rather than recreated independently.

In particular, we're excited about several directions.

Rather than reinventing these efforts, our vision is for Indic-LLM-Eval-Suite to become a common, open collection of high-quality offline benchmarks contributed to and maintained by the research community. We have already taken steps in this direction by incorporating benchmarks developed by the community, including IndiVibe and Indic-Translate-Bench from Sarvam AI, which evaluate conversational quality and machine translation, respectively, and BhashaBench from BharatGen, which evaluates India-centric knowledge. By bringing these complementary benchmarks together under a common evaluation framework, while preserving their original methodologies, we hope to make comprehensive Indic LLM evaluation more accessible and encourage further contributions from the broader research community.

Better Evaluation, Better Models

We're at an exciting point in the development of AI for Indian languages.

With increasingly capable models and growing training resources, the community has made remarkable progress. But building better models is only half the story. We also need reliable ways to understand what these models can do, where they still fall short, and whether our improvements are actually making a difference.

Indic-LLM-Eval-Suite is our attempt to help close this gap. It's a starting point and not a finished product. And we hope it becomes a resource that the broader research community actively uses, contributes to, and improves.

Acknowledgments

This work is part of the joint effort between AI4Bharat at IIT Madras and NVIDIA. We acknowledge the contributions of the researchers, annotators and organizations whose benchmarks form the foundation of the initial release, and look forward to collaborating with the broader community on future versions of the suite.

Resources

• Technical Report Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs — NVIDIA and AI4Bharat
• Benchmarks & Datasets huggingface.co/collections/ai4bharat/indic-llm-eval-suite
• Code github.com/AI4Bharat/indic-llm-eval-suite
• Companion Data Blog Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

References

1. Shi et al. (2023). Language Models are Multilingual Chain-of-Thought Reasoners. ICLR 2023. arXiv:2210.03057
2. Gema et al. (2025). Are We Done with MMLU? NAACL 2025. ACL Anthology
3. Thellmann et al. (2026). Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite. LREC 2026. ACL Anthology
4. Singh et al. (2025). Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. ACL 2025. ACL Anthology

Citation

@techreport{indicllmevalsuite2026,
  title       = {Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs},
  author      = {{NVIDIA} and {AI4Bharat}},
  institution = {NVIDIA and AI4Bharat, IIT Madras},
  year        = {2026}
}