D2AR is the most comprehensive benchmark of diffusion language models for Hindi to date. It evaluates 8 models across 4 Hindi NLP tasks with a rigorous, reproducible evaluation framework. The research was published as a full paper and the evaluation framework is fully open-source.
The Evaluation Pipeline
1. Data Preprocessing
Transliteration normalization, tokenization (using IndicNLP and HuggingFace tokenizers), and an 80/20 stratified train-test split. The dataset covers four Hindi language tasks balanced for domain diversity: news text, social media, formal documents, and conversational Hindi. Special attention was paid to code-mixed Hindi-English (Hinglish) samples which make up a significant portion of real-world Hindi NLP data.
2. Zero-Shot Prompting
All 8 models evaluated on identical prompts across 4 Hindi NLP tasks: sentiment analysis (3-class: positive, negative, neutral), named entity recognition (10 entity types), question answering (extractive and abstractive), and text summarization (3 dataset domains). Each prompt is templated in Hindi with consistent formatting across models to eliminate prompt engineering bias.
3. Metric Computation
Four metrics computed per task: BLEU (n-gram precision up to 4-grams), ROUGE-L (longest common subsequence based recall), BERTScore (multilingual variant using mBERT embeddings), and perplexity (causal LM perplexity for auto-regressive models, pseudo-perplexity for diffusion models). Each metric was computed with 95% confidence intervals using bootstrap sampling with 1,000 resamples.
4. Statistical Significance
Paired t-tests and bootstrap confidence intervals establish whether performance differences between model classes are statistically meaningful. The bootstrap analysis uses 10,000 resamples per model pair. Results are reported with Bonferroni-corrected p-values to account for multiple comparisons across the 28 model pairs.
5. Qualitative Error Analysis
Error taxonomy by task type and model architecture. A random sample of 200 errors per model was manually annotated into 12 error categories (gender agreement errors, case marking errors, compound verb splitting, entity boundary errors, etc.). This qualitative layer reveals failure patterns that aggregate metrics alone would miss.
Models Benchmarked
Auto-Regressive: GPT-NeoX (20B), OPT (6.7B and 13B), BLOOM (7.1B) — these models use causal attention where each token attends only to previous tokens, enabling left-to-right generation with strong sequential reasoning capabilities.
Diffusion-Based: Diffusion-LM, MDLM (Masked Diffusion Language Model), SEDD (Score Entropy Discrete Diffusion), PLM-Discrete (Permutation Language Model — Discrete) — these models learn the data distribution through iterative denoising rather than autoregressive factorization, allowing bidirectional context but lacking causal attention scaffolding.
Key Findings
Diffusion models showed competitive performance on generation tasks (text summarization: within 2.3% of auto-regressive) but lagged significantly on structured NLP tasks (NER: 14.7% gap, sentiment analysis: 11.2% gap). The architectural reason: diffusion models lack the causal attention scaffolding that auto-regressive models use for sequential reasoning, which becomes critical for morphologically rich languages like Hindi where word order and inflection carry semantic weight.
The full paper, dataset splits, and evaluation code are available on GitHub.