Resources
Benchmarks and datasets for Urdu AI
Agha Ali Raza's group at LUMS builds benchmarks and datasets that show how well AI works for Urdu and for Pakistani contexts, covering bias, safety and cultural alignment in language models, question answering, speech recognition, emotion in speech, and audio deepfakes.
PakBBQ
PakBBQ is a benchmark for measuring social bias in language models in a Pakistani context. It adapts the Bias Benchmark for Question Answering to Pakistan, with more than 214 templates and 17,180 question-answer pairs in English and Urdu across eight bias dimensions, including religion, regional affiliation, socio-economic status, and language formality.
Paper page and citation · EMNLP 2025 (Main), Suzhou, China
Pak3H
Pak3H is the first human-validated, culturally contextualized Urdu benchmark suite for testing whether language models are helpful, harmless, and honest. It contains three datasets, PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty), built through manual cultural adaptation and dictionary-guided post-editing rather than machine translation alone.
Paper page and citation · EMNLP 2026 (Main), Budapest, Hungary
UrduCultureQA
UrduCultureQA is a benchmark for evaluating language models on Pakistani cultural awareness and robustness to bias. It contains 1,211 question triplets (an original, a counterfactual, and a confounder) across fourteen cultural domains and six Pakistani regions, in both Urdu script and Roman Urdu, and requires consistent answers across each triplet.
Paper page and citation · Findings of EACL 2027, Athens, Greece
IRTIQA
IRTIQA is a benchmark for testing whether self-evolving AI agents improve equally across languages. It provides aligned English, Urdu, and Urdu-English code-switched versions of four agent benchmarks, human-verified so that every item has the same correct answer in all three conditions.
Paper page and citation · Findings of EACL 2027, Athens, Greece
UQA
UQA is a question-answering and reading-comprehension dataset for Urdu, created by translating the Stanford Question Answering Dataset (SQuAD2.0) with a technique called EATS (Enclose to Anchor, Translate, Seek) that preserves the position of each answer in its paragraph. The dataset and code are public.
Paper page and citation · Download · LREC-COLING 2024, Torino, Italy
Urdu deepfake audio dataset
The Urdu deepfake audio dataset is a public collection of real and synthetic Urdu speech for training and testing deepfake detectors. It covers two text-to-speech spoofing attacks, Tacotron and VITS, and was designed with attention to covering and balancing the sounds of the language.
Paper page and citation · Download · Findings of ACL 2024, Bangkok, Thailand
SEMOUR+
SEMOUR+ is the first scripted, emotion-tagged speech database for Urdu with diverse accents: 14 hours and 27,640 recordings by 24 native speakers, gender-balanced and phonetically balanced, for building systems that recognize emotion in speech.
Paper page and citation · Language Resources and Evaluation (LRE), 2023
Urdu conversational speech datasets
This release includes the first spontaneous conversational speech dataset for Urdu (24 hours and 212 speakers, compiled from public talk shows) and 283 hours of synthetic multi-speaker conversations from 50 speakers, generated from non-conversational recordings, for training systems that work out who spoke when.
Paper page and citation · Computer Speech & Language, vol. 89, 2025
Urdu speech recognition benchmark
WER We Stand benchmarks three families of speech recognition models on Urdu read speech and conversational speech, and presents the first conversational speech dataset designed for benchmarking Urdu speech recognition.
Paper page and citation · COLING 2025, Abu Dhabi, UAE
Urdu paraphrase corpus
This corpus contains 3,147 Urdu sentence pairs, generated semi-automatically and manually tagged as paraphrased or not, for training and evaluating paraphrase and plagiarism detection systems. The corpus, models, and code are freely available.
Paper page and citation · Natural Language Engineering (NLE), vol. 30(2), 2024