Agha Ali Raza

Projects

Resources

Benchmarks and datasets for Urdu AI

Agha Ali Raza's group at LUMS builds benchmarks and datasets that show how well AI works for Urdu and for Pakistani contexts, covering bias, safety and cultural alignment in language models, question answering, speech recognition, emotion in speech, and audio deepfakes.

PakBBQ

PakBBQ is a benchmark for measuring social bias in language models in a Pakistani context. It adapts the Bias Benchmark for Question Answering to Pakistan, with more than 214 templates and 17,180 question-answer pairs in English and Urdu across eight bias dimensions, including religion, regional affiliation, socio-economic status, and language formality.

Paper page and citation · EMNLP 2025 (Main), Suzhou, China

Pak3H

Pak3H is the first human-validated, culturally contextualized Urdu benchmark suite for testing whether language models are helpful, harmless, and honest. It contains three datasets, PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty), built through manual cultural adaptation and dictionary-guided post-editing rather than machine translation alone.

Paper page and citation · EMNLP 2026 (Main), Budapest, Hungary

UrduCultureQA

UrduCultureQA is a benchmark for evaluating language models on Pakistani cultural awareness and robustness to bias. It contains 1,211 question triplets (an original, a counterfactual, and a confounder) across fourteen cultural domains and six Pakistani regions, in both Urdu script and Roman Urdu, and requires consistent answers across each triplet.

Paper page and citation · Findings of EACL 2027, Athens, Greece

IRTIQA

IRTIQA is a benchmark for testing whether self-evolving AI agents improve equally across languages. It provides aligned English, Urdu, and Urdu-English code-switched versions of four agent benchmarks, human-verified so that every item has the same correct answer in all three conditions.

Paper page and citation · Findings of EACL 2027, Athens, Greece

UQA

UQA is a question-answering and reading-comprehension dataset for Urdu, created by translating the Stanford Question Answering Dataset (SQuAD2.0) with a technique called EATS (Enclose to Anchor, Translate, Seek) that preserves the position of each answer in its paragraph. The dataset and code are public.

Paper page and citation · Download · LREC-COLING 2024, Torino, Italy

Urdu deepfake audio dataset

The Urdu deepfake audio dataset is a public collection of real and synthetic Urdu speech for training and testing deepfake detectors. It covers two text-to-speech spoofing attacks, Tacotron and VITS, and was designed with attention to covering and balancing the sounds of the language.

Paper page and citation · Download · Findings of ACL 2024, Bangkok, Thailand

SEMOUR+

SEMOUR+ is the first scripted, emotion-tagged speech database for Urdu with diverse accents: 14 hours and 27,640 recordings by 24 native speakers, gender-balanced and phonetically balanced, for building systems that recognize emotion in speech.

Paper page and citation · Language Resources and Evaluation (LRE), 2023

Urdu conversational speech datasets

This release includes the first spontaneous conversational speech dataset for Urdu (24 hours and 212 speakers, compiled from public talk shows) and 283 hours of synthetic multi-speaker conversations from 50 speakers, generated from non-conversational recordings, for training systems that work out who spoke when.

Paper page and citation · Computer Speech & Language, vol. 89, 2025

Urdu speech recognition benchmark

WER We Stand benchmarks three families of speech recognition models on Urdu read speech and conversational speech, and presents the first conversational speech dataset designed for benchmarking Urdu speech recognition.

Paper page and citation · COLING 2025, Abu Dhabi, UAE

Urdu paraphrase corpus

This corpus contains 3,147 Urdu sentence pairs, generated semi-automatically and manually tagged as paraphrased or not, for training and evaluating paraphrase and plagiarism detection systems. The corpus, models, and code are freely available.

Paper page and citation · Natural Language Engineering (NLE), vol. 30(2), 2024