Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2027
UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation
Muhammad Shazil Nadeem, Muhammad Taimur Jahanzeb, Muhammad Usman Zahid, Muhammad Farooq, Aleema Faisal, Agha Ali Raza, Awais Athar
Findings of EACL 2027, Athens, Greece (to appear) · CORE A
Summary
Does a language model actually understand Pakistani culture, or does it fall back on stereotypes? Existing benchmarks are Western-centric and blur these two failures together. UrduCultureQA separates them using triplets of questions (an original, a counterfactual, and a confounder) and gives credit only when a model answers all three consistently. It contains 1,211 triplets across fourteen cultural domains and six Pakistani regions, in both Urdu script and Roman Urdu. Evaluating ten models, we find that single-question accuracy systematically overstates cultural competence, and that cultural awareness and robustness to bias are largely separate abilities.
Cite this paper
Muhammad Shazil Nadeem, Muhammad Taimur Jahanzeb, Muhammad Usman Zahid, Muhammad Farooq, Aleema Faisal, Agha Ali Raza, Awais Athar. UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation. Findings of EACL 2027, Athens, Greece (to appear).
@inproceedings{nadeem2027urducultureqa,
author = {Muhammad Shazil Nadeem and Muhammad Taimur Jahanzeb and Muhammad Usman Zahid and Muhammad Farooq and Aleema Faisal and Agha Ali Raza and Awais Athar},
title = {{UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation}},
booktitle = {Findings of EACL 2027, Athens, Greece},
year = {2027},
note = {To appear},
url = {https://aghaaliraza.com/papers/2027-urducultureqa-a-culturally-grounded-benchmark-for-urdu.html}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco
- Large Language Models Show Dunning-Kruger-Like Effects in Multilingual Fact-CheckingScientific Reports 16, 7594 (2026), a Nature Portfolio journal