Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2027

UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation

Muhammad Shazil Nadeem, Muhammad Taimur Jahanzeb, Muhammad Usman Zahid, Muhammad Farooq, Aleema Faisal, Agha Ali Raza, Awais Athar

Findings of EACL 2027, Athens, Greece (to appear) · CORE A

Summary

Does a language model actually understand Pakistani culture, or does it fall back on stereotypes? Existing benchmarks are Western-centric and blur these two failures together. UrduCultureQA separates them using triplets of questions (an original, a counterfactual, and a confounder) and gives credit only when a model answers all three consistently. It contains 1,211 triplets across fourteen cultural domains and six Pakistani regions, in both Urdu script and Roman Urdu. Evaluating ten models, we find that single-question accuracy systematically overstates cultural competence, and that cultural awareness and robustness to bias are largely separate abilities.

Cite this paper

Muhammad Shazil Nadeem, Muhammad Taimur Jahanzeb, Muhammad Usman Zahid, Muhammad Farooq, Aleema Faisal, Agha Ali Raza, Awais Athar. UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation. Findings of EACL 2027, Athens, Greece (to appear).

@inproceedings{nadeem2027urducultureqa,
  author = {Muhammad Shazil Nadeem and Muhammad Taimur Jahanzeb and Muhammad Usman Zahid and Muhammad Farooq and Aleema Faisal and Agha Ali Raza and Awais Athar},
  title = {{UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment Evaluation}},
  booktitle = {Findings of EACL 2027, Athens, Greece},
  year = {2027},
  note = {To appear},
  url = {https://aghaaliraza.com/papers/2027-urducultureqa-a-culturally-grounded-benchmark-for-urdu.html}
}

More on this theme

All 18 works on this theme →