Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2025

PakBBQ: A Culturally Adapted Bias Benchmark for QA

Abdullah Hashmat, Muhammad Arham Mirza, Agha Ali Raza

EMNLP 2025 (Main), Suzhou, China · CORE A*

Summary

Language models are mostly tested for bias using Western examples, which miss the stereotypes that matter elsewhere. PakBBQ adapts a widely used bias benchmark for question answering to Pakistan: more than 214 templates and 17,180 question-answer pairs in English and Urdu, covering age, disability, appearance, gender, socio-economic status, religion, regional affiliation, and language formality. Testing multilingual models, we find that adding clarifying context raises accuracy by 12% on average, that models show stronger counter-bias behavior in Urdu than in English, and that negatively framed questions reduce stereotypical answers.

Cite this paper

Abdullah Hashmat, Muhammad Arham Mirza, Agha Ali Raza. PakBBQ: A Culturally Adapted Bias Benchmark for QA. EMNLP 2025 (Main), Suzhou, China.

@inproceedings{hashmat2025pakbbq,
  author = {Abdullah Hashmat and Muhammad Arham Mirza and Agha Ali Raza},
  title = {{PakBBQ: A Culturally Adapted Bias Benchmark for QA}},
  booktitle = {EMNLP 2025 (Main), Suzhou, China},
  year = {2025},
  url = {https://aclanthology.org/2025.emnlp-main.818/}
}

More on this theme

All 18 works on this theme →