Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2025
PakBBQ: A Culturally Adapted Bias Benchmark for QA
Abdullah Hashmat, Muhammad Arham Mirza, Agha Ali Raza
EMNLP 2025 (Main), Suzhou, China · CORE A*
Summary
Language models are mostly tested for bias using Western examples, which miss the stereotypes that matter elsewhere. PakBBQ adapts a widely used bias benchmark for question answering to Pakistan: more than 214 templates and 17,180 question-answer pairs in English and Urdu, covering age, disability, appearance, gender, socio-economic status, religion, regional affiliation, and language formality. Testing multilingual models, we find that adding clarifying context raises accuracy by 12% on average, that models show stronger counter-bias behavior in Urdu than in English, and that negatively framed questions reduce stereotypical answers.
Cite this paper
Abdullah Hashmat, Muhammad Arham Mirza, Agha Ali Raza. PakBBQ: A Culturally Adapted Bias Benchmark for QA. EMNLP 2025 (Main), Suzhou, China.
@inproceedings{hashmat2025pakbbq,
author = {Abdullah Hashmat and Muhammad Arham Mirza and Agha Ali Raza},
title = {{PakBBQ: A Culturally Adapted Bias Benchmark for QA}},
booktitle = {EMNLP 2025 (Main), Suzhou, China},
year = {2025},
url = {https://aclanthology.org/2025.emnlp-main.818/}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco