Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2026
Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark
Abdullah Hashmat, Usman Naseem, Agha Ali Raza
EMNLP 2026 (Main), Budapest, Hungary · CORE A*
Summary
AI models are tuned to be helpful, harmless, and honest, but mostly in English and for Western contexts. We introduce Pak3H, the first human-validated, culturally contextualized Urdu benchmark suite for these three qualities, with one dataset each for helpfulness, harmlessness, and honesty. Rather than relying on machine translation, our pipeline uses manual cultural adaptation and dictionary-guided post-editing that prioritizes native speaker judgment. Testing open and proprietary models, we find systematic gaps: helpfulness drops in localized contexts, safety guardrails break down against regional risks, and honesty degrades on local facts. Fair multilingual evaluation needs human-guided localization.
Cite this paper
Abdullah Hashmat, Usman Naseem, Agha Ali Raza. Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark. EMNLP 2026 (Main), Budapest, Hungary.
@inproceedings{hashmat2026pak3h,
author = {Abdullah Hashmat and Usman Naseem and Agha Ali Raza},
title = {{Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark}},
booktitle = {EMNLP 2026 (Main), Budapest, Hungary},
year = {2026},
url = {https://aclanthology.org/2026.emnlp-main.883/}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco
- Large Language Models Show Dunning-Kruger-Like Effects in Multilingual Fact-CheckingScientific Reports 16, 7594 (2026), a Nature Portfolio journal