Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2026

Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark

Abdullah Hashmat, Usman Naseem, Agha Ali Raza

EMNLP 2026 (Main), Budapest, Hungary · CORE A*

Summary

AI models are tuned to be helpful, harmless, and honest, but mostly in English and for Western contexts. We introduce Pak3H, the first human-validated, culturally contextualized Urdu benchmark suite for these three qualities, with one dataset each for helpfulness, harmlessness, and honesty. Rather than relying on machine translation, our pipeline uses manual cultural adaptation and dictionary-guided post-editing that prioritizes native speaker judgment. Testing open and proprietary models, we find systematic gaps: helpfulness drops in localized contexts, safety guardrails break down against regional risks, and honesty degrades on local facts. Fair multilingual evaluation needs human-guided localization.

Cite this paper

Abdullah Hashmat, Usman Naseem, Agha Ali Raza. Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark. EMNLP 2026 (Main), Budapest, Hungary.

@inproceedings{hashmat2026pak3h,
  author = {Abdullah Hashmat and Usman Naseem and Agha Ali Raza},
  title = {{Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark}},
  booktitle = {EMNLP 2026 (Main), Budapest, Hungary},
  year = {2026},
  url = {https://aclanthology.org/2026.emnlp-main.883/}
}

More on this theme

All 18 works on this theme →