Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2024
Generalists vs. Specialists: Evaluating Large Language Models for Urdu
Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar
Findings of EMNLP 2024, Miami, FL, USA · CORE A*
Summary
Is one large general-purpose AI model enough for Urdu, a language with 70 million native speakers that remains underrepresented in language technology? We compare general-purpose language models with special-purpose models fine-tuned for individual tasks, across seven classification and seven generation tasks. The special-purpose models consistently come out ahead. For the generation tasks we also compare human evaluation with evaluation by language models, and find that the judgments of the strongest proprietary model track human ratings more closely than those of an open model.
Cite this paper
Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar. Generalists vs. Specialists: Evaluating Large Language Models for Urdu. Findings of EMNLP 2024, Miami, FL, USA.
@inproceedings{arif2024generalists,
author = {Samee Arif and Abdul Hameed Azeemi and Agha Ali Raza and Awais Athar},
title = {{Generalists vs. Specialists: Evaluating Large Language Models for Urdu}},
booktitle = {Findings of EMNLP 2024, Miami, FL, USA},
year = {2024},
url = {https://aghaaliraza.com/publications/2024--EMNLP--Generalists_vs_Specialists-Evaluating_Large_Language_Models_For_Urdu.pdf}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco