Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2024

Generalists vs. Specialists: Evaluating Large Language Models for Urdu

Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar

Findings of EMNLP 2024, Miami, FL, USA · CORE A*

Summary

Is one large general-purpose AI model enough for Urdu, a language with 70 million native speakers that remains underrepresented in language technology? We compare general-purpose language models with special-purpose models fine-tuned for individual tasks, across seven classification and seven generation tasks. The special-purpose models consistently come out ahead. For the generation tasks we also compare human evaluation with evaluation by language models, and find that the judgments of the strongest proprietary model track human ratings more closely than those of an open model.

Cite this paper

Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais Athar. Generalists vs. Specialists: Evaluating Large Language Models for Urdu. Findings of EMNLP 2024, Miami, FL, USA.

@inproceedings{arif2024generalists,
  author = {Samee Arif and Abdul Hameed Azeemi and Agha Ali Raza and Awais Athar},
  title = {{Generalists vs. Specialists: Evaluating Large Language Models for Urdu}},
  booktitle = {Findings of EMNLP 2024, Miami, FL, USA},
  year = {2024},
  url = {https://aghaaliraza.com/publications/2024--EMNLP--Generalists_vs_Specialists-Evaluating_Large_Language_Models_For_Urdu.pdf}
}

More on this theme

All 18 works on this theme →