Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2027
IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?
Abdullah Hashmat, Danish Javed, Agha Ali Raza, Usman Naseem
Findings of EACL 2027, Athens, Greece (to appear) · CORE A
Summary
Self-evolving AI agents improve by learning from their own experience, but almost everything known about them comes from English. We ask whether that self-improvement carries over to other languages and release IRTIQA, a benchmark with aligned English, Urdu, and Urdu-English code-switched versions of four agent benchmarks, human-verified so that every item has the same correct answer in all three. Testing three self-evolving frameworks, we find that evolution reliably raises capability but rarely closes the language gap: evolved systems remain 12 to 26 points stronger in English on grounded reasoning and multi-step execution, and code-switching behaves as a distinct condition rather than as degraded translation.
Cite this paper
Abdullah Hashmat, Danish Javed, Agha Ali Raza, Usman Naseem. IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages? Findings of EACL 2027, Athens, Greece (to appear).
@inproceedings{hashmat2027irtiqa,
author = {Abdullah Hashmat and Danish Javed and Agha Ali Raza and Usman Naseem},
title = {{IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?}},
booktitle = {Findings of EACL 2027, Athens, Greece},
year = {2027},
note = {To appear},
url = {https://aghaaliraza.com/papers/2027-irtiqa-evolving-in-english-evaluated-in-urdu.html}
}
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco
- Large Language Models Show Dunning-Kruger-Like Effects in Multilingual Fact-CheckingScientific Reports 16, 7594 (2026), a Nature Portfolio journal