Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2027

IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?

Abdullah Hashmat, Danish Javed, Agha Ali Raza, Usman Naseem

Findings of EACL 2027, Athens, Greece (to appear) · CORE A

Summary

Self-evolving AI agents improve by learning from their own experience, but almost everything known about them comes from English. We ask whether that self-improvement carries over to other languages and release IRTIQA, a benchmark with aligned English, Urdu, and Urdu-English code-switched versions of four agent benchmarks, human-verified so that every item has the same correct answer in all three. Testing three self-evolving frameworks, we find that evolution reliably raises capability but rarely closes the language gap: evolved systems remain 12 to 26 points stronger in English on grounded reasoning and multi-step execution, and code-switching behaves as a distinct condition rather than as degraded translation.

Cite this paper

Abdullah Hashmat, Danish Javed, Agha Ali Raza, Usman Naseem. IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages? Findings of EACL 2027, Athens, Greece (to appear).

@inproceedings{hashmat2027irtiqa,
  author = {Abdullah Hashmat and Danish Javed and Agha Ali Raza and Usman Naseem},
  title = {{IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?}},
  booktitle = {Findings of EACL 2027, Athens, Greece},
  year = {2027},
  note = {To appear},
  url = {https://aghaaliraza.com/papers/2027-irtiqa-evolving-in-english-evaluated-in-urdu.html}
}

More on this theme

All 18 works on this theme →