Publications/Natural Language Processing & LLMs for Low-Resource Languages
Workshop · 2025
The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation
Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza
GEM² Workshop at ACL 2025
Summary
Aligning language models with human preferences requires datasets in which competing answers are ranked, work that human annotators usually do. We automate both halves of it with teams of models. For judging answers, we compare a single model as judge, several as a jury, and models debating; a single strong judge proved the most consistent. For writing answers, a feedback loop in which one open model drafts and another reviews beat either model working alone about 72% to 74% of the time. We use the resulting pipeline to generate preference-optimization datasets.
Cite this paper
Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza. The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation. GEM² Workshop at ACL 2025.
@inproceedings{arif2025fellowship,
author = {Samee Arif and Sualeha Farid and Abdul Hameed Azeemi and Awais Athar and Agha Ali Raza},
title = {{The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation}},
booktitle = {GEM² Workshop at ACL 2025},
year = {2025},
url = {https://aclanthology.org/2025.gem-1.4/}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note GenerationCHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco