Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Workshop · 2025

The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation

Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza

GEM² Workshop at ACL 2025

Summary

Aligning language models with human preferences requires datasets in which competing answers are ranked, work that human annotators usually do. We automate both halves of it with teams of models. For judging answers, we compare a single model as judge, several as a jury, and models debating; a single strong judge proved the most consistent. For writing answers, a feedback loop in which one open model drafts and another reviews beat either model working alone about 72% to 74% of the time. We use the resulting pipeline to generate preference-optimization datasets.

Cite this paper

Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza. The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation. GEM² Workshop at ACL 2025.

@inproceedings{arif2025fellowship,
  author = {Samee Arif and Sualeha Farid and Abdul Hameed Azeemi and Awais Athar and Agha Ali Raza},
  title = {{The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation}},
  booktitle = {GEM² Workshop at ACL 2025},
  year = {2025},
  url = {https://aclanthology.org/2025.gem-1.4/}
}

More on this theme

All 18 works on this theme →