Agha Ali Raza

Publications/Speech Processing, Recognition & Audio Security

Conference · 2025

WER We Stand: Benchmarking Urdu ASR Models

Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, Awais Athar

COLING 2025, Abu Dhabi, UAE · CORE A

Summary

How good is today's speech recognition for Urdu? We benchmark three families of models (Whisper, MMS, and Seamless-M4T) on read speech and on conversational speech, and present the first conversational dataset designed for benchmarking Urdu speech recognition. Seamless performs best on read speech, while Whisper leads on conversation. Looking beyond the headline word error rate to the most frequent wrong words and to insertions, deletions, and substitutions, we show that numbers alone can mislead for a low-resource language, and that Urdu needs a robust text normalization system for fair evaluation.

Cite this paper

Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, Awais Athar. WER We Stand: Benchmarking Urdu ASR Models. COLING 2025, Abu Dhabi, UAE.

@inproceedings{arif2025wer,
  author = {Samee Arif and Aamina Jamal Khan and Mustafa Abbas and Agha Ali Raza and Awais Athar},
  title = {{WER We Stand: Benchmarking Urdu ASR Models}},
  booktitle = {COLING 2025, Abu Dhabi, UAE},
  year = {2025},
  url = {https://aghaaliraza.com/publications/2025--COLING--WER_We_Stand-Benchmarking_Urdu_ASR_Models.pdf}
}

More on this theme

All 12 works on this theme →