Publications/Speech Processing, Recognition & Audio Security
Conference · 2025
WER We Stand: Benchmarking Urdu ASR Models
Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, Awais Athar
COLING 2025, Abu Dhabi, UAE · CORE A
Summary
How good is today's speech recognition for Urdu? We benchmark three families of models (Whisper, MMS, and Seamless-M4T) on read speech and on conversational speech, and present the first conversational dataset designed for benchmarking Urdu speech recognition. Seamless performs best on read speech, while Whisper leads on conversation. Looking beyond the headline word error rate to the most frequent wrong words and to insertions, deletions, and substitutions, we show that numbers alone can mislead for a low-resource language, and that Urdu needs a robust text normalization system for fair evaluation.
Cite this paper
Samee Arif, Aamina Jamal Khan, Mustafa Abbas, Agha Ali Raza, Awais Athar. WER We Stand: Benchmarking Urdu ASR Models. COLING 2025, Abu Dhabi, UAE.
@inproceedings{arif2025wer,
author = {Samee Arif and Aamina Jamal Khan and Mustafa Abbas and Agha Ali Raza and Awais Athar},
title = {{WER We Stand: Benchmarking Urdu ASR Models}},
booktitle = {COLING 2025, Abu Dhabi, UAE},
year = {2025},
url = {https://aghaaliraza.com/publications/2025--COLING--WER_We_Stand-Benchmarking_Urdu_ASR_Models.pdf}
}
- Dissecting ASR Failures in Low-Resource South Asian LanguagesInterspeech 2026, Sydney, Australia
- Conversations in the Wild: Data Collection, Automatic Generation and EvaluationComputer Speech & Language, vol. 89, 2025
- Deepfake Defense: Constructing and Evaluating a Specialized Urdu Deepfake Audio DatasetFindings of ACL 2024, Bangkok, Thailand
- Speech Emotion Recognition for the Urdu LanguageLanguage Resources and Evaluation (LRE), 2023
- Using Self-Attention DNNs to Discover Phonemic Features for Audio Deep Fake DetectionASRU 2021, Cartagena, Colombia