Publications/Natural Language Processing & LLMs for Low-Resource Languages
Conference · 2026
Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation
Ehsan Latif, Aleema Faisal, Shaheer Hammad, Dayyan Ali Akhtar, Agha Ali Raza, Ihsan Ayyub Qazi
CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333
Summary
AI scribes that turn doctor-patient conversations into clinical notes could reduce paperwork and burnout, but only if the notes are faithful. We propose an evaluation framework that breaks each note into individual facts, compares them with clinician-written notes, and then checks any extra content against the transcript. Across eight LLM-based note generators, omissions are the main problem (rates of 8.5% to 24.0%), while contradictions stay between 6.2% and 7.9%; 92% of content that looks added is in fact supported by the conversation. When speech recognition errors are introduced, repetition in the conversation often rescues the note, but negation flips and sound-alike medical terms are more likely to slip through.
Cite this paper
Ehsan Latif, Aleema Faisal, Shaheer Hammad, Dayyan Ali Akhtar, Agha Ali Raza, Ihsan Ayyub Qazi. Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation. CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333.
@inproceedings{latif2026evaluating,
author = {Ehsan Latif and Aleema Faisal and Shaheer Hammad and Dayyan Ali Akhtar and Agha Ali Raza and Ihsan Ayyub Qazi},
title = {{Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation}},
booktitle = {CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333},
year = {2026},
url = {https://proceedings.mlr.press/v333/latif26a.html}
}
- IRTIQA: Evolving in English, Evaluated in Urdu — Does Agent Self-Evolution Transfer Across Languages?Findings of EACL 2027, Athens, Greece
- UrduCultureQA: A Culturally Grounded Benchmark for Urdu Question Answering and Alignment EvaluationFindings of EACL 2027, Athens, Greece
- Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu BenchmarkEMNLP 2026 (Main), Budapest, Hungary
- Kahaani: A Multimodal Co-Creative Storytelling SystemEACL 2026 Student Research Workshop, Rabat, Morocco
- Large Language Models Show Dunning-Kruger-Like Effects in Multilingual Fact-CheckingScientific Reports 16, 7594 (2026), a Nature Portfolio journal