Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2026

Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation

Ehsan Latif, Aleema Faisal, Shaheer Hammad, Dayyan Ali Akhtar, Agha Ali Raza, Ihsan Ayyub Qazi

CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333

Summary

AI scribes that turn doctor-patient conversations into clinical notes could reduce paperwork and burnout, but only if the notes are faithful. We propose an evaluation framework that breaks each note into individual facts, compares them with clinician-written notes, and then checks any extra content against the transcript. Across eight LLM-based note generators, omissions are the main problem (rates of 8.5% to 24.0%), while contradictions stay between 6.2% and 7.9%; 92% of content that looks added is in fact supported by the conversation. When speech recognition errors are introduced, repetition in the conversation often rescues the note, but negation flips and sound-alike medical terms are more likely to slip through.

Cite this paper

Ehsan Latif, Aleema Faisal, Shaheer Hammad, Dayyan Ali Akhtar, Agha Ali Raza, Ihsan Ayyub Qazi. Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation. CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333.

@inproceedings{latif2026evaluating,
  author = {Ehsan Latif and Aleema Faisal and Shaheer Hammad and Dayyan Ali Akhtar and Agha Ali Raza and Ihsan Ayyub Qazi},
  title = {{Evaluating Robustness of LLM-Based Ambient Scribes for SOAP Note Generation}},
  booktitle = {CHIL 2026 (Conference on Health, Inference, and Learning), PMLR vol. 333},
  year = {2026},
  url = {https://proceedings.mlr.press/v333/latif26a.html}
}

More on this theme

All 18 works on this theme →