Say a doctor asks whether the chest pain ever travels down the left arm, and the patient, mid-swallow, answers “uh-uh.” The AI note-taker on the desk writes “uh-huh.” Those two answers sit one breath of air apart, and the chart now records a symptom the patient just denied.
Automatic speech recognition gets clinic talk wrong for language reasons first. It was built to guess the likeliest words from tidy audio, and an exam room hands it overlapping voices in every accent, plus a vocabulary where near-identical sounds name different things.
Two syllables can flip a symptom
Spoken English is full of tiny sounds that carry a whole answer. “Uh-huh” and “uh-uh” are separated by little more than an h and a change in pitch, while “mm-hm” can mean yes or can mean “keep going, I’m listening.” Linguists call that second kind a backchannel, and people produce them constantly while someone else is still talking. A human listener sorts them out without effort, reading the speaker’s face and remembering the question that came just before.
A speech recognizer can’t see any of that. It hears a short burst, often half buried under someone else’s voice, and has to commit to one written word. There’s no spelling for a shrug.
Overlap makes all of this harder. Patients start answering before the question ends, and a parent will finish a child’s sentence while the nurse is reading back a dose. Software that labels who’s speaking, a step engineers call speaker diarization, works best when one voice stops before the next begins. When two voices stack, the system may drop one of them or credit the patient’s words to the clinician. In a medical note, who said it is part of the meaning. “I stopped taking it” from the patient is history; from the doctor, it’s an instruction.
Whose English the system learned
In a 2020 Stanford study published in PNAS, five commercial speech recognizers from Amazon, Apple, Google, IBM and Microsoft misheard Black speakers far more often than white speakers, with average word error rates of 0.35 against 0.19. The researchers traced the gap to the acoustic models, the part of the system that maps raw sound to speech units, because the difference held even when both groups said identical phrases.
The study looked only at Black and white American speakers, but the same logic suggests the problem doesn’t stop at one dialect. A recognizer learns what English sounds like from the recordings it was fed, so any vowel system or stress pattern that was thin in that pile turns into errors later. If that holds, it’s reasonable to expect similar trouble for a lot of people in an American clinic, such as the patient who learned English as an adult or the physician who trained abroad and has spoken English all her life with a different set of vowels. Everyone has an accent, of course. The machine just has a narrower idea of normal than the people in the room do.
Bilingual patients add another layer. Someone describing pain may switch into Spanish for the part that matters most, “me duele aquí,” and then slide back into English. A system set up to transcribe English will try to force that phrase into English words, and what comes out can be nonsense. A human listener who hears the switch knows it’s the most important sentence in the visit.
Medicine has its own sound-alikes
Celebrex and Celexa sound close enough that the Institute for Safe Medication Practices keeps them on its list of confused drug names, and that list exists because people mix them up. Software faces the same trap with less context. When the audio is unclear, a recognizer tends to fall back on the word it has seen more often, which is one way “hypotension” could turn into “hypertension,” or fifteen milligrams into fifty, since those two numbers differ by little more than a final n and where the stress falls.
Clinicians speak in shorthand as well, saying “PRN” or “BID” out loud, and a general-purpose recognizer that has rarely heard those letters together may spell them as ordinary words that happen to sound close. The bigger worry is how clean these mistakes look. A garbled phrase gets caught by anyone reading, but a real word in correct grammar reads as though someone meant it.
Where a person still has to listen
Ask who signs the note. The clinician does, and the chart carries their name whether the words came from their own typing or from an ambient AI tool. So every practice that automates documentation ends up with a human checking the output somewhere, and the real decision is where that person sits.
Some clinicians review each machine draft themselves after the last patient leaves, which works until the evening backlog grows. Others bring in a remote medical scribe who listens to the visit as it happens and drafts the note while the details are fresh, so the person who heard “uh-uh” is the one who types it. Someone in that seat can also ask for a repeat while the patient is still in the chair, and that’s something no transcript can do after the fact.
What to read first
Whoever does the checking, a few spots in a machine-drafted note deserve attention before anything else, because they’re where language trips the software:
- Every yes or no answer to a symptom question, especially anything written as “uh-huh” or “mm-hm.”
- Numbers that sound alike out loud, like fifteen and fifty.
- Drug names and any word that starts with hypo- or hyper-.
- Lines credited to one speaker during stretches when two people talked at once.
A patient who says “uh-uh” with a small shake of the head has given a perfectly clear answer. The note should say so, and for now that still takes someone who heard it.
Lily-Ann Greene is a freelance writer with a background in corporate branding. She writes about language, technology and the places where one trips over the other.