In 2019, researchers publishing in the journal Science took apart a piece of software that hospitals and insurers across the United States were using to decide which patients needed extra care. It was widely deployed, well built, and performing exactly as designed.
The design was the problem.
The algorithm did not measure illness. It measured cost, on the assumption that sicker people generate higher medical bills. Reasonable on its face. But it meant the software was reading a spending record, not a health record.
The researchers found that Black patients assigned the same risk score as White patients were, in fact, sicker. Less money had historically been spent on their care at the same level of need, and the algorithm read that lower spending as lower need. The journal summarised the effect plainly: the bias cut the number of Black patients identified for extra care by more than half.
Why the mechanism matters beyond that one study
Worth being precise here. That study examined one algorithm and one population. It makes no finding about any other community, and I am not going to claim otherwise.
The mechanism, though, is not specific to who it was studied on. Any system that infers how sick someone is from what was previously spent on them, or from what was previously written about them, inherits whatever gaps already existed in that record. If a community has historically had less access to specialists, fewer diagnostic tests, shorter appointments or more turnover in who provides care, the record is thinner. Software reading that record concludes the people are healthier.
That is not the software making an error. It is answering the question it was given, accurately, using data that was already incomplete.
Where this shows up in practice
Medicare Advantage, the privately administered side of Medicare, pays insurers more to cover sicker patients. That design is deliberate and sound. Without it, insurers would have every reason to avoid people who are expensive to care for.
How sick a patient is gets established through diagnosis codes drawn from clinical notes. Which puts real weight on what a clinician actually writes down during a visit, and on whether anyone revisits it later.
Most of the software built around this was designed to work backwards. It reads charts after the fact, looking for conditions that should have been recorded and were not. That approach depends entirely on there being enough in the record to find. Where documentation is already thin, working backwards finds thin results, and the patient stays underrepresented in the data that determines what resources follow them.
The federal audits point the same way
The US Department of Health and Human Services Office of Inspector General has been auditing this. Across its work on diagnosis codes it considers high risk, it reports that approximately 70 percent were not supported by the associated medical records, with some categories exceeding 90 percent.
One national audit examined 97 records where an acute stroke had been submitted. None were supported as acute strokes by the underlying documentation. In 68 of those cases the patient had genuinely had a stroke, recorded in their history. The submitted code described it as currently happening.
Nothing in those findings was invented. They describe records that did not say clearly enough what was true, which is a different failure and a more common one.
The alternative is simpler than it sounds
If working backwards from an incomplete record produces incomplete answers, the other option is to get it right while the patient is in the room.
That means supporting the clinician during the visit: surfacing what the record already shows, flagging what needs confirming or ruling out, and prompting for the specificity that makes a note useful to whoever reads it next. The condition gets documented because a clinician observed and assessed it that day, not because an algorithm inferred it from spending patterns months later.
This approach is called prospective risk adjustment, and its advantage for underserved communities is structural rather than sentimental. A visit-based record does not inherit the gaps in the old one. It starts from what the clinician sees in front of them. For any population whose historical record is thinner than their actual health status, that difference decides whether the data reflects them accurately or keeps repeating what was missing before.
The 2019 study ends with a point worth carrying: the fix was not a better model. It was changing what the model was asked to predict. That is a design decision, and design decisions are made by people who can be asked to explain them.