Can AI code a hospital chart?
Not without a coder. The best model gets the main diagnosis right on most charts, but misses more than half of the other diagnoses.
of charts where the best model chose the same main diagnosis as a certified coder.
of the coder's other diagnoses that each model found.
charts that the best model coded with no corrections needed.
- 1 No corrections needed
- 45 Main diagnosis right, other codes needed fixing
- 9 Main diagnosis wrong
Why it matters. Codes decide what a hospital is paid. Revenue cycle firms are selling AI coding on the promise of fewer coders, so the gap between the pitch and the result is a diligence question.
Source: Caliper Lab medical coding benchmark, Edition 1, 19 Oct 2026.
Show the data
| Item | Value |
|---|---|
| Claude Opus 5.5 | 49 |
| Kimi K3 | 44 |
| GPT-6.1-sol | 43 |
What we did
Three frontier AI models coded 55 synthetic inpatient charts in infectious disease, endocrinology and gastroenterology. A certified coder's codes were the reference: 55 main diagnoses, 269 other diagnoses and 18 procedures.
Source: Caliper Lab medical coding benchmark, Edition 1, 19 Oct 2026.
Show the data
| Group | Value |
|---|---|
| All three models | 83% |
| Two of three | 55% |
| One model only | 31% |
A practical review rule follows: accept codes all three models agree on, and send the rest to a coder. Even then, about 1 in 6 agreed codes would still be wrong.
Full results
| Model | Coding score | Main diagnosis right | Other diagnoses found | Charts with no corrections | Cost per chart | Time per chart |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 49 | 84% | 43.9% | 1 of 55 | $0.35 | 65 s |
| Kimi K3 | 44 | 78% | 37.5% | 1 of 55 | $0.12 | 100 s |
| GPT-6.1-sol | 43 | 69% | 38.3% | 0 of 55 | $0.25 | 350 s |
How we measured it
- Each model received the full chart and the same instructions, with no examples and no retrieval.
- Coding score balances codes found against codes wrongly added (an F1 score), across all 342 codes the coder assigned, out of 100.
- Cost and time are per chart, at list API prices on the run date.
Limits
- One coder set the reference; codes were not adjudicated by a second coder.
- Charts are synthetic, written to match real inpatient cases.
- Grouping to a payment category (DRG) is not yet scored; Edition 2 adds it.
See everything we cover in healthcare.
AI in healthcare