It Seems I've Struck a Nerve: Clinical AI Has a Measurement Problem and It Is Mostly Borne of Ignorance

The post

On October 6 I put up the meme above with this caption: "The idea that no one understands how to evaluate Clinical AI technology the right way is largely false. A new technology does not undo decades of improvements in ML/CS evaluation methods, nor centuries of statistics. Personal ignorance does not entail universal ignorance."

For the record, I've been all three guys. I started on the left, learned quick that LLMs do in fact go brrr, spent about two months in the middle, and then worked my way right.

My claim is narrow: the methods for evaluating clinical AI exist, applying them is often hard, and "nobody knows how" is false. I am not claiming eval is solved (I'll repeat that a couple more times, bc a few people read it that way).

The charitable reading

The charitable version of "nobody knows how to evaluate this" is "applying stats and research methods to these systems is hard." I agree with that completely. Per use case, the applied answer is a known unknown. You know the questions (what you're measuring, what the reference is, how reliable your raters are, how much the output moves from run to run), and you don't know the answers until you do the work, which is slow, expensive, and sometimes needs a new instrument.

There are also real open problems: whether an LLM grader validly measures anything (especially given the paucity of clinical outcomes data to validate it against, though we still need to know how the LLM performs factually and stylistically in the meantime), evals for agents acting over long horizons, and doing any of this cheaply at scale. Very smart people work on these. No dispute there.

Why "nobody knows" is still wrong

"Nobody knows how to evaluate X" is a universal claim, and prima facie a wild one to defend. In my experience the evidence behind it is usually that the speaker doesn't know the methods, and personal ignorance does not entail universal ignorance. (I mean ignorance literally, as in not knowing a thing. I didn't know most of this a few years ago either.)

To be fair to the claim, as a description of the people who actually sign off on these tools, it's largely true. In my experience most health system AI governance leaders were never trained to evaluate ML systems, and that is not their fault. Nobody trained them for this, the tools landed on their desks fast, and they're being asked to judge evidence they were never taught to read, a lot of it written by the vendor.

Further upstream it's a different story. In my experience most of the people posting about this on LinkedIn and publishing in the space have no formal ML or AI background, or any technical background, and I think a lot of the gap comes from that. I don't mind autodidacts at all. I learned more about LLM evaluation on the job than I did in my computational biology masters, though the masters gave me a good foundation in statistics, which is most of what this needs. None of this is meant to gatekeep. It's meant to encourage folks to learn what they actually need to know to do right by patients and the world, whatever the optics (and I get it, companies have reps to uphold and academics have to publish or perish).

Here's the argument, in a semi-syllogism style:

• Premise 1: Evaluating any system means measuring a property of it against a reference whose error you've characterized, and quantifying the uncertainty of that measurement.
• Premise 2: Existing fields have methods for every piece of that: psychometrics for building and validating new instruments [1], statistics for rater agreement and for splitting variance across raters, cases, and occasions [2], clinical epidemiology for reference standards or gold labels, adjudication, and outcomes, and ML evaluation for held out data, leakage, distribution shift, and repeated sampling.
• Premise 3: LLM based clinical systems produce new objects to measure: free text, multistep actions, and outputs that change from run to run. A valid measurement of a new object has the same requirements as any other measurement.
• Conclusion: The methods for evaluating clinical AI exist, even where applying them is hard, expensive, or still being worked out, so "nobody understands how to do this right" is false.

The corollary is the part I actually care about. Treat the field as methodless and you get ad hoc evals that fail checks the existing methods would have caught. Two examples, one in each direction.

Example 1: a new output type gets a new instrument

For years generated text was scored with ROUGE and BLEU [3][4], which reward word overlap with a reference and say basically nothing about whether the content is factually right. In clinical summarization, Van Veen et al. found NLP metrics like these correlated with physician reader scores at around 0.2 [5].

Then came FActScore [6]. You break the output into atomic facts, check each one against a knowledge source, and report the fraction that's supported. The authors validated their automated scorer against human annotators, and its estimates came within 2% of the human scores. Extending the paper, you can also check the knowledge source for facts the output omitted. It's surprisingly simple but extremely effective at reducing uncertainty, bc it turns an LLM output into a set of singleton claims, each one a proposition. Propositions are either true or false, so an LLM judge (or a human) now has a much more discrete and granular measuring tool. This paper kind of evangelized me ngl.

A commenter raised a fair tension: you can't call FActScore a new method and also say it's the same old techniques. I think you can. FActScore is a new instrument built from old parts:

• decompose the output into atomic claims (borrowed from philosophy and logic)
• measure precision over those claims
• validate the automated scorer against careful humans before trusting it

Inventing instruments is part of the methods base, and psychometrics has explicit rules for doing it. New tech changes what you point the methods at, and the methods carry over.

To note: FActScore itself measures precision only, and its authors say so [6]. The omission check is the extension, and in a clinical note it matters a lot, since an omission can hurt a patient as much as a hallucination. Another commenter proposed matching atomic facts between the transcript and the note to catch both. That's obviously intuitive, bc it's what I've been doing for a while now and others are catching on. I recently interviewed someone who said they like to measure this at the most atomic level, and then they couldn't answer a basic question about IRR. That's not meant as an insult to them. It shows how intuitive this is: someone pretty junior in the technical space figured out on their own that sequential data needs to be atomized. (IRR is the part you have to go learn.) DocLens [7] does a version of this for medical text: claim recall for completeness, claim precision for conciseness, and attribution back to the source. Same old parts, pointed at a new question.

FActScore is just one way to get a reliable signal, something you can check performance against and, if improvement is the goal, optimize against. Whatever method you pick, the signal has to be reliable first.

Example 2: an old instrument fails, and an old method catches it

PDQI-9 is the standard instrument for rating clinical note quality, and it's become a common way to grade AI scribe notes. Stetson et al. validated it in 2012 on nine human written notes, with an interrater ICC of 0.83 [8].

At Suki we ran PDQI-9 on 84 paired notes from two production candidate LLMs across internal medicine, oncology, pediatrics, and orthopedics [9][10][11]. Reviewers agreed no more often than they would have by flipping a coin, and on made up or incorrect facts they didn't agree at all. The two models could trade places in the rankings on reviewer opinion alone. (I left Suki in September, and all of this is public.)

Interrater reliability is what caught it, about as old school as a method gets. When the object you're measuring changes (human notes to LLM notes), you recheck reliability before you trust the scores. The rebuild runs on old methods too. Croxford et al. adapted the instrument for LLM summaries as PDSQI-9, using a semi Delphi process for content validity, factor analysis, and discriminant validity testing, and got an ICC of 0.867 across seven physician raters and 779 summaries [12]. That's what inventing an instrument the right way looks like.

Spilling the beans

I've said versions of this in a bunch of posts over the years, so here it is in one place, for free. If you're paying someone to do evals for you, at least now you can check their work lol.

1. Atomized, fact based evals beat holistic, rubric based evals.

A holistic rating (a 1 to 5 on "accuracy," a Likert per dimension, a pass or fail on the whole note) compresses a bunch of distinct errors into one number. One fabricated dose in an otherwise great note barely moves an average, and that's the error that hurts someone. Raters also disagree on holistic scores bc each of them is weighting different things in their head, which is roughly what happened with PDQI-9 above.

So atomize. Break the source and the output into claims and ask small questions: is this claim in the note supported by the transcript, yes or no? Is this claim from the transcript missing from the note? In my experience small questions get much better agreement, and every error gets a type and a denominator:

• unsupported claims (hallucinations)
• omitted claims
• misplaced or misattributed claims (right fact, wrong section, wrong date, wrong person)
• contradicted claims

Each type gets a rate with a confidence interval, and each individual error gets its own severity grade.

2. Generalizable, sequential ML pipelines beat frameworks that don't scale.

The framework that doesn't scale goes like this: someone writes a checklist per condition (a chest pain note needs onset, character, radiation, associated symptoms, etc.) or a rubric per case, then repeats that for every condition, specialty, and case. Cost grows with the number of conditions, coverage ends wherever the authoring ended, and the checklists rot when guidelines change. For a sense of scale, HealthBench had 262 physicians write 48,562 rubric criteria for 5,000 conversations [13], and a recent case specific rubric paper had 20 clinicians write 1,646 rubrics for 823 cases [14]. For a fixed benchmark, that's a defensible trade. For a product that sees whatever patient walks in the door tomorrow, you can't hand write a rubric for every patient.

The generalizable version is a sequence of ML steps, each one validated against humans once:

1. Decompose the source (transcript, chart) and the output (note, answer, plan) into atomic claims.
2. Normalize the claims to clinical concepts (SNOMED, RxNorm, LOINC, etc.) so "metformin 500 BID" and "metformin 500 mg twice daily" count as the same fact.
3. Align output claims to source claims with an entailment check.
4. Label whatever doesn't line up using the error types above.
5. Grade the severity of each error.
6. Validate each step against human annotation (extractor precision and recall, judge agreement with clinicians), then report error rates with confidence intervals and compare versions with paired tests.

Nothing in that pipeline needs to know what condition the patient has, so it runs on any note in any specialty, and when it breaks you can see which step broke. VeriFact does a version of this against the EHR and reported up to 92.7% agreement with an adjudicated clinician reference, where clinicians agreed with each other at most 88.5% of the time [15]. And yes, it's six steps. Every one of them gets validated, which is what separates it from the middle guy's framework.

Clinicians still matter a ton here. Their time goes to the parts only clinicians can do: defining the error types and the severity scale, labeling the validation set, adjudicating disagreements, and reviewing the high severity flags. Per case rubrics still have a place too, as a targeted test set for known failure modes and the rare, atypical cases. They just can't be the whole eval.

The objections

"What's the right answer when the output is free text?" In measurement, a right answer means a reference standard, and every reference standard has error. You can build an expert reference, adjudicate when the experts disagree, and report their agreement so you know the ceiling. When there's no gold standard at all, there's the imperfect reference standard literature, going back at least to Hui and Walter in 1980 [16], with latent class models and composite references reviewed by Reitsma et al. [17]. Or you use patient outcomes as the reference. And yes, statistics only hold under assumptions. Obviously. Knowing the methods includes knowing how to check them.

"Agents are nondeterministic, multistep, and long horizon." This was the most liked dissent (quoted in full at the bottom, since it's the comment that sparked this post). We already evaluate decision makers with all three properties: clinicians. A resident gives different answers to the same case on different days, works a patient up over many steps, and makes serial decisions across a whole admission. We evaluate them with OSCEs [18], board exams, peer review, and outcomes research, and complex multistep interventions have the MRC framework [19].

Nondeterminism is a test retest reliability problem. Sample repeatedly, report the variance, and use generalizability theory if you want runs, cases, and graders in one model. The ML side got there too: tau-bench's pass^k asks whether an agent succeeds on all k attempts at the same task, and gpt-4o came in under 25% at pass^8 in their retail domain [20]. Some of that variability is aleatoric, so you characterize it, and the epistemic part you can reduce. A commenter who's an IM resident and a software engineer made the software version of this point: we've written software for nondeterministic, unpredictable humans for decades, and least privilege and input validation still apply. If some agent property breaks the existing methods in principle, I'd still love an example :)

"It's still hard and expensive." Yes. One commenter pointed out that getting two radiologists to agree on the reference read usually takes longer than evaluating the model, and nobody budgets time for it. Another said the gap is the industry's willingness to enforce these methods before deployment. Agreed on both. Cost, time, and willingness are real constraints, and each one is a separate claim from "nobody knows how."

What doing it right looks like

• Define the construct: what property you're measuring and why it matters clinically.
• Pick an existing instrument or build one, and if you build one, follow the rules [1].
• Check reliability before you trust any scores. If raters can't agree, look at the instrument and the rater training before you blame the labelers.
• Define the reference standard, its error, and how you adjudicate.
• Measure run to run variability. One sample per case tells you very little about a nondeterministic system.
• Test on the cases you'll see in deployment, including the rare and atypical ones. Spectrum bias has had a name since 1978 [21].
• Tie results to patient outcomes where you can.
• Report per TRIPOD+AI, DECIDE-AI, or CONSORT-AI, depending on the stage [22][23][24].

Someone said the moral of the story is don't be Sloth from the Goonies. I say be the Sloth. The left guy and the right guy say the same thing, the right guy just knows why. Don't be the middle guy building a wildly complex (and usually unvalidated) eval framework when simple, tried and true methods work. That also doesn't mean throw ROUGE at it and pray.

Struck a nerve, again (a parable)

Timestamps first, bc they matter here. The meme went up October 6. In that thread, the most liked dissent came from the founder of an eval startup whose value prop is simulation environments built on tasks and rubrics written by in house clinicians. Here's the comment, verbatim, since it's what sparked this post:

"You're heavily oversimplifying the challenge of evals for autonomous AI systems that are non-deterministic, have multi-step processes, and/or take serial decisions over long horizons.

Evaluations for LLMs are an incredibly challenging problem and many of the smartest people that I know in the world are working on them. Perhaps for a deterministic rule-based harness that limits to minimal outputs, evals are easier, but then you're not actually using or evaluating the models' capabilities."

I started this draft October 8, with the spilling the beans section already on my list bc of that value prop. Then on October 20 something happened that convinced me to actually post it. In what follows, names and identifying details are removed to protect everyone, including me.

Once upon a time (October 20), a professional acquaintance got on a call with me about a critique I'd posted of a paper from a well known academic group. He was cordial. He asked me for a favor, to be less adversarial, and said he was worried it might affect me in the future. He also mentioned that the group is a good ally to have when critics come around.

That's an interesting thing to hear, so I asked him to explicate how it could affect me. The hypothetical was that two years from now, if I wanted to work with that group and had spent two years posting bad things about them, I wouldn't be able to. A bit of a non sequitur, bc it entails I'd want to work at a place I've spent two years critiquing. It also assumes academia is a door I'm trying to keep open. I already walked away from a real shot at it once. I quit my UCSF clinical informatics fellowship after getting into the CRISP program, with a postdoc on the table with Atul Butte and Vivek Rudrapatna, two people I deeply respected for their work in the space. The goal was a K to R track. I decided to go into industry instead bc that's where the frontier was moving, and there weren't enough clinicians who could do damage with NLP in the space at the time.

What I told him:

• If my ethics ever conflict with someone's goals, I'd want to be told that directly.
• Adversarial takes are necessary for growth, especially for the people and things you care about, and they shouldn't be taken personally.
• At a previous company I posted several critical takes about a frontier lab that directly funded us and that I worked with closely (I still do). No one at either place ever said anything negative to me about good faith critique.
• If someone critiques work I'm part of, I'd only want that group's support if the critique was wrong or defamatory. Support for any other reason is definitionally cronyism.

The moral: if the goal is to democratize medicine, meaning good care for people who can't get it now, the evidence behind these systems has to hold up, and critique is how evidence holds up. Science is adversarial by design (peer review is literally a structured argument), and dialectical truth seeking stops working when one side is asked to go quiet for optics. None of this is personal or meant to inflame. I critique that group's work bc I think it matters, and I'd want the same done to mine.

My goal is truth seeking, and sometimes that means upping the rhetoric a bit to get attention on a space that badly needs it, especially while we're plowing full steam ahead toward autonomous care. I don't have a fancy degree pedigree from a top 10 academic institution. I went to state schools that were affordable and local bc that's what was pragmatic for me, then a community residency, then one year at a top 3 research academic center before I left for industry. As above, I left a real shot at academics because I deeply care about these problems. I don't have friends at the big publishers or in the press. I have my blog.

Eval is not solved (third time, for the folks in the back). The methods exist, and checking your interrater reliability takes an afternoon. Go compute a kappa :)

Sources

[1] COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments). Mokkink LB et al., "The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes," J Clin Epidemiol, 2010. https://doi.org/10.1016/j.jclinepi.2010.02.006 (guides and checklists: https://www.cosmin.nl/)

[2] Bloch R, Norman G, "Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68," Medical Teacher, 2012. https://doi.org/10.3109/0142159X.2012.703791

[3] Lin CY, "ROUGE: A Package for Automatic Evaluation of Summaries," Text Summarization Branches Out (ACL workshop), 2004. https://aclanthology.org/W04-1013/

[4] Papineni K et al., "Bleu: a Method for Automatic Evaluation of Machine Translation," ACL, 2002. https://aclanthology.org/P02-1040/

[5] Van Veen D et al., "Adapted large language models can outperform medical experts in clinical text summarization," Nature Medicine, 2024. https://doi.org/10.1038/s41591-024-02855-5

[6] Min S et al., "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation," EMNLP, 2023. https://arxiv.org/abs/2305.14251

[7] Xie Y et al., "DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generation," ACL, 2024. https://aclanthology.org/2024.acl-long.39/

[8] Stetson PD et al., "Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)," Applied Clinical Informatics, 2012. https://doi.org/10.4338/ACI-2011-11-RA-0070

[9] Swanson K, Gupta N, "One score, nine dimensions, zero signal: The problem with holistic Likert ratings," Suki whitepaper, 2026. https://www.suki.ai/whitepapers/one-score-nine-dimensions-zero-signal-the-problem-with-holistic-likert-ratings/

[10] Fierce Healthcare (Heather Landi), "As AI scribe adoption grows, researchers at Suki challenge the industry's quality playbook," Jul 27, 2026. https://www.fiercehealthcare.com/ai-and-machine-learning/ai-scribe-adoption-grows-researchers-suki-challenge-industrys-quality

[11] Suki, "The broken ruler: Why the healthcare industry is measuring AI notes the wrong way," Aug 19, 2026. https://www.suki.ai/blog/the-broken-ruler-why-the-healthcare-industry-is-measuring-ai-notes-the-wrong-way-and-what-they-should-be-doing-instead/

[12] Croxford E et al., "Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models," arXiv, 2025. https://arxiv.org/abs/2501.08977

[13] Arora RK et al., "HealthBench: Evaluating Large Language Models Towards Improved Human Health," arXiv, 2025. https://arxiv.org/abs/2505.08775

[14] Shah A et al., "Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters," arXiv, 2026. https://arxiv.org/abs/2604.24710

[15] Chung P et al., "VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records," arXiv, 2025. https://arxiv.org/abs/2501.16672

[16] Hui SL, Walter SD, "Estimating the error rates of diagnostic tests," Biometrics, 1980. https://doi.org/10.2307/2530508

[17] Reitsma JB et al., "A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard," J Clin Epidemiol, 2009. https://doi.org/10.1016/j.jclinepi.2009.02.005

[18] Harden RM et al., "Assessment of clinical competence using objective structured examination," BMJ, 1975. https://doi.org/10.1136/bmj.1.5955.447

[19] Skivington K et al., "A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance," BMJ, 2021. https://doi.org/10.1136/bmj.n2061

[20] Yao S et al., "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains," arXiv 2024 (ICLR 2025). https://arxiv.org/abs/2406.12045

[21] Ransohoff DF, Feinstein AR, "Problems of spectrum and bias in evaluating the efficacy of diagnostic tests," NEJM, 1978. https://doi.org/10.1056/NEJM197810262991705

[22] Collins GS et al., "TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods," BMJ, 2024. https://doi.org/10.1136/bmj-2023-078378

[23] Vasey B et al., "Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI," Nature Medicine, 2022. https://doi.org/10.1038/s41591-022-01772-9

[24] Liu X et al., "Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension," Nature Medicine, 2020. https://doi.org/10.1038/s41591-020-1034-x