Which AI is Best for Doctors? We Still Don't Know
In March I wrote that OpenEvidence, now used by roughly two-thirds of US doctors (up from about 40% when I wrote), and its specialized competitors — UpToDate Expert AI, DoxGPT, Glass Health, Vera Health, among others — had never all been independently tested against each other.
Three months later, two head-to-head studies of AI-for-medical-purposes arrived within weeks of each other. They reached opposite conclusions. One includes a comparison of OpenEvidence and UpToDate's AI, but neither answers the question every practicing physician has: which of these should I use?
Two studies, opposite conclusions
Last month, an NYU team led by Krithik Vishwanath and Eric Oermann published a comparison in Nature Medicine: OpenEvidence and UpToDate Expert AI against three general-purpose AI (GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6), tested on 500 MedQA licensing-exam questions, 500 HealthBench items, and 100 real physician queries, with twelve clinicians doing blinded review against fixed answer sheets. The general-purpose AI won every evaluation. Amazingly, the medical AI performed roughly as well as Google's AI Overview — the free feature above your search results.
Two weeks later, a UCSF-led team headed by Jean Feng found ... the opposite. Using 620 real point-of-care questions from the OpenEvidence platform, 149 practicing physicians across 36 states graded AI responses relevant to their specialties, with data collection run and paid for by OpenEvidence. The tested AIs: OpenEvidence, Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5.
And OpenEvidence both funded and helped to design the Feng study — making it unsurprising that OpenEvidence came out on top by a wide margin.
| NYU (Vishwanath / Oermann) | UCSF (Feng) | |
|---|---|---|
| Winner | General-purpose AI | OpenEvidence |
| Medical AIs tested | OpenEvidence, UpToDate Expert AI | OpenEvidence |
| General-purpose AI | GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6 | GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8 |
| Questions | 500 MedQA + 500 HealthBench + 100 physician queries | 620 point-of-care queries from OpenEvidence |
| Graders | 12, one institution, not specialty-matched | 149, 36 states, specialty-matched |
| Scoring | Fixed answer sheet, scored in isolation | Blinded, answers compared side by side |
| Independence | Fully independent | OpenEvidence co-designed, ran, and paid for collection |
| Weird bias | — | Longer answers scored higher on accuracy and completeness |
The table highlights the major flaws in both studies, including: the NYU study was small; the UCSF study was designed and paid for by OpenEvidence.
The question neither study fully answered
The NYU study, for all its limitations, does contain an independent medical-vs-medical comparison. OpenEvidence and UpToDate Expert AI were both included — a genuine head-to-head between two medical AIs, run by a team with no stake in either. Both scored near Google's AI Overview, and pretty close to each other. That is the first independent comparison of two of these products.
But: DoxGPT, Glass Health, and Vera Health were not included, and remain untested against anything. So we now have a comparison of UpToDate vs OpenEvidence that was close to a tie (and close to a tie with generic Google AI), and no evaluation of three other important medical AIs. That's slightly better than when I wrote in March, but still short of what a physician choosing among five would need.
Why the study you want doesn't exist
Short answer: the AI companies don't profit from the answer.
Pharmaceutical companies rarely test their drugs against competitors — head-to-head trials happen mainly when a company is confident it will win. Losing one costs market share, so the comparisons clinicians most need don't get run. The same logic governs clinical AI. OpenEvidence will collaborate on a study where OpenEvidence beats ChatGPT. Wolters Kluwer will fund one where UpToDate shines. Neither will fund OpenEvidence versus UpToDate versus DoxGPT versus Glass Health — unless they can have a strong hand in guiding the design, as in the Feng study.
The same incentive explains why outsiders can't run the comparison themselves: the medical AIs are enterprise products whose terms of service discourage benchmarking and offer no access at the scale a real study needs. A determined team can still test them by hand, as the NYU group did with two of them — but not across the number of tools or the volume of questions the full comparison requires. The general-purpose AI — ChatGPT, Gemini, Claude — have open APIs, so anyone can test them, which is why every study includes them.
Plus:
Expert participation is expensive. The UCSF study needed 149 specialty-matched physicians to generate 1,156 ratings, and the authors still concede that they don't have enough data to support conclusions within any single specialty. Getting enough ratings to answer that would be a million dollar activity, and NIH won't fund it.
The field is moving too fast. Between the NYU preprint and its Nature Medicine publication, the model lineup had to be updated — GPT-5 became GPT-5.2, Claude Sonnet 4.5 became Claude Opus 4.6. A rigorous evaluation takes months or years; by the time it's published, every system tested is obsolete.
What would a useful study look like
So what would a study look like that could actually help doctors choose which AI to use?
Neutral questions — not drawn from any competitor's platform — spanning diagnosis, triage, and management, not just point-of-care lookup.
All five medical AIs — and any newcomers — alongside the latest versions of the general-purpose AI.
Specialist panels large enough to grade each answer as acceptable or not, so questions with several right answers still get scored fairly.
A standing setup that reruns as models change, instead of a snapshot obsolete by publication.
The one missing ingredient is a funder with no dog in the fight — a medical society, a large payer, a health system consortium, anyone whose interest is the answer rather than a particular winner. Or an appropriate regulator that, like FDA, would dictate design and audit the data.
Every other part of medicine demands more. A new statin can't claim superiority over an old one without a head-to-head comparison. A new diagnostic test has to report its sensitivity and specificity against a reference standard. A clinical AI shaping millions of decisions a month? That just needs a good marketing team.
And until that changes, no study will answer the question doctors actually need: which AI is best for my practice?