A new review suggests otherwise
Artwork by Dave Coverly at Speedbump.com. Reprinted by perrmission.
Cardiologist/author Eric Topol tends to be more bullish than I am about AI in medicine, but in his latest post he concludes that at least thus far there is “very little evidence for LLMs benefiting patients or doctors for health outcomes” (aside from helping with administrative work and the like).
As Topol notes, his review was partly inspired by thisrecent editorial in Nature Medicine, which points in the same direction:
You can read Topol’s detailed review here.
All of this fits with my recent Please don’t trust your chatbot for medical advice, which focused more on unsupervised use of LLMs by patients, rather than overall clinical outcomes, but in many ways converges to a similar place.
AI will surely someday be a major boon for medicine, but current tools such as domain-general chatbots may not be up to the job.
Discussion about this post
The Topol finding is interesting precisely because hes more bullish than you and still arrived at the same conclusion. When the optimist and the sceptic converge on "not yet," thats a stronger signal than either position alone.
The administrative work exception is the tell though. LLMs are genuinely good at summarising notes, drafting letters, handling paperwork — tasks where being wrong has low consequences and being fast has high value. The moment you move into clinical decision-making where being wrong means someone gets hurt, the accuracy threshold jumps from "good enough" to "better than the doctor" and thats a completely different bar.
The gap isnt really about whether LLMs can pass medical exams. They can. Its about whether they fail gracefully. A doctor who isnt sure orders more tests. An LLM that isnt sure sounds exactly as confident as one that is. And in medicine, the difference between uncertainty expressed and uncertainty hidden is sometimes the difference between a patient who lives and one who doesnt.
The amount of money wasted by AI in the past two years could have been better utilized to improve healthcare, leading to better outcomes.