Google's new LLM doctor is right way more often than a real doctor

The LLM's differential diagnosis list had the correct diagnosis 59% of the time, vs. 34% for human doctors.

"Specialist-rated top-k diagnostic accuracy. AMIE and PCPs top-k differential diagnosis (DDx) accuracy are compared across 149 scenarios with respect to the ground truth diagnosis (a) and all diagnoses listed within the accepted differential diagnoses (b)." From the project site.

Physicians face an immense challenge when evaluating patients with confusing constellations of symptoms and clinical findings. They must mentally generate a list of possible diagnoses that could explain the patient's presentation. This list, known as a differential diagnosis, provides a roadmap to guide further testing and treatment. But arriving at an accurate differential diagnosis list can be extraordinarily difficult, even for the most experienced doctors when dealing with complex, atypical cases.

Now, researchers at Google have developed a promising new AI system (paper here) that could aid physicians in this difficult task. The system is based on conversational large language models - a type of AI algorithm that has recently shown immense progress by being trained on massive textual datasets.