Artificial intelligence tools, such as the iconic ChatGPT, are not good for making a diagnosis, a study published Monday suggests, as the medical interest of AI is being closely scrutinized around the world.
Published in Nature Medicine, this study, conducted with 1,300 people in the United Kingdom, shows that several AI models – ChatGPT, Llama (Meta) or Command R+ – do no better than a simple online search when a patient wonders about their symptoms.
"There is a lot of hype around AI, but it is simply not ready to replace a doctor," said Rebecca Payne, a researcher at the University of Oxford and co-author of the study, in a statement.
The study participants were not actually ill. They took part in a kind of role-playing exercise: the researchers distributed ten different sets of symptoms to them, for which there was unanimous agreement in the medical community regarding the diagnosis to be associated.
Only a third of the study participants received a correct diagnosis. This is no better than in a group that simply had to rely on a standard internet search.
However, some studies had shown that ChatGPT and other models can successfully pass medical tests: placed in the place of a medical student facing, for example, multiple choice questions, the AIs do well.
However, this new study shows that the situation changes when it comes to interacting with real people. These people may lack precision in describing their symptoms and may not provide all the essential information, the researchers suggest.
This study comes amid a great deal of questioning about the potential contribution of AI to medical information, given that access to a doctor is often lengthy in many countries, including developed ones.
In France, the High Authority for Health (HAS) is expected to issue a ruling soon on the benefits of directly using AI for patients. It had already concluded at the end of 2025 that these tools could be useful to healthcare professionals, provided they are used judiciously.
The study published on Monday has some limitations, such as its methodology based on fictional scenarios. Furthermore, the models used have since been largely replaced by more effective versions.
But "this is a very important study which highlights that chatbots pose real medical risks for the general public," David Shaw, a bioethics specialist at Maastricht University (Netherlands), told AFP.
English 
French