While diagnostic breakthroughs are coming thick and fast, a new study from the Technical University of Munich (TUM) has found that AI chatbots, similar to ChatGPT, are not yet suitable for diagnosing patients or creating treatment plans in hospitals.
What did the researchers do?
The research team tested AI programmes using real patient data from a US hospital emergency room. They focused on 2,400 cases where patients came in with abdominal pain.
What did they find?
The study revealed several concerning issues with the AI chatbots. The programmes often made incorrect diagnoses, failing to provide accurate assessments of patients' conditions. Additionally, they didn't always request all necessary tests, potentially missing crucial information for proper diagnosis. In some instances, the chatbots suggested unsafe treatments that could have put patients at risk. Perhaps most alarmingly, their diagnoses became less accurate as they received more information about each case.
How did AI compare to human doctors?
In a direct comparison between AI and human doctors, the results were clear. Human doctors were correct in their diagnoses 89% of the time. In contrast, even the best AI system managed to be correct only 73% of the time, showing a significant gap in diagnostic accuracy.
What does this mean for patients?
For patients, these findings indicate that AI chatbots are not currently safe or reliable enough to be used for diagnosing illnesses or planning treatments in hospitals. Implementing these systems in their current state could potentially put patients at risk of misdiagnosis or inappropriate treatment.
Is there any good news?
While the current AI systems aren't ready for clinical use, researchers remain optimistic about the technology's potential. They believe AI might eventually become a useful tool to help doctors discuss cases and consider different perspectives. However, it's important to note that AI is not intended to replace medical professionals but rather to augment their capabilities.
What's next?
To facilitate further research and improvement in this field, the TUM team has made their testing method available to other researchers. This shared resource will help evaluate future AI systems as the technology continues to develop and improve.
Remember: Always consult with qualified healthcare professionals for medical advice and diagnoses. AI technology in healthcare is still developing and should not replace expert medical opinion.
Diagnoses that are missed altogether, wrong, or should have been made much earlier
Full medical glossary