The Role of Artificial Intelligence in Medical Diagnoses
Artificial intelligence (AI) is becoming increasingly important in the field of medicine, particularly for diagnosing health conditions. These tools can quickly and accurately identify problems in medical records, X-rays, and other forms of data that might not be immediately apparent to healthcare professionals. However, recent studies have raised some questions about the reliability of AI, especially as it ages.
Concerns About AI Performance Over Time
A study published at the end of 2024 has highlighted troubling findings regarding the cognitive performance of AI technologies, including advanced chatbots and large language models (LLMs). The authors of the study argue that similar to humans, these AI systems may experience cognitive decline as they "age," which could impact their ability to diagnose medical conditions effectively.
The Implications of Cognitive Decline in AI
The study emphasizes that these findings challenge the common belief that AI will soon replace human doctors. The researchers suggest that as AI models show signs of cognitive impairment, it may lead to doubts about their efficiency and reliability in a clinical setting. This could negatively affect the trust that patients place in AI technologies during medical consultations.
Testing the AI Systems
Researchers conducted tests on several publicly available LLM-powered chatbots, including:
- ChatGPT by OpenAI
- Sonnet by Anthropic
- Gemini by Alphabet
The tests used the Montreal Cognitive Assessment (MoCA), a standard set of tasks neurologists typically use to evaluate cognitive abilities such as attention, memory, language, and more.
What is the Montreal Cognitive Assessment?
MoCA includes various tasks designed to assess different cognitive functions. Some examples are:
- Drawing: Participants draw a clock showing a specific time.
- Subtraction: Starting from 100, participants repeatedly subtract 7.
- Word Recall: Listening to a list of words and then recalling as many as possible later.
In human subjects, a score of 26 out of 30 indicates no significant cognitive impairment.
Performance Results of the AI Systems
The AI systems performed variably across different cognitive tasks. While they excelled in tests involving naming, language, and basic attention skills, they struggled with visual/spatial tasks and executive function evaluations. Notably, several AI models performed worse than expected in areas like delayed memory recall.
The latest version of ChatGPT achieved the highest score of 26, while an older model, Gemini 1.0, only scored 16. This disparity suggests that older AI models may exhibit signs of cognitive decline.
Study Limitations
The authors of the study stress that the findings are observational, meaning they cannot directly compare AI systems to human cognitive function. However, they point out significant weaknesses in AI, particularly concerning tasks requiring visual abstraction and executive function tasks. This raises concerns about the safe and effective use of AI in clinical settings.
Future of AI in Healthcare
The implications of these findings suggest that the medical community may need to rethink how and where AI is implemented in healthcare. There is a possibility that as these technologies continue to evolve, they may need specialized oversight or even evaluation similar to that which human cognitive assessments undergo.
In a lighter vein, the study raises an amusing thought: perhaps neurologists will find themselves evolving into a new field, diagnosing AI systems that show signs of cognitive decline.
Conclusion
AI technology has a lot of potential in helping with medical diagnoses, but it is essential to address the limitations and risks identified in recent studies. As AI tools become an integral part of healthcare, ensuring their reliability and trustworthiness will be crucial for both healthcare providers and patients. The advancement of AI technologies should go hand in hand with rigorous evaluations to maintain their effectiveness and safety in clinical environments.
