A recent study has revealed that the AI chatbot ChatGPT, which is powered by the language model GPT-3.5, is highly inaccurate in making pediatric diagnoses. The researchers examined 100 pediatric cases and found that ChatGPT failed to correctly diagnose 83% of them. The study, published in the journal JAMA Pediatrics, is the first to assess ChatGPT’s ability in diagnosing pediatric cases.
In previous research, GPT-4, a newer language model, correctly diagnosed only 39% of challenging medical cases, including those involving both adults and children. This shows that there is still a long way to go in developing accurate AI chatbots for medical diagnoses.
During the study, the researchers asked ChatGPT to provide a differential diagnosis and a final diagnosis for each case challenge. However, ChatGPT provided incorrect diagnoses for 72 out of the 100 cases. In some instances, the chatbot’s responses were clinically related but too broad to be considered a correct diagnosis.
One of the cases highlighted in the study involved a teenager with autism who displayed symptoms of a rash and joint stiffness. While the initial physician diagnosed the teen with scurvy, ChatGPT diagnosed the teenager with immune thrombocytopenic purpura, a disorder that affects blood clotting. Another case involved an infant with a draining abscess on the side of their neck. While the original physician attributed it to Branchiootorenal (BOR) syndrome, ChatGPT claimed the infant had a branchial cleft cyst.
However, there were a few cases where ChatGPT reached the same diagnosis as the doctors. For example, in a case of a 15-year-old girl with idiopathic intracranial hypertension (IIH), ChatGPT correctly matched the physician’s original diagnosis of Addison’s disease.
The researchers acknowledged that large language models like ChatGPT still have value as an administrative tool for physicians, such as in note-taking. However, the study highlights the limitations of AI chatbots in making accurate diagnoses, particularly in recognizing the relationships between medical disorders.
Furthermore, the researchers emphasized the need for more selective training and improved access to real-time medical information to enhance the diagnostic accuracy of AI chatbots in the future.
In conclusion, while AI chatbots have potential in the healthcare industry, further research and development are necessary to ensure their reliability and accuracy in diagnosing pediatric cases.
The source of the article is from the blog maestropasta.cz