MobiHealthNews.com (05/26/23) Hagen, Jessica

OpenAI's ChatGPT-3 and ChatGPT-4 large language models failed the 2021 and 2022 American College of Gastroenterology Self-Assessment Tests, according to a new study in the American Journal of Gastroenterology. The tests each have 300 multiple-choice questions. Overall, ChatGPT-3 and ChatGPT-4 answered 455 questions, with the chatbots correctly answering 296 and 284 questions, respectively. Passing requires a minimum score of 70%, but ChatGPT-3 only scored 65.1% and ChatGPT-4 only scored 62.4%. The researchers suggested ChatGPT failed from a lack of access to paid medical journals or outdated information in its system, and more research is necessary before it can be employed reliably. "Based on our research, ChatGPT should not be used for medical education in gastroenterology at this time and has a ways to go before it should be implemented into the healthcare field," advised Arvind Trindade, associate professor at the Feinstein Institutes' Institute of Health System Science and senior author on the paper.

Read More