Medscape (01/26/23) Ault, Alicia

Researchers at Ansible Health found that OpenAI's ChatGPT chatbot can pass parts of the U.S. Medical Licensing Examination (USMLE). The bot's algorithms are trained to forecast the probability of a given word sequence based  on the preceding words' context. The authors initially investigated its ability to aggregate all communications about housebound patients with chronic lung disease by asking it multiple choice questions. They presented a theoretical patient's signs and symptoms and asked ChatGPT to consider underlying causes or diagnoses, then refreshed and asked it to answer a sampling of USMLE questions. After another refresh the chatbot was asked to justify the correct and incorrect answers. ChatGPT's accuracy for the first step ranged from 43% — when "indeterminate" answers were added to the analysis — to 68%, when they were excluded. Its pass rate was 51%-58% in the second step and 56%-62% in the third step. Asking ChatGPT questions verbatim yielded accuracy of 36%-55% for Step 1, 57%-59% for Step 2,and 55%-61% for Step 3. Response justification produced 40%-62% accuracy for Step 1, 49%-51% for Step 2, and 60%-65% for Step 3. The results prompted the researchers to have ChatGPT co-author a preprint paper that has been posted on medRxiv. Ansible's Victor Tseng, MD, said ChatGPT composed the abstract and results sections "with minimal prompting and largely cosmetic adjustments from the human co-authors," in addition to significantly contributing to the introduction and methods sections. The findings are set to be published in a peer-reviewed journal soon.

Read More