Almanac - Insights and Applications for the Healthcare CPD Community
Powered by
Alliance for Continuing Education in the Health Professions
  • Education
  • Outcomes
  • Leadership
  • Podcasts
  • Industry News
Exploring the Potential and Limitations of Chat Generative Pre-trained Transformer (ChatGPT) in Generating Board-style Dermatology Questions: A Qualitative Analysis
Monday, September 11, 2023

Exploring the Potential and Limitations of Chat Generative Pre-trained Transformer (ChatGPT) in Generating Board-style Dermatology Questions: A Qualitative Analysis

By: Cureus

Cureus (08/18/2023) Ayub, Ibraheim; Hamann, Dathan; Hamann, Carsten R.; et al.

Researchers qualitatively analyzed the potential and constraints of OpenAI's Chat Generative Pre-trained Transformer (ChatGPT) language model as a study tool in dermatology. They tasked ChatGPT with formulating multiple-choice American Board of Dermatology Applied Exam (ABD-AE)-style questions extrapolated from eight continuing medical education articles published in the Journal of the American Academy of Dermatology. Two board-certified dermatologists assessed the questions in terms of accuracy, complexity and clarity, along with each question's suitability for the needed depth of knowledge for the ABD-AE and the clarity of its wording and structure. ChatGPT usage yielded 40 questions for the articles; however, reviewers rated 10 as being of low complexity, nine as vague or unclear and five as inaccurate. Just 16 questions generated using ChatGPT 3.5 were accurate and had an appropriate degree of complexity for trainees studying for ABD-AE. "This study identified the limited domain-specific knowledge of ChatGPT as a major limitation as dermatology requires a deep understanding of skin anatomy, physiology and pathology, which ChatGPT lacks," the authors wrote. Additional limitations include the model's failure to understand context and produce high-quality distractor options, and its lack of image-generating abilities. The researchers recommend that future investigation should concentrate "on developing domain-specific language models that possess deep knowledge of dermatology." They concluded that ChatGPT cannot substitute for dermatologists and medical educators when devising questions for evaluating candidates' knowledge and reasoning capabilities.

Read More

Keywords:   

Related Articles

Pharmacy School-Led Program Focuses on Rural Pharmacies
industry news
Pharmacy School-Led Program Focuses on Rural Pharmacies

By: University of North Carolina at Chapel Hill

Higher Scores on ABIM's Longitudinal Knowledge Assessment Associated with Reduced Post-Hospitalization Death and Readmissions
industry news
Higher Scores on ABIM's Longitudinal Knowledge Assessment Associated with Reduced Post-Hospitalization Death and Readmissions

By: ABIM News Release

ACCME Releases 2024 Data Report: Evolving Impact in a Shifting Landscape
industry news
ACCME Releases 2024 Data Report: Evolving Impact in a Shifting Landscape

By: ACCME News Release

Education of Healthcare Professionals to Improve Guideline Adherence in Atrial Fibrillation: The STEEER-AF Cluster-Randomized Clinical Trial
industry news
Education of Healthcare Professionals to Improve Guideline Adherence in Atrial Fibrillation: The STEEER-AF Cluster-Randomized Clinical Trial

By: Nature Medicine

Alliance for Continuing Education in the Health Professions
2001 K Street NW, 3rd Floor North, Washington, DC 2006
P: (202) 367-1151 | F: (202) 367-2151 | E: acehp@acehp.org
Contact Us | Privacy Policy | Disclaimer | About
© Alliance for Continuing Education in the Health Profession
Login
Search