O2093: Does AI Promote the Dunning-Kruger Effect? Evaluating ChatGPT Answers to Frequently Asked Questions about Adolescent Idiopathic Scoliosis

Background: Patients have long turned to the internet for answers to common medical questions. As the ability to access information evolves beyond conventional search engines, patients with adolescent idiopathic scoliosis (AIS) and their parents may use artificial intelligence chatbots such as ChatGPT as a new source of information. Within the orthopaedic literature, there has already been interest in the information ChatGPT provides about common orthopaedic conditions. In learners, one frequently described effect is that individuals don’t recognize or understand that which they do not know. This has been described as the Dunning-Kruger effect. We hypothesized that ChatGPT would provide generally accurate answers to generalized questions about AIS, but would not provide accurate answers to more in depth questions.

Methods: Ten frequently asked questions regarding AIS were posed to ChatGPT. The accuracy and adequacy of the responses were graded as excellent not requiring clarification, satisfactory requiring minimal clarification, satisfactory requiring moderate clarification, and unsatisfactory requiring substantial clarification. We defined results as follows: minimal: 1-2 items omitted or requiring clarification; moderate: 3-4 items omitted or requiring clarification, and unsatisfactory: greater than 3 items omitted or requiring clarification, or responses with significant false information.

Results: The weighted Cohen’s kappa was 0.722, indicating substantial agreement between the authors. ChatGPT gave one response that was excellent not requiring clarification, four responses that were satisfactory requiring minimal clarification, three responses that were satisfactory requiring moderate clarification, and two responses that were unsatisfactory requiring substantial clarification, with information about higher level, more complex areas of discussion such as surgical options being less accurate.

Conclusion: ChatGPT provides answers to FAQs about AIS that were generally accurate, although correction was needed on specific surgical treatments. Patients may be at risk for developing a Dunning-Kruger effect by proxy from the superficial and sometimes inaccurate information provided by ChatGPT on more complex aspects of AIS. Generative AI, as it compiles its “fund of knowledge,” displays a reasonable basic repository of correct information, but does not fully acknowledge limitations to this information for more complex, nuanced components of medical decision making, such as surgical management.