O2092: Artificial Intelligence Based Large Language Models Have the Potential to Facilitate Patient Education

Background: Artificial intelligence (AI) large language models (LLM) are increasingly popular. Patients and families are increasingly likely to use LLM when conducting internet-based research about scoliosis, so it is important to understand the abilities and limitations of this technology in disseminating accurate medical information. Here, we use an expert panel to compare LLM-generated and professional society-authored responses to frequently asked questions about pediatric scoliosis.

Methods: We used 3 publicly available LLMs to generate answers to 15 frequently asked questions (FAQs) regarding pediatric scoliosis. The FAQs were derived from the Pediatric Orthopedics Society of North America, the Scoliosis Research Society, the American Academy of Orthopaedic Surgeons, and the Pediatric Spine Foundation. We gave minimal training to the LLM other than specifying the response length and requesting answers at a 5th-grade reading level. A 15-question survey was distributed to an expert panel. Panel members were presented with an AI and a physician-generated response and asked to select which they preferred or neither. They were then asked to grade their agreement with their responses individually on a Likert scale.

Results: Panel members had a mean of 8.9 years of experience post fellowship (range: 3-23 years). Overall, the panel reported an equivalent agreement between AI-generated and physician-generated responses (Figure 1). For 40% of questions, the panel favored physician-written responses, and for 40%, it favored AI. For 13% of questions the panel felt responses were equally good and for 7% there was a tie between AI and "equally good". For two physician-generated responses and three AI-generated responses, the error bars of the expert panel mean score for accuracy and appropriateness fell below neutral, indicating a consensus near disagreement with the response.

Conclusion: Based on the expert panel review, AI delivered accurate and appropriate responses as frequently as physician-authored FAQ responses from professional Society websites. Both AI and professional Society websites were roughly equally likely to generate responses with which the expert panel disagreed.