O1016: Chat-GPT and Its Potential to Assist Pediatric Emergency Triage

Background: Rapid and efficient triage in Pediatric Emergency Departments is vital, yet often variable. This study evaluates GPT-3, an advanced AI large language model, in generating Emergency Severity Index (ESI) scores. By comparing AI-derived scores against nursing-assigned scores, this research attempts to determine the potential triage assistance a large language model like GPT-3 can provide.

Methods: This retrospective chart review was conducted in the Pediatric Emergency Department of a level one trauma center, from March through September 2023. It included 300 patients aged 18 and under. Patients within pre-selected shift blocks were included in the study. Patients who received an ESI triage score of 1, were excluded from the study. No chief complaints were excluded. A dual scoring system was employed where each patient received a nursing-assigned ESI triage score as part of their normal hospital triage and an AI-generated ESI triage score, at a later date, from GPT-3. Both were naturally blinded from each other's assessments. Chat GPT was only provided the patients, age, gender, triage vitals, and triage note (edited for the omission of patient identifiers, acronyms, and spelling.) It was then instructed to provide an ESI triage score, between 1-5. Data analysis focused on the concordance and statistical significant between any differences between these two sets of scores. This study was deemed exempt by the IRB.

Results: 300 patients were scored by nurses, as part of routine triage, and by Chat-GPT. 133 were less than 5 years old. 158 patients were male. 142 patients were female. Scores ranges from 2 to 4 as nursing scores of 1 were excluded and no patients in this set were given a score of 5. Chat GPT and nursing scores were identical 181 times. Nursing scored the patient a higher number (lower severity) 105 times, compared to GPT doing so 14 times. 26% of all patients were admitted to the hospital. Both Nursing and GPT scores correlated significantly with admission (p < 0.01). The average nursing ESI score was 3.22 (95% CI: 3.14 to 3.30). The average Chat-GPT triage score was 2.91 (95% CI: 2.81 to 3.00). Of the patients where GPT's score was a lower number (higher severity) than the nursing score, 37.5% of those patients were admitted.

Conclusion: Compared to nursing-assigned ESI scores, Chat-GPT assigned lower ESI triage scores (higher severity) on average. This suggests that while GPT-3 may help generate ESI triage scores, such scores tend to be more conservative. However, given a large portion of these patients in our study were admitted, there may be a role for GPT-3, or a future iteration, to enhance triage precision. Further studies should explore the scalability of AI applications, including those specifically trained for this task, to confirm these findings.