P2.D068: Evaluation of AI Models in Predicting Pediatric Emergency Severity Index Levels

Background: Efficient and accurate triage in Emergency Departments (ED) is vital for timely and appropriate medical care, particularly for pediatric patients. The Emergency Severity Index (ESI) is a widely used tool used for triage. Despite its utility, the subjective nature of clinical judgments can lead to variability in triage accuracy. Recent advances in artificial intelligence (AI), especially generative AI models like ChatGPT, offer the potential to enhance triage processes. This study aims to evaluate the accuracy of AI models, specifically ChatGPT versions 3.5 and 4.0, T5, Llama-2, and Mistral Large in predicting ESI levels for pediatric ED patients and to assess the impact of medically oriented fine-tuning on these predictions.

Methods: A cross-sectional analysis was conducted using 70 pediatric vignettes from the Emergency Severity Index Handbook version 4 as the test standard. Each AI model was prompted to predict the ESI level for each vignette, with each model undergoing three repetitions for a total of 210 simulated triages. ChatGPT-4.0 and T5 models were then fine-tuned with reference criteria for each ESI level, specific keywords, and practice clinical vignettes. The models' performance was evaluated based on accuracy, undertriage, and overtriage rates.

Results: ChatGPT-4.0 exhibited the highest overall accuracy among untrained models (62.6%), with an undertriage rate of 14.3% and an overtriage rate of 23.3%. T5, Llama-2, and Mistral were less accurate in comparison (Figure 1). Fine-tuning significantly improved the performance of ChatGPT-4.0, raising its accuracy to 76.5%, undertriage rate to 8.7%, and overtriage rate to 14.8%. The accuracy of the trained T5 was 63.9% with an undertriage rate of 24.3% and an overtriage rate of 12.9%. ChatGPT models displayed enhanced performance at higher acuity, with the fine-tuned GPT-4.0 achieving 91.7% and 86.7% accuracy in predicting ESI 1 and 2 respectively (Figure 2). T5 performed poorly at higher acuity but outperformed all other models at ESI 3 (untrained: 72.7%; trained: 81.8%).

Conclusion: AI models, particularly when fine-tuned, have shown promising results in accurately predicting ESI levels for pediatric patients, potentially improving the efficiency of triage in pediatric ED settings. These findings suggest that AI could be a valuable adjunct to clinical judgment in pediatric triage, though further research is needed to validate these models in real-world settings.