P3.E083: Dr. ChatGPT, is my child normal? An Investigation into a Public Artificial Intelligence Chatbot’s Categorizations of Potential Developmental Delays

Background: Just as Google is often consulted for medical information, public artificial intelligence (AI) tools like ChatGPT may also be used to answer questions typically directed at physicians. It is therefore important to understand what kind of medical information ChatGPT produces, especially in domains where people often seek assurances – such as parents concerned about the development of their child.

Methods: Developmental concerns (n=108) from ages 0-5 were sourced from articles on the American Academy of Pediatrics’s www.healthychildren.org. These concerns are labeled as “warning signs'' that warrant discussion with a pediatrician. Concerns were pasted into ChatGPT-3.5 in the format of “My (insert age) child is (insert concern). Is this normal?” ChatGPT’s response was scored by its displayed concern level (see Table 1). Two board-certified DBP physicians scored ChatGPT’s response accuracy and completeness using five point Likert scales. Physicians also scored each concern according to Table 1 and categorized delays by developmental domains (Motor, Social/Emotional/Behavioral (SEB), Cognitive, etc.)

Results: ChatGPT produced accurate (4.91/5) and complete (4.5/5) information, deemed 20.4% of cases (n=22) “Abnormal” and 74.1% of cases (n=80) as “Not Abnormal”, and did not categorize six cases. Conversely, physicians scored 53.7% of cases (n=58) “Abnomal” and 43.5% of cases (n=47) as “Not Abnormal”, while not categorizing three cases. A Chi-square test of independence revealed that the proportions of Abnormal/Not Abnormal decisions between physicians and ChatGPT significantly differed, with ChatGPT having a lower number of cases classified as “Abnormal” (X2 (1, N=207) = 24.3765, p < .0001). There was agreement in 59% of cases categorized by both physicians and ChatGPT (Figure 1). ChatGPT showed lower concern (denoting a physician-scored “Abnormal” case as “Not Abnormal”) in 36% of cases and higher concern (denoting a physician-scored “Not Abnormal” case as “Abnormal”) in 5% of cases (Figure 1). Finally, ChatGPT had slightly less alignment with physicians in cases that were SEB versus motor (50% versus 62%; ), as well as cases that concerned < 12 month-old children versus >12 month-old children (52% versus 66%), though neither of these differences were significant via Chi-squares.

Conclusion: ChatGPT displayed that while it can generate responses containing accurate and complete information, it is still not yet ready to advise parents on their child’s developmental delays. The chatbot has a high rate of displaying lower concern than physicians for potential developmental delays, indicating the bot’s reluctance to deem a case abnormal. ChatGPT also seems slightly less equipped to handle SEB delays for older children, as evidenced by its comparatively lower alignment rates in those domains.