Comparing Ophthalmologist and Artificial Intelligence Chatbot Responses to Patient Questions

Introduction

The introduction of innovative artificial intelligence (AI) digital technologies at a rapid rate has provided healthcare workers with a new potential opportunity for more efficient and comprehensive care for patients.¹ Innovative ways to utilize telehealth, AI, and machine learning in the field of ophthalmology have shown considerable promise and capacity to improve health access.¹ Concomitantly, the transition to virtual care during the COVID-19 pandemic has led to an increase in clinicians’ time spent on electronic health records (EHR) addressing patient messages,² which may increase physician burnout.

Patients often turn to online resources for health information. In the United States, approximately 80% of internet users rely on the web for health information.³ The emergence of large language models (LLM), such as ChatGPT,⁴ may serve to corroborate online ocular health resources and information from eye specialist consultations to help patients find their answers more quickly and efficiently,⁵ while providing empathetic responses.⁶ Currently, ChatGPT boasts over 180 million users.⁷

Emerging studies have illustrated ChatGPT’s ability to generate ophthalmic differential diagnoses,⁸ answer patient health questions,⁹ and perform well on formal ophthalmology examinations.^10,11 Furthermore, studies have demonstrated how AI can even be used to employ automated assessment, such as when screening articles within systematic reviews,^12,13 or for the screening, diagnosis, and monitoring of ocular pathologies.^14–18 A comparison between physician and ChatGPT responses to general health questions on Reddit, an online social media forum, found ChatGPT responses to be of higher quality, empathy, and rated more preferably by physicians.⁹ Bernstein et al, analyzed questions and physician responses on The Eye Care Forum, and found that ChatGPT responses did not differ significantly from ophthalmologist-written responses and were difficult to distinguish.¹⁹ Lyons et al, found that ChatGPT-4 was able diagnose and triage 44 de novo ophthalmology clinical vignettes with comparable accuracy to ophthalmology trainees at a single centre.²⁰

While many online forums exist to ask physicians health-related questions, the American Academy of Ophthalmology (AAO) “Ask an Ophthalmologist” forum is an accredited platform for patients to ask ophthalmologists about their eye health.²¹ In this retrospective cross-sectional study, we assessed the accuracy, similarity, readability, empathy, and length of ChatGPT’s (version GPT-4o, OpenAI) responses to patient questions in comparison to ophthalmologists’ responses on a public AAO forum.

Materials and Methods

This is a retrospective cross-sectional study of patient questions and ophthalmologist responses from the “Ask an Ophthalmologist” forum from January 2016 to December 2022. In accordance with AAO Terms of Service, the data was anonymized, and formal permission was obtained from the AAO to use the forum data for this study.²² The AAO granted approval and permitted the input of up to 250 questions into ChatGPT for comparison with ophthalmologist responses. We did not report any identifying information, including physician names in this study. Each patient question was inserted directly into a new ChatGPT session without editing the wording, grammar, or spelling of said question (in September 2024). ChatGPT was selected for this study as it was the most widely used publicly accessible large language model among patients at the time, providing the most relevant platform for assessing AI-generated responses to patient questions. Replies which referred patients to other resources or videos to find the answer to their question were excluded. Unanswered questions were excluded as a comparison could not be made. ChatGPT responses were anonymized by removing revealing information (eg, “As an AI language model…”). This study was exempted from requiring ethics approval by the University of Toronto Research Ethics Board (REB) as it utilized publicly available information with no expectation of privacy.

The readability of ophthalmologist and ChatGPT (version GPT-4o, OpenAI) responses to patient questions were assessed using the Flesch-Kincaid Grade Level, Flesch Reading Ease score, and Gunning Fog Index.²³ To compare response accuracy, two board-certified ophthalmologists (EI, CL) independently evaluated 100 patient questions and their anonymized responses from both sources using a 7-point Likert scale. They rated accuracy based on agreement with the statement: “The response provided is accurate” (1=Strongly disagree, 2=Disagree, 3=Somewhat disagree, 4=Neither agree nor disagree, 5=Somewhat agree, 6=Agree, 7=Strongly agree). Evaluators also indicated which response they preferred.

To assess empathy, graders rated responses based on agreement with the statement: “The response provided is empathetic” using the same Likert scale. Characteristics of empathetic responses included acknowledging the user’s frustration, confusion, or concern, providing reassurance, providing guidance on all components of a user’s health-related query, and demonstrating support for the user.²⁴ Ratings of accuracy, and empathy were subsequently converted to a numerical scale ranging from 1 to 7, and the mean score between the two raters was used.

Text similarity between ophthalmologists and ChatGPT responses were also compared using CopyLeaks, which is an AI-powered tool used to compare the extent of similarity by categorizing text as “identical” (exact word-for-word matches), “minor changes” (minor variations in a sentence but with the same meaning), and paraphrased text (re-written using different words or sentence structures while retaining the same core idea).²⁵

Statistical Analysis

The distribution of continuous variables was examined for normality using a histogram of data spread, Q–Q plots, and the Shapiro–Wilk test. After assessing for assumptions of normality and similar variance, the paired samples t-test was used to compare the readability, empathy, accuracy, and number of words in ophthalmologists and ChatGPT responses.²⁶ If these assumptions were not met, the Wilcoxon signed-rank test was used instead. Statistical analyses were conducted using R version 4.4.2 (R Foundation for Statistical Computing, Vienna, Austria). All tests were two-tailed, and P values less than 0.05 were considered statistically significant.

Results

A total of 1079 questions and responses from 41 ophthalmologists within 30 designated subtopics over the study (2016–2022) period were considered, after 2 questions were excluded. One question was excluded as no response was provided, and the other because the patient was provided a link to an existing video to find the answer to their question. All physicians were board certified ophthalmologists with either an MD (40/41, 97.6%) or DO (1/41, 2.4%), with an average of 28.7 years in practice. Subspecialty representation included retina/vitreoretinal surgery (9/41), comprehensive ophthalmology (8/41), cornea (7/41), glaucoma (6/41), pediatric ophthalmology and strabismus (6/41), and oculoplastics (4/41). The average length of patient questions was 34.1 words (SD=25.1). The number of responses from any single author ranged from answering between 1 and 151 patient questions. Most responses were tagged with more than one subtopic (983/1079, 91.1%). The most tagged topics were “Surgery” (296/1519, 19.5%), “Cataracts” (219/1519, 14.4%), “Glasses, Contacts and Vision Correction” (126/1519, 8.3%), and “General Eye Health” (125/1519, 8.2%). The mean number of words in ophthalmologist responses were significantly lower than ChatGPT responses (80.6 [SD=56.4] words vs (337.8 [SD=141.6] words, p<0.001).

The results indicated that AAO responses were easier to read and could be understood by patients with a lower level of education. AAO responses had a higher mean Flesch Reading Ease Score than ChatGPT responses (50.2 [SD=14.0] vs 38.2 [SD=10.4], p<0.001) and a lower mean Flesch-Kincaid Grade Level (Grade 11.0 [SD=2.7] vs Grade 12.7 [SD=1.9], p<0.001, Table 1), making them more accessible. Similarly, the Gunning Fog Index showed that AAO responses required a lower reading level compared to ChatGPT responses (Grade 14.7 [SD=3.9] vs Grade 15.2 [SD=2.3], p<0.001, Table 1).

Table 1 Readability of ChatGPT and Ophthalmologists’ Responses to Patient Questions

The mean text similarity between ophthalmologist and ChatGPT responses, as measured by CopyLeaks, was less than 1%. Only three responses exhibited any similarity (mean=33.17%), all of which were classified as “paraphrased” rather than “identical” or containing only “minor changes”.

The mean accuracy of ChatGPT responses was comparable to AAO responses (5.8 [SD=1.1] vs 5.5 [SD=1.1], V=1087.5, p=0.07; Figure 1). Evaluators preferred ChatGPT over physician responses in approximately half (49.5%) of cases. Empathy ratings did not differ significantly between ChatGPT and AAO responses (4.42 [SD=0.57] vs 4.38 [SD=0.58], V=1265.5, p=0.5), as shown in Figure 2.

Figure 1 Distribution of mean accuracy ratings for ophthalmologists and ChatGPT responses.

Figure 2 Distribution of mean empathy ratings for ophthalmologists and ChatGPT responses.

Discussion

In this study, ChatGPT demonstrated slightly higher accuracy but was preferred at a similar rate to ophthalmologists’ responses. However, response preference did not always align with accuracy, as more detailed, textbook-like responses were sometimes rated as less appropriate for patients. In some cases, responses with lower accuracy ratings were preferred because they were clearer and more suitable for patients. Similarly, Nanji et al, compared ChatGPT to other online materials for providing postoperative patient instructions, and found that while ChatGPT provided comparable procedure-specific information, its responses were less understandable.²⁷ Our analysis showed that physician responses on the AAO “Ask an Ophthalmologist” forum were generally easier to comprehend, required a lower reading grade level, and were shorter than ChatGPT responses. Notably, users can prompt ChatGPT to simplify its language (eg, “please use simpler language”) if a response is too complex. This adaptability has been successfully applied in other healthcare settings, such as simplifying radiological reports for patient education.²⁸

Previous studies have shown that physicians have difficulty differentiating between chatbot and human-written content,^12,29–32 including responses to online patient questions.¹⁹ This high degree of similarity suggests that ChatGPT responses could serve as templates, allowing physicians to make minor edits before sending them to patients. Although not formally assessed, evaluators in this study also reported recognizing ChatGPT responses based on writing style and response length. In our study, no instances of “hallucinations” or fabricated information were observed. However, prior literature has documented that large language models, including ChatGPT, can generate inaccurate or misleading information, particularly when addressing complex or specialized medical questions.^33–35 This phenomenon likely stems from the models’ reliance on pattern recognition in text rather than true domain-specific understanding. Consequently, chatbots should not be used to answer patient questions without oversight, and physicians should be aware of the potential medico-legal implications of disseminating unvetted or incorrect information.^36,37

The use of online chatbots or AI technology has been widely implemented in other industries to reduce workload burden and improve efficiency,^38,39 and similar strategies can be implemented in medicine. In the context of health information, the consequences of inaccurate information carry greater risks.^10,40 Concerns around information accuracy on ChatGPT are valid, but one must consider that alternatives to ChatGPT from the patient perspective include finding this information online. Studies evaluating ocular information online have raised similar concerns about quality and accuracy.⁴¹ Within our dataset, we noted that after providing medical advice, ChatGPT generally recommended that users bring up their concerns to their physicians or eye specialists for further clarification or investigation of a complaint when warranted. Thus, ChatGPT usage by patients serves to corroborate care delivered by physicians, as patients find the abundance of health information on the web overwhelming, conflicting, and confusing.⁴² ChatGPT may help make health information more easily accessible to patients.

Applications of AI in ophthalmology, and in particular deep learning (DL),^15,43,44 has shown tremendous potential in the screening, diagnosis, and monitoring of ocular disease progression.^{14–18,43,44} These include AI-based detection of retinal fluid in spectral domain OCT,¹⁸ AI-enabled monitoring glaucoma disease progression and severity,¹⁷ and detection of diabetic retinopathy.¹⁶ Similar to imaging-based machine learning applications, when using AI-based responses to patient questions, black-box limitations apply, as it is unclear to what extent ChatGPT is able to evaluate and prioritize sources of information. In our study, the use of ophthalmologist-rated accuracy may also present additional biases.

Our findings are consistent with other studies on the applications of ChatGPT in patient medical education.^8–11 Ayers et al, evaluated physician responses to general patient health questions, and found ChatGPT responses to patient questions to be shorter and more empathetic,⁹ while our study found ChatGPT responses to be longer and similar in empathy. This may be due to differences in how empathy was judged by raters, differences in comparison groups, and other methodological differences. For instance, Yılmaz et al employed a sentiment-analysis approach, categorizing the emotional tone and attitude of responses using automated and manual approaches, rather than using structured empathy scoring.⁴⁵ Their results similarly highlight that AI-generated ChatGPT responses conveyed more supportive and instructive emotional content. Responses in our study were extracted from the AAO forum, while Ayers et al, utilized responses on a social media forum called Reddit.⁹ Bernstein et al, compared ChatGPT and ophthalmologist responses to patient questions, and found human-rated response accuracy did not differ significantly, and physicians had a difficult time distinguishing human and AI responses.¹⁹ Several other studies have demonstrated the difficulty of differentiating human and AI responses.^12,29,30

Previous studies have demonstrated that chatbot performance may also vary across ophthalmic subspecialties.^45–50 For example, a comparison of ChatGPT, Bard, and Copilot against a trusted patient-information resource (AAPOS) found that, although chatbots demonstrated potential, the AAPOS website consistently outperformed them in both accuracy and readability.⁴⁵ Additionally, chatbot performance has been shown to differ across platforms, highlighting variability in responses depending on the AI model and interface used.^20,47,49

One must consider the limitations and biases of these tools when used in isolation, such as the provision of inaccurate information.⁴⁰ Accordingly, the potential benefit of chatbot usage by physicians may be limited to drafting responses to patient questions that the physician can then review, thus reduce physician workload.⁹ While the utility of generative AI models in ophthalmology were initially criticized due to their inability to process images,¹⁰ and for being trained on a dataset using information up until 2021,⁵¹ newer generative AI models, including ChatGPT-4o, are capable of image processing and have access to the most contemporary information on a topic.⁵² The medico-legal and ethical implications of using ChatGPT in patient communication must also be considered. ChatGPT may generate inappropriate or inaccurate medical advice,³⁶ and, lacking legal personhood, the responsibility for any resulting decisions ultimately rests with the user.³⁷

Limitations and Future Directions

This study compared chatbot to ophthalmologist responses from a single online forum, thus limiting the generalizability of the findings. It is also likely that the publicly available online forum used in this study was incorporated into the training dataset for ChatGPT. In addition, it is unclear exactly how ChatGPT processes information and determines which sources of information are credible. This study evaluated only ChatGPT, which, while being one of the most widely used and publicly accessible large language models at the time of analysis, represents just one of several available platforms. As such, the findings may not be generalizable to other models. Furthermore, all responses were evaluated by board-certified ophthalmologists rather than by patients themselves. While expert assessment allows for objective evaluation of accuracy, readability, and empathy, it may not fully capture patient perspectives or experiences, potentially limiting the interpretation of real-world utility. Future studies should compare the time ophthalmologists spend responding to patients’ questions with or without prior ChatGPT-generated responses, to quantify the degree to which using these tools may affect physician workload. Future studies involving patients may consider how these new technologies offer utility within various demographics, such as older adults. In addition, the different ways AI-assisted responses can be safely integrated into hospital or clinical settings should be further investigated.

Conclusion

This retrospective analysis demonstrates the feasibility of utilizing AI chatbots to address patient eye health queries. Our findings suggest that chatbots have the potential to reduce physician workload by drafting initial responses to patient ocular concerns and increase efficiency.