Evaluating Large Language Models for Accuracy and Completeness of Vitiligo Patient Education: A Comparative Analysis

Introduction

Vitiligo is a prevalent chronic acquired condition characterized by the loss of skin pigmentation.¹ Over half of individuals with vitiligo experience psychological disorders, which significantly impair their quality of life.^2,3 Patients and their families frequently express questions and concerns about the disease. Currently, patient education predominantly relies on online educational websites, support groups, community activities, and social media platforms. However, these approaches are limited by inconsistent information quality and challenges in ensuring accuracy. Alternative methods, such as face-to-face consultations with physicians, institutional brochures, and lectures, are constrained by geographic and temporal limitations, rendering them less accessible as on-demand educational resources. These shortcomings underscore the urgent need for dependable educational tools beyond clinical settings.⁴ Large Language Models (LLMs) have gained attention across various medical domains due to their versatility, driven by extensive parameter spaces that enhance their utility in medical patient education.⁵ Dermatology is a medical specialty that heavily relies on visual assessment, patient communication, and complex decision-making. LLMs demonstrate exceptional performance in image recognition, aligning seamlessly with dermatology’s emphasis on visual diagnosis. Early detection of skin cancer represents the most mature and widely studied application of artificial intelligence (AI) in this field. LLMs have shown impressive capabilities in providing clinical decision support and assisting with medical writing. A study by Iqbal et al⁶ found that LLMs, utilizing medical databases from Taiwan and the United States, generated medication recommendations highly consistent with those of dermatologists, effectively optimizing clinical workflows. However, due to the diverse nature of dermatological conditions and individual patient variability, clinical decision-making schemes provided by ChatGPT exhibit limitations in subspecialty areas. Dunn et al⁷ compared the quality of ChatGPT-generated dermatology case reports with published reports, demonstrating through blinded expert review that ChatGPT-generated texts surpassed human-written reports in both quality and readability.

Dermatology patients frequently have questions regarding their conditions, treatments, and care plans. Traditional face-to-face communication models are time-intensive for healthcare providers. The advent of LLMs offers a novel approach to patient education. Ayers et al⁸ highlighted the ability of LLMs to deliver high-quality, empathetic responses to patient inquiries on social media, achieving higher patient satisfaction compared to physician responses. Similarly, Michelle et al⁹ reported that patients preferred responses generated by large language models over those prepared by the American Society for Mohs Surgery.

The application of LLMs in dermatology is rapidly expanding. Several commercial AI-based dermatology diagnostic applications, such as SkinVision and MoleMapper, have entered the market. LLMs are enhancing global patient education by extending dermatological knowledge and preliminary diagnostic capabilities to resource-limited regions, thereby reducing geographical and economic barriers.¹⁰ LLMs can generate tailored health education materials based on patients’ specific conditions, cultural backgrounds, and educational levels. Through intelligent applications, patients can access real-time skin care advice and learn to accurately identify and manage common dermatological issues. The accessibility and personalization of LLMs address deficiencies in clinical patient education. Reasoning-LLMs, a new generation of LLMs, differ from traditional LLMs, which rapidly generate outputs based solely on statistical guesses of the next word. Instead, reasoning-LLMs take time to break down problems into individual steps, deriving more accurate answers through “thought” and self-verification,¹¹ making them highly suitable for complex, precision-demanding tasks such as medical education.

Chatbot Arena, an open platform by LMSYS, evaluates LLMs using paired, anonymized comparisons and Elo scoring based on human preferences.¹² From its leaderboard, we selected three top-ranking reasoning LLMs—OpenAI o1, DeepSeek-R1, and Grok3—for this study. To date, no research has assessed the efficacy of LLMs in addressing vitiligo-related questions. With the increasing number and rapid evolution of LLMs, there is also a lack of studies evaluating whether significant differences exist among them in the context of vitiligo. This study aims to fill this gap by evaluating the accuracy, completeness, and comprehensibility of different LLMs, thereby assisting vitiligo patients in selecting tools for medical inquiries and enhancing patient education and dermatological support.

Materials and Methods

This cross-sectional study received approval from the Research Ethics Committee of Jinan Central Hospital (Registration No. 20250530023). All participants provided informed consent, in accordance with the Declaration of Helsinki.

Material Sources and Processing

Drawing on established guidelines,^13,14 prior literature,^15–18 and clinical expertise, a questionnaire comprising 16 vitiligo-related questions was developed by two experienced dermatologists (Hang Li and Le Zhuang), each with more than 10 years of expertise in the field of vitiligo. The questions used include 8 questions aimed at describing the topic of vitiligo and 8 questions aimed at providing advice related to vitiligo. The questions could each be divided into “basic” or “advanced” content. “Basic” included simple questions in terms of content, and “advanced” included questions with specific inquiries in the field. The expert panel consulted a group of five patients with vitiligo, recruited from the vitiligo clinic of our hospital, and two other dermatologists from the Department of Dermatology to validate the generated questions. Their feedback was incorporated to refine the questions further and ensure that the questions effectively captured the perspectives and information needs of the patient community. The questions used to prompt LLMs are listed in Box S1 in the Supporting Information.Three Large Language Models (LLMs)—OpenAI o1 (OpenAI, San Francisco, California, USA), DeepSeek R1 (DeepSeek, Hangzhou, Zhejiang, China), and Grok3 (xAI, San Francisco, California, USA)—were selected to assess the performance of AI-generated content. In February 2025, a series of questions were individually submitted by Jiaxuan Chen to three distinct chatbot platforms. To ensure response consistency and independence, each question was entered separately using the “new chat” function provided by the respective platform interfaces. Responses generated by each LLM for every question were systematically documented. Each LLM generated one detailed answer for each question, resulting in 48 answers (16 questions×3 LLMs). To maintain objectivity, all responses were converted into plain text format, and the presentation of the materials was standardized using uniform font type and size. No additional modifications to the format were applied.

Assessment of Materials

Each response was assessed from both patient and professional perspectives. A panel of three dermatologists (Xi Yang, Yi wang, and Jieyan Su), each with over five years of experience in vitiligo, who were not involved in the question generation process, assessed the accuracy, completeness, and relevance of the responses using a 5-point Likert scale. Additionally, similar to previous report,⁸ three patients with vitiligo, who had not participated in question generation, contributed to the evaluation. These patients rated the comprehensibility using a similar 5-point Likert scale. Participating vitiligo patients were required to be aged 18 years or older, possess proficient English reading and comprehension skills, and successfully complete a simplified Flesch-Kincaid reading comprehension test designed for a 10th- to 12th-grade level. Likert scale definitions are provided in Table 1. To ensure that the information was presented in an accessible manner, readability was further evaluated using the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) metrics.¹⁹ The FRE scale, ranging from 0 to 100, measures text readability, with higher scores indicating greater ease of comprehension and lower scores reflecting increased difficulty.²⁰ The FKGL estimates the educational level required to understand the text, with higher scores denoting greater complexity.²¹

Table 1 Medical Experts and Patients Evaluation Scores Criteria

Data Processing

The final score for each response was determined as follows: if all three raters assigned identical scores, that score was adopted. If two raters provided the same score, their agreed-upon score was used. In cases where all three raters assigned different scores, the lowest score was selected to ensure a conservative evaluation.

Statistical Analysis

Descriptive analyses were conducted to evaluate the accuracy, completeness, relevance, and comprehensibility of the LLM responses. Pairwise comparisons within the Friedman test were conducted to assess differences between models. Inter-rater agreement was assessed using Kendall’s W coefficient of concordance. The coefficient value (W) below 0.4 suggested poor agreement, and moderate agreement had the value between 0.4 and 0.59, high agreement with 0.6–0.79, excellent agreement with value ≥0.8.²² Data analyses were performed by SPSS Statistics 27.0 (IBM Corp., Armonk, NY, USA) and GraphPad Prism 10.4.0 (GraphPad Software, San Diego, CA, USA). Data recording and organization were facilitated by Microsoft Excel 2021. p < 0.05 was considered significant.

Results

Categorization of the Questions and Agreement Between Reviewers

Table S1 in the Supporting Information provides the scores assigned to the responses. Inter-rater reliability, evaluated using Kendall’s coefficient of concordance (W), was high for both the patient group (W = 0.653) and the physician group (W = 0.772), indicating strong agreement among evaluators. Responses with scores exceeding 4 were classified as accurate. All three LLMs—OpenAI o1, DeepSeek-R1, and Grok3—achieved 100% accuracy on descriptive questions. For recommendatory questions, accuracy rates were 75% for OpenAI o1, 87.5% for DeepSeek-R1, and 75% for Grok3.

Accuracy Assessment

Accuracy reflects the precision and verifiability of the facts, data, and knowledge presented in the responses. Median accuracy scores were 4 for OpenAI o1 and 5 for both DeepSeek-R1 and Grok3, with DeepSeek-R1’s first quartile surpassing that of Grok3 (Table 2). The Friedman test revealed a statistically significant difference between OpenAI o1 and DeepSeek-R1 (p = 0.042), whereas no significant differences were observed between Grok3 and OpenAI o1 or between Grok3 and DeepSeek-R1 (p = 0.157, p = 0.536, respectively) (Figure 1a).

Table 2 Median Scores for Answers From Three Large Language Models

Figure 1 Comparison of Medical Expert Evaluation Scores for Vitiligo Assessment Across Large Language Models. (A) Accuracy scores showed a statistically significant difference between OpenAI o1 and DeepSeek-R1 (p = 0.042), but not between Grok3 and OpenAI o1 (p = 0.157) or Grok3 and DeepSeek-R1 (p = 0.536). (B) Completeness scores indicated a significant difference between OpenAI o1 and DeepSeek-R1 (p = 0.020), with no significant differences between Grok3 and OpenAI o1 (p = 0.052) or Grok3 and DeepSeek-R1 (p = 0.724). (C) Relevance scores showed no statistically significant differences among the three models (p > 0.05).

Specific inaccuracies were noted. DeepSeek-R1 erroneously recommended a mid-potency topical glucocorticoid (eg, triamcinolone 0.1%) for vitiligo, whereas global experts in vitiligo diagnosis and management recommend potent to very potent topical corticosteroids.²⁰ This error is also commonly observed among dermatologists not specializing in vitiligo. Meanwhile, OpenAI o1 and Grok3 incorrectly deemed narrowband ultraviolet B (NB-UVB) phototherapy unsafe during pregnancy, despite evidence establishing NB-UVB as a safe treatment option for various dermatological conditions during pregnancy.^23,24 These errors underscore the limitations of LLMs in addressing highly specialized queries.

Completeness Assessment

Table 2 indicates that the median completeness scores for all three models exceeded 4, reflecting their ability to deliver comprehensive analyses and integrate diagnostic and therapeutic information. Median completeness scores were 5 for DeepSeek-R1 and 4 for both OpenAI o1 and Grok3, with Grok3’s third quartile surpassing that of OpenAI o1. The Friedman test identified a statistically significant difference between OpenAI o1 and DeepSeek-R1 (p = 0.02), whereas no significant differences were observed between Grok3 and OpenAI o1 or between Grok3 and DeepSeek-R1 (p = 0.052, p = 0.724, respectively) (Figure 1b).

Relevance Assessment

All three LLMs achieved maximum median, first quartile, and third quartile scores of 5. No statistically significant differences were observed among the three LLMs (Friedman test, two-way analysis of variance by ranks, p > 0.05) (Figure 1c), indicating that their responses were consistently pertinent to the questions posed. The systematic logical reasoning capabilities of reasoning-LLMs enhance comprehension and minimize the likelihood of irrelevant outputs.

Patient Assessment

OpenAI o1 and Grok 3 achieved the highest median scores for perceived understandability (median: 5, IQR: 5–5), indicating that patients found their responses highly comprehensible. DeepSeek-R1 had a slightly lower median score of 5 (IQR: 4.75–5), suggesting marginally reduced comprehensibility. No statistically significant differences were observed among the three LLMs (OpenAI o1 vs DeepSeek R1, p = 0.289; OpenAI o1 vs Grok 3, p = 0.289; DeepSeek R1 vs Grok 3, p = 1.000) (Figure 2a).

Figure 2 Comparison of Patient Evaluation and Readability Scores for Vitiligo Assessment Across Large Language Models. (A) Comprehensibility scores showed no statistically significant differences among the three models (OpenAI o1 vs DeepSeek-R1, p = 0.289; OpenAI o1 vs Grok3, p = 0.289; DeepSeek-R1 vs Grok3, p = 1.000). (B) Flesch Reading Ease scores revealed highly significant differences between DeepSeek-R1 and both OpenAI o1 (p < 0.001) and Grok3 (p < 0.001), with no significant difference between OpenAI o1 and Grok3 (p = 1.000). (C) Flesch-Kincaid Grade Level scores indicated significant differences between DeepSeek-R1 and both OpenAI o1 (p = 0.030) and Grok3 (p = 0.020), with no significant difference between OpenAI o1 and Grok3 (p = 0.855).

Readability Assessment

DeepSeek-R1 demonstrated lower readability relative to OpenAI o1 and Grok3, with a mean Flesch Reading Ease (FRE) median score of 17.95, compared to OpenAI o1 (median: 34.1) and Grok3 (median: 37.4). The Friedman test indicated highly significant differences between DeepSeek-R1 and both OpenAI o1 (p < 0.001) and Grok3 (p < 0.001), with no significant difference between OpenAI o1 and Grok3 (p = 1.000)(Figure 2a). Correspondingly, DeepSeek-R1’s mean Flesch-Kincaid Grade Level (FKGL) median score was 13.45, higher than OpenAI o1 (median: 13.35) and Grok3 (median: 12.05). The Friedman test indicated highly significant differences between DeepSeek-R1 and both OpenAI o1 (p = 0.03) and Grok3 (p = 0.02), with no significant difference between OpenAI o1 and Grok3 (p = 0.855) (Figure 2b).

We constructed a summary table (Table 3) that visually compares the strengths of the three LLMs.

Table 3 Comparison of LLM Advantages

Comparisons Across Difficulty Levels

Analysis of responses stratified by question difficulty revealed no statistically significant differences among the three LLMs at any individual difficulty level (p > 0.05) (Figure 3). This suggests that no single model exhibited superior performance in addressing complex questions.

Figure 3 Performance comparison of three large language models across difficulty levels. No statistically significant differences were observed among the models at any individual level (p > 0.05). (A) Accuracy scores across difficulty levels. (B) Completeness scores across difficulty levels.

Discussion

The errors observed in the perspectives of all three large language models (LLMs) regarding the use of topical corticosteroids and phototherapy for vitiligo mirror common misconceptions held by many dermatologists not specializing in vitiligo. These inaccuracies may stem from the incorporation of erroneous information into the training datasets. Accuracy could be improved by training models with the latest clinical practice guidelines or regularly updated, domain-specific datasets.^25,26 In scenarios where questions were posed as if from a 10-year-old child, all three LLMs adjusted their readability to an FKGL of 8th grade, demonstrating commendable humanistic care. Compared to traditional LLMs, reasoning-LLMs exhibited a marked improvement in contextual awareness. Based on prior research,²⁶ this study designed a consensus-based scoring rule and conservative estimation for data preprocessing to minimize the impact of subjective differences among raters. Additionally, analyses using all individual scores were conducted, yielding results consistent with the consensus-based approach.Different LLMs show varied strengths in patient education for different diseases. A comparison of ChatGPT and Gemini for melanoma patient education found Gemini outperformed ChatGPT in response completeness, personalization, and readability.²⁷ In a study comparing ChatGPT-4 and Google Bard on common Mohs surgery patient questions, ChatGPT-4 demonstrated significantly higher accuracy and utility, matching the American College of Mohs Surgery’s response accuracy.²⁸ For syphilis patient education, ChatGPT-4 and Claude achieved the highest accuracy, aligning with WHO standards at 92% and 89%, respectively. Perplexity and Llama 3 were less reliable, scoring between 60–70%. Errors increased in rarer conditions like tertiary and neurosyphilis, with LLMs providing outdated treatment protocols and incorrect transmission descriptions.²⁹

LLMs and their applications in healthcare represent a globally significant topic in contemporary technological advancements. Patient use of AI can heighten disease awareness and promote early diagnosis.³⁰ Beyond serving as a source of medical information for patients, LLMs can also assist professional healthcare providers through electronic messaging.^19,31 By offering preliminary assessments and guidance on patient conditions, LLMs can support remote patient consultations, thereby increasing access to dermatological care.³² Studies indicate widespread user enthusiasm for LLM-derived health advice, suggesting that these models can serve as an alternative information source for patients, particularly when direct contact with a physician is unavailable.^33,34 In regions where access to specialized dermatologists is limited, LLMs can provide timely medical advice, bridging healthcare disparities in rural and underserved areas while offering initial guidance on dermatological issues.¹⁰ OpenAI exhibits robust reasoning capabilities and multimodal support, enabling it to process both text and images. This allows it to deliver intuitive educational content for patients, such as explaining the symptoms and treatment options for vitiligo, potentially supplemented by images illustrating skin changes. Such multimodal functionality is particularly valuable for helping patients comprehend the visual manifestations of the disease and the effects of treatment. In contrast, Grok 3, developed by xAI, offers distinct advantages through its real-time data access and deep integration with the X platform. It can provide the latest updates on vitiligo treatment advancements, research developments, and relevant discussions from social media. This capability is especially beneficial for patients seeking current medical insights and community support, which are critical in managing a condition like vitiligo that may impact mental health, as social support can significantly alleviate anxiety. Meanwhile, DeepSeek-R1, an open-source and freely accessible model, has a running cost approximately 13 times lower than comparable OpenAI models.³⁵ Designed to prioritize deep reasoning and complex problem-solving, it demonstrated superior accuracy and completeness in this study. Optimized for Chinese-language contexts, DeepSeek-R1 enhances accessibility and understanding for non-English-speaking vitiligo patients. To improve the readability of its English responses, training can incorporate real doctor-patient dialogue scripts to generate natural, patient-oriented language while adapting expressions for diverse cultural backgrounds.

Several limitations of this study warrant acknowledgment. First, consistent with previous exploratory studies, the relatively small sample size may have limited statistical power for detecting differences in comparative analyses, particularly when applying Bonferroni correction for multiple comparisons. Second, this study focused exclusively on text-based responses and did not incorporate images or diagnostic evaluations, potentially limiting its applicability to broader clinical contexts. Third, despite moderate-to-high inter-rater reliability, the scoring process remained inherently subjective. Future research should develop a standardized scoring system tailored for medical information to more comprehensively evaluate the feasibility of AI applications in healthcare.³⁶

Conclusion

Reasoning-LLMs can provide high-quality responses to general questions about vitiligo, with their high-performance reasoning capabilities, combined with real-time data integration, offering significant advantages in delivering patient education information. Among the three models evaluated, DeepSeek-R1 achieved the highest accuracy. However, these models currently cannot offer definitive treatment recommendations and may produce errors when addressing the needs of special patient populations. These limitations highlight the need for further refinement, including improved filtering mechanisms by developers and mandatory human oversight in clinical applications. Although LLMs can support clinical decisions and enhance patient education, they should not be viewed as substitutes for dermatologists.