Comparative Impact of ChatGPT and Conventional Search Tools on Clinical Reasoning Performance: A Randomized Crossover Study in Preclinical Medical Students

Introduction

Digital transformation has made online learning and just-in-time information seeking routine in undergraduate medical education. A meta-analysis of online versus offline learning in undergraduate medical education reported that online learning can achieve learning outcomes comparable to or better than offline instruction, supporting its widespread adoption in curricula.¹ In parallel, medical students commonly use mobile devices and general search engines to address knowledge gaps; a survey of preclinical medical students identified Google as the most frequently used search engine for researching unfamiliar terms or concepts.²

The public release of ChatGPT accelerated attention to artificial intelligence (AI) tools built on large language models (LLMs). GPT-4 can generate coherent, context-sensitive responses, but the technical report emphasizes that the model is not fully reliable and can produce hallucinations, warranting caution when reliability is important. In clinical reasoning contexts, such limitations may affect diagnostic accuracy and management decisions, highlighting the need for empirical evaluation in educational settings. A systematic review of LLMs in medical education describes expanding applications, including tutoring, feedback, assessment support, and exam preparation, while also highlighting variability in study rigor and the need for controlled empirical evaluations.³ In a crossover study in a surgery clerkship, both ChatGPT and Google search improved postintervention quiz scores, and postintervention performance did not differ significantly between tools, reinforcing that educational effects may depend on how tools are embedded into learning activities rather than on tool novelty alone.⁴

From a cognitive load perspective, LLM-generated synthesis may reduce extraneous cognitive load by organizing dispersed information and offering structured explanations, potentially improving efficiency in hypothesis generation and differential diagnosis formulation.^3,5 At the same time, integrating generative AI into medical education raises concerns about inaccurate or fabricated content and uncritical reliance that may promote cognitive offloading, weaken cognitive autonomy, and reduce opportunities for deliberate practice unless paired with faculty guidance and reflective verification strategies.^6,7 These considerations align with scaffolding theory, in which supports are provided to learners and progressively withdrawn as competence develops, and with self-regulated learning frameworks that emphasize metacognitive monitoring and active control of learning.^6,8

Clinical reasoning is widely framed as a core competence for physicians, encompassing information gathering, problem representation, hypothesis generation, differential diagnosis, diagnostic justification, and management planning.⁹ Yet definitions and conceptual boundaries vary, complicating alignment across teaching, learning activities, and evaluation.¹⁰ A scoping review of clinical reasoning assessment methods emphasizes selecting complementary assessments that sample different components of this complex construct in ways matched to intended purpose and context.¹¹ However, despite the increasing use of artificial intelligence tools in medical education, there remains limited empirical evidence directly comparing their impact on clinical reasoning performance with that of conventional search strategies, particularly in preclinical learners within structured case-based learning environments. Given that clinical reasoning is a core competency in medical education and is central to case-based learning, it was selected as the primary outcome of this study. Accordingly, this study compared ChatGPT with conventional search tools during structured case-based learning to evaluate their relative effects on clinical reasoning performance in preclinical medical students.

Materials and Methods

This randomized crossover study was conducted during a scheduled 180-minute instructional session within the musculoskeletal system course for second-year medical students at the Faculty of Medicine, Kasetsart University. The study was implemented within the complete academic cohort of 46 enrolled students.

Participants were randomly assigned in a 1:1 ratio using computer-generated simple random allocation to begin with either ChatGPT or conventional search methods (Google or PubMed). Following a baseline 10-minute pre-test, students completed a 20-minute structured case-based search activity using their assigned modality. A parallel 10-minute post-test (post-test 1) was then administered. A 60-minute washout period involving unrelated academic activities was implemented to minimize potential carryover effects and reduce immediate recall and tool-specific priming before crossover. Participants subsequently switched to the alternate search modality for a second 20-minute session, followed by a second parallel 10-minute post-test (post-test 2). Participants assigned to the ChatGPT group were allowed to use self-directed prompting strategies when interacting with the model. General guidance was provided to encourage engagement with the clinical case, reflecting authentic information-seeking behavior within a structured learning context.

All assessments were designed to evaluate clinical reasoning rather than factual recall. Questions required integration of pathophysiology, prioritization of differential diagnoses, and selection of appropriate investigations and initial management strategies. Assessment design and rubric-based scoring were aligned with established frameworks for clinical reasoning evaluation in medical education.^9,11 Each assessment was scored using a predefined rubric with a maximum score of eight points. The rubric was developed for this study to reflect key domains of clinical reasoning relevant to the course objectives, including problem representation, generation of differential diagnoses, selection of appropriate investigations, and initial management. Scores were assigned based on the completeness and appropriateness of responses across these domains. The rubric and scoring criteria were reviewed by faculty members involved in the course to ensure content validity, relevance, and clarity.

The primary outcome was objective clinical reasoning performance measured across three time points: baseline, post-test 1, and post-test 2. Secondary outcomes included self-perceived understanding and confidence, collected through structured evaluation forms.

Sample size was calculated for a paired t-test comparing post-test scores between learning modalities. Assuming a two-sided α of 0.05, 80% power, an anticipated mean difference of 1.0 point, and a standard deviation of paired differences of 1.5, the minimum required sample size was 18 participants. Inclusion of the full cohort of 46 students exceeded this requirement and increased statistical precision.

Statistical analyses were performed using paired t-tests to compare within-subject score differences across phases. Effect sizes were calculated using Cohen d for paired samples to quantify magnitude beyond statistical significance, consistent with recommendations for reporting educational intervention outcomes.¹² Statistical significance was defined as p < 0.05.

The study was approved by the Kasetsart University Research Ethics Committee under expedited review (KUREC-HSR68/072; COA No. COA69/02; approval date 14 January 2026; and was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants prior to enrollment.

Results

All 46 second-year medical students were assessed for eligibility, randomized into two intervention sequences, and completed the baseline pre-test, both intervention phases, and all post-test assessments. No participants were excluded from analysis. Participant progression through the study is presented in Figure 1.

Figure 1 CONSORT flow diagram and schematic representation of the randomized crossover study design. Forty-six second-year medical students were randomized into two intervention sequences. Group A received ChatGPT followed by conventional search, while Group B received conventional search followed by ChatGPT, separated by a 60-minute washout period. Each phase included a 20-minute learning activity followed by a 10-minute post-test. All participants completed both intervention phases and were included in the final analysis.

Descriptive statistics and paired comparisons of assessment scores across the three time points are summarized in Table 1. The mean pre-test score was 3.96 (SD 1.65), which increased to 4.96 (SD 1.71) after the first intervention phase and further to 5.70 (SD 1.50) following crossover and completion of the second intervention phase.

Table 1 Changes in Clinical Reasoning Performance Across Study Phases and results of Paired Comparisons

Paired analyses demonstrated statistically significant improvements across all comparisons, including pre-test to post-test 1 (t(45) = −3.79, p < 0.05), post-test 1 to post-test 2 (t(45) = −2.96, p < 0.05), and pre-test to post-test 2 (t(45) = −6.57, p < 0.05), with corresponding effect sizes ranging from moderate to large. No reduction in performance was observed following crossover.

Qualitative evaluation indicated that students perceived both learning modalities as beneficial. ChatGPT was described as facilitating rapid organization of differential diagnoses and management plans, whereas conventional search required more time but promoted comparison across multiple information sources. Students reported increased confidence when applying knowledge immediately after the instructional session and emphasized the importance of verifying artificial intelligence outputs against reliable references. Several participants noted that experiencing both approaches enhanced their ability to critically evaluate information sources. Qualitative feedback regarding the perceived strengths of each learning modality is summarized in Table 2.

Table 2 Summary of Qualitative Feedback on the Two Learning Modalities

Discussion

This randomized crossover study demonstrates that structured integration of both ChatGPT and conventional search tools significantly enhances short-term clinical reasoning performance among preclinical medical students. The large cumulative effect size observed across study phases suggests meaningful educational impact beyond simple exposure to digital tools, and the improvement across both intervention phases is consistent with reinforcement of reasoning processes rather than a transient novelty effect.

Recent literature has begun to explore the comparative utility of ChatGPT and traditional search tools in medical education. In a crossover study of third-year medical students during a surgery clerkship, ChatGPT was comparable to Google Search in improving quiz performance, with no statistically significant difference in postintervention scores between groups.⁴ Although that study provided early evidence that LLM-based tools may function as viable learning aids, its emphasis was feasibility and short-term score comparison within a small sample. In contrast, our study extends prior work by incorporating a larger cohort, reporting standardized effect sizes to convey magnitude beyond statistical significance,¹² and examining cumulative improvement across sequential learning phases. Importantly, our primary outcome targeted clinical reasoning performance assessed through structured rubrics rather than simple knowledge recall, aligning with recommendations to match assessment methods to specific components of the clinical reasoning construct.^9,11 Because crossover studies increase efficiency by using learners as their own controls but can be threatened by period and carryover effects if not addressed explicitly, careful design and transparent reporting remain essential.^13,14

From a cognitive load perspective, AI-generated synthesis may reduce extraneous load by organizing dispersed information and offering structured explanations, allowing learners to allocate attention to integration and application.^5,15 In parallel, scaffolding theory suggests that supports should be deliberately calibrated and gradually withdrawn to promote independent performance.⁸ However, LLM outputs can be fluent yet incorrect; the GPT-4 technical report acknowledges residual reliability limitations, and clinical guidance highlights hallucinations as a practical risk that requires verification against trusted sources.¹⁶ The progressive improvement observed after crossover may therefore reflect transfer of reasoning strategies across tools within a structured activity rather than dependence on a single tool.

Our interpretation of attitudes also differs from some prior reports. While the clerkship study reported reluctance among students to use ChatGPT for learning during clinical rotations,⁴ our qualitative findings suggest emerging critical appraisal behavior and metacognitive monitoring. This is educationally important because uncritical reliance on generative AI has been argued to promote cognitive offloading and threaten cognitive autonomy without explicit pedagogical scaffolding and faculty oversight.^7,17 Taken together with recent systematic and scoping reviews emphasizing both educational promise and persistent accuracy, ethics, and governance concerns, our findings support a hybrid approach that integrates LLMs as supervised cognitive supports while maintaining conventional search and appraisal skills.^3,18

In addition, large language models may generate hallucinations, producing responses that appear plausible but contain inaccuracies. In musculoskeletal case-based learning, such outputs may affect diagnostic reasoning or management planning if not critically appraised.^19–21 Furthermore, variability in prompting strategies can influence the quality and accuracy of responses, potentially affecting learning outcomes and limiting reproducibility. These considerations highlight the importance of verifying AI-generated information against reliable sources and providing appropriate guidance when integrating artificial intelligence into medical education. Artificial intelligence tools may therefore be incorporated into medical curricula as supportive learning resources within structured educational frameworks, with emphasis on maintaining critical appraisal and verification skills.

Despite these contributions, several strengths and limitations warrant consideration. Strengths of this study include its randomized crossover design, which improves internal validity by allowing each participant to serve as their own control, the complete participation rate within an authentic classroom setting, and the use of structured rubric-based assessment aligned with clinical reasoning theory. In addition, reporting standardized effect sizes enhances interpretability beyond statistical significance alone. However, limitations include the single-institution context and relatively small sample size, which may limit the generalizability of the findings; the short-term outcome assessment without evaluation of long-term retention or transfer to clinical environments; and potential residual period effects inherent to crossover designs, including carryover effects, which cannot be fully excluded despite implementation of a 60-minute washout interval to reduce immediate recall and tool-specific priming. Furthermore, the quality of large language model outputs is dependent on prompt formulation and model version, which may affect reproducibility over time. Future studies should investigate longitudinal retention, application in clinical-year learners, and structured artificial intelligence literacy curricula designed to optimize safe and effective integration.

Conclusion

Both artificial intelligence–assisted and conventional search strategies improved clinical reasoning performance, supporting their complementary roles within a structured learning environment rather than the superiority of one approach. The integration of artificial intelligence into medical education should be accompanied by appropriate faculty oversight and emphasis on verification of information to mitigate potential risks, including inaccurate or hallucinated outputs.