A Theoretical Framework for Shared Reasoning Fragility in Clinician-Chatbot Interactions Through the Example of Antibiotic Prescribing

Introduction

ChatGPT 3.5 (OpenAI), a general-purpose large language model (LLM)-based tool that allows users to chat with a non-human counterpart in an unprecedentedly fluent human language, was released to the public on 30 November 2022.¹ This release marked the beginning of broader public discovery and use of both free and subscription versions of ChatGPT 3.5 and subsequent versions (up to 5.2 at the time of writing), as well as other LLM-based chatbots such as Gemini (Google) and Claude (Anthropic), among others.

Following this development, for example, the number of weekly active ChatGPT users on consumer plans (Free, Plus, Pro) has grown exponentially over the past two years, approaching 800 million in September 2025.² The Gemini app has similarly been reported to exceed 650 million monthly users, while Claude was reported in 2025 to have 18.9 million monthly active website users and 2.9 million monthly active app users, with substantial growth between early and late 2024.^3,4 Given these figures and the ease of access such tools via smartphones and computers, clinicians are likely among the millions of users discussing medical matters with frontier chatbots daily. This does not necessarily imply inappropriate chatbot use or breaches of patient privacy, as simulated cases and general medical and research questions can be discussed without disclosing identifiable patient data or violating current laws or regulations. Consistently, a growing number of research has examined the potential uses and performance of chatbots, both independently and as decision-support tools for clinicians, in diagnostic and therapeutic tasks, including antibiotic prescribing.^5–15

Using chatbots for these purposes creates a novel context for clinical reasoning, in which thinking - from the consideration of symptoms and signs through diagnosis and treatment - is no longer exclusively human but is dynamically shared with a machine. This interaction may introduce instability between chatbot outputs, generated through opaque associative patterns structurally distinct from classical human reasoning, and clinicians’ own reasoning processes, giving rise to an insufficiently characterized mode of decision-making. We have previously introduced and briefly defined this interaction as shared reasoning fragility.¹⁶ The present perspective develops this concept into a more explicit theoretical framework for healthcare, using antibiotic prescribing as an illustrative case. Notably, it is possible that the framework may also apply to domain-specific LLMs; however, we focus here on general-purpose LLMs, as they likely represent the most widely used tools at present.

The Black Box Dilemma: The Nuanced Different Angle of Human-Chatbot Interactions vs Classification Tasks

One of the most debated issues in applying artificial intelligence (AI) to healthcare is the black box nature of some predictive algorithms. This refers to the partial or complete inscrutability of the internal calculations and learned associations through which some models use data to predict clinical events, such as diagnosing a particular infection or selecting empirical antibiotic therapy in septic shock.^17,18 This concern also applies to general-purpose LLMs, which underlie contemporary chatbot applications, because they similarly rely on “black box” architectures to predict the next token, meaning a word or part of a word, in response to a user query.^7,19–22 For example, as shown in Figure 1, such a query may concern whether an antibiotic should be administered in a given clinical scenario and, if so, which one(s).

Figure 1 Workflow of the interaction between clinicians and LLMs-based chatbots. Abbreviations: NLP, natural language processing; LLMs, large language models.

For non-LLMs black box predictive algorithms used in classification tasks, such as predicting the presence of infection due to a multidrug-resistant organism, the debate has largely focused on the balance between explainability and external validation. Explainability relates to the field of explainable AI (XAI), in which a black box model’s prediction is “explained” by a more interpretable surrogate model.^23–27 Conceptually, such models are typically less accurate than the underlying black box; otherwise, they would replace it. This creates a well-recognized tension: explanations may help identify omissions, misleading associations, or biases, but, as approximations of a more complex process, they may also be unreliable. For this reason, some experts prioritize external validation over explainability.²⁸ However, this position entails accepting at least partial non-explainability: some reasons for a given prediction remain inaccessible in exchange for improved performance. Whether, and to what extent, such a compromise is acceptable remains debated and likely depends on the specific clinical task. Importantly, accepting reduced explainability also raises ethical concerns, particularly regarding informed consent (Table 1).^29–35

Table 1 Key Ethical Issues Related to Informed Consent in AI-Supported Decision-Making

While some of these considerations also apply to next-token prediction by general-purpose LLMs, we argue that chatbot outputs raise distinct conceptual issues that are central to the present framework. First, in our view, the explanatory techniques described above are not conceptually applicable to chatbot responses. Providing references and sources can help ground claims by identifying their evidentiary basis, but it does not technically explain why the chatbot generated a particular response from that evidence. Similarly, prompting the chatbot to generate chain-of-thought reasoning may improve performance and reduce hallucinations,^36,37 but it does not necessarily explain why a given output was produced. Chain-of-thought reasoning may offer an initial mechanistic window into how LLMs operate by modulating task-dependent node activation,³⁸ but what these activated nodes do - including which hidden features they construct or how semantic and meta-semantic information is integrated - remains largely unknown. Furthermore, the reliability of chain-of-thought reasoning must be interpreted in light of the broader instability of LLM outputs. Studies on persistent instability in LLMs personality measurements suggest that model responses are highly sensitive to prompt phrasing, reasoning style, and conversational context.³⁹ This raises concerns about whether chain-of-thought outputs can be interpreted as stable reflections of underlying reasoning. In parallel, recent work on chain-of-thought monitorability recognizes that reasoning traces may make model behaviour partly observable, but their interpretation remains context-dependent and may change as models are updated or optimized differently.⁴⁰

A common objection is that clinicians’ reasoning is also partly opaque. The proverbial “clinical eye” integrates experience, knowledge, and intuition in ways that are often difficult to explain or reproduce. This argument has been used to downplay the black box dilemma and to prioritize correctness, safety, usefulness, and validation of outputs, over explanation of reasoning, regardless of human or chatbot origin.⁴¹ Nonetheless, we argue that this position implicitly assumes that human and chatbot reasoning are interchangeable in how they influence behavior and subsequent reasoning. This assumption is not necessarily justified, as illustrated below through a narrative example.

In Kazuo Ishiguro’s novel “Klara and the Sun”, Klara is a robot companion who supports Josie, a girl with a life-threatening disease.⁴² Although perceived as highly intelligent and capable of sophisticated reasoning, Klara comes to believe that communicating with the sun could lead to Josie’s cure. By abductive reasoning (eg., inferring the most plausible explanation from observed patterns),⁴³ most humans would consider such a causal relationship scientifically implausible. Human reasoning, despite its partial opacity, is grounded in broadly shared evolutionary and cognitive frameworks. No comparable alignment can be assumed for machines, including chatbots. Thus, even when chatbot outputs appear correct and scientifically sound, they may arise from associations that are epistemically misaligned with human expectations. In complex settings such as healthcare, such misalignments could potentially influence decision-making if unrecognized. While fictional, this example illustrates a key conceptual point: when interacting with chatbots, responses appear fluent, authoritative and coherent, which may obscure the possibility that their underlying reasoning differs substantially from human reasoning and may include weakly grounded or implausible associations. Importantly, in this context, shared reasoning fragility should be clearly distinguished from related concepts. First, it differs from automation bias, which describes overreliance on automated systems; shared reasoning fragility may arise even when clinicians remain actively engaged and do not defer decisions. Second, it is distinct from hallucinations, which refer to factually incorrect outputs; fragility may occur even when outputs are factually correct but derived through reasoning pathways that do not align with clinician expectations. Third, while related to epistemic opacity, the inaccessibility of the internal processes underlying model outputs, shared reasoning fragility does not primarily concern the opacity itself, but rather how such opacity interacts with human reasoning during ongoing dialogue. In this sense, epistemic opacity can be understood as a precondition, whereas shared reasoning fragility denotes a potential consequence of the interaction between opaque machine processes and human cognition.

Finally, A second key distinction concerns the difference between chatbot-based systems and traditional classification-based clinical decision support systems (CDSSs). In conventional CDSSs settings, clinicians typically complete most reasoning before consulting the system at a final decision point. By contrast, chatbots may be used throughout the entire reasoning process, from initial hypothesis generation to diagnostic work-up, interpretation of results, and treatment selection. This enables a form of continuous co-reasoning, in which clinicians are repeatedly exposed to interactions between two fundamentally different modes of association and inference. This interaction is both unprecedented and pervasive. Given the novelty of this interaction and the limited experience in managing it, we define shared reasoning fragility as the potential instability arising from continuous co-reasoning and from the current limited ability to anticipate and manage its effects. In this context, “fragility” does not imply an inherently negative phenomenon, but rather highlights the need for systematic investigation to understand and appropriately manage these interactions.

In the following section, we present descriptive examples from the literature suggesting the possible presence of shared reasoning fragility in chatbot-assisted antibiotic prescribing. These examples may inform the design of empirical studies and support conceptual extension to other areas of healthcare.

Shared Reasoning Fragility in Antibiotic Prescribing

Several studies have investigated the use of general-purpose LLM-based chatbots as CDSSs for antibiotic prescribing, primarily focusing on performance metrics such as accuracy and the presence of incorrect or potentially harmful responses.^5,6,9–15 These performance measures are not the focus of this work, although we share the impression that technical advances, ideally supported by interaction with medical professionals during development, are likely to enhance performance. Instead, we consider selected illustrative examples concerning the nature and potential cognitive impact of suboptimal responses within the framework of shared reasoning fragility. Importantly, these examples are not intended as empirical evidence but as conceptual prompts suggesting that such fragility may exist and warrants systematic investigation to confirm or refute it. No quantitative analysis was performed in the present theoretical paper, and dedicated empirical studies are required before drawing any clinical implications.

Among the largest studies on the topic, De Vito et al assessed the performance of 14 LLM-based tools for recommending antibiotic choice, dosage, and duration using simulated infectious diseases vignettes.¹⁵ While methodologically robust, the study provides limited insight into shared reasoning fragility. The models were queried at a late stage of the prescribing process, with diagnosis, infection site, causative agent, and susceptibility profile already defined. In this respect, the scenario resembles traditional CDSSs, where systems are applied to a final classification task after most clinical reasoning has been completed. Although some variability and misalignment with guidelines were observed, the study reported only on structured outputs (eg., drug, dose, duration) rather than full natural language outputs. This limits the ability to examine how chatbot responses might interact with clinician reasoning, particularly how arguments are framed and how convincingly suboptimal recommendations are presented.

Smaller studies provide more informative qualitative insights into shared reasoning. For example, Maillard et al evaluated ChatGPT 4 by requesting comprehensive management plans from a positive blood culture, including work-up, antibiotic therapy, source control, and follow-up.¹¹ The chatbot generated detailed and well-structured responses across 44 anonymised real cases. Only one full conversation was reported, involving methicillin-susceptible Staphylococcus aureus (MSSA) bloodstream infection (BSI). Nonetheless, some illustrative aspects can be noted from this single example. The clinical presentation included features compatible with pneumonia (polypnea and right basal crepitations), but also inconsistent elements (absence of dyspnoea and cough), making pneumonia uncertain. Clinical features were also compatible with skin and soft tissue infection (erythematous plaque at an ulcerated forearm lesion). The chatbot appropriately identified at least one of these potential sources of BSI. Within its management plan, it recommended continuing amoxicillin–clavulanate (initiated empirically for suspected pneumonia) based on its “good coverage for skin and lung infections” and proposed a reasonable diagnostic work-up (eg., exclusion of endocarditis) with clinical and laboratory monitoring. While this reasoning may appear plausible, amoxicillin/clavulanate is not an optimal choice for treating MSSA BSI.^44,45 For infectious diseases specialists, this limitation is evident; however, non-specialists - who may benefit most from CDSS support - may be more susceptible to such framing based on shared reasoning. Unlike traditional CDSSs, the chatbot provides fluent, contextually coherent justifications that, during shared reasoning, may subtly shift clinicians’ attention away from critical evaluative questions, such as the appropriateness of the antibiotic for the specific pathogen.

Notably, similar risks may arise even from correct elements of a response. For example, guideline-concordant diagnostic suggestions may convey a sense of completeness when expressed in natural language aligned with clinicians’ reasoning. In the same example, pneumonia was suspected but not confirmed, yet a chest X-ray was not included in the proposed diagnostic work-up, and this omission was not highlighted in the study.¹¹ This illustrates how gaps may be less salient within otherwise coherent responses. A comparable observation emerged from our own interaction with ChatGPT 4o in a fictional case of a 70-year-old man with ventilator-associated pneumonia and septic shock in a ward endemic for Klebsiella pneumoniae carbapenemase (KPC)-producing K. pneumoniae.⁴⁶ Although the proposed plan broadly aligned with guidelines, a basic safety check (eg., assessment of antibiotic allergies) was initially overlooked. Notably, this omission was not immediately apparent, despite its routine importance in standard clinical reasoning. At a conceptual level, we hypothesize that this may reflect shared reasoning fragility, whereby interaction with chatbot outputs subtly reshapes the structure of clinical reasoning.

These observations remain illustrative and do not support definitive conclusions. However, they suggest the plausibility of shared reasoning fragility and the need for targeted empirical research to determine its presence, extent, and clinical implications. Our concept partially resonates with “epistemia”, as recently described by Quattrociocchi et al, whereby linguistic plausibility becomes a structural substitute for epistemic evaluation.⁴⁷ In our view, while different, the two concepts are complementary: epistemia describes the epistemic substitution mechanism, whereas shared reasoning fragility focuses on the interactional conditions under which such mechanisms may influence clinical reasoning.

Finally, traditional clinical reasoning typically involves iterative selection of a manageable subset of hypotheses to optimize patient outcomes.^48,49 From this perspective, sharing reasoning with chatbots may either improve or undermine patient benefit by subtly shifting hypothesis selection throughout decision-making phases (eg., through non-consideration of relevant tests or safety checks). In our view, this further underscores the need for systematic investigation of shared reasoning processes and for strategies to mitigate potential fragility, alongside broader efforts to address risks in human-AI interaction, including amplification of cognitive biases or the influence of implicit value prioritization in AI-generated responses.^50–52

Conclusion

In this article, we provided a theoretical framework for shared reasoning fragility, illustrated through examples related to antibiotic prescribing (Figure 2). These examples are strictly illustrative and do not constitute empirical evidence. These baseline, theoretical considerations are intended to inform future quantitative and qualitative research, including vignette-based experiments, think-aloud studies, simulation studies, and assessments of whether chatbot interaction affects diagnostic completeness, hypothesis generation, or omission rates. Further research should also examine how shared reasoning fragility may manifest in clinical reasoning, supervision, trust calibration, and moral experience. More broadly, our framework highlights the need to move beyond accuracy-based evaluations of chatbot outputs toward a more comprehensive understanding of how these systems interact with, and potentially reshape, clinical reasoning processes. It may also serve to identify mitigating strategies, such as structured prompting, human oversight, antimicrobial stewardship workflows, and institutional safeguards. Clarifying these dimensions will be essential for the safe, effective, and ethically grounded integration of chatbots into clinical practice.

Figure 2 Theoretical framework for shared reasoning fragility through the example of antibiotic prescribing. Created in BioRender. Giacobbe, D.R. (2026) https://BioRender.com/wofjwey. Abbreviations: AI, artificial intelligence; CDSSs, clinical decision support systems; LLM, large language model.