On-Premises AI-Tool for Generating Nursing Care Summaries: A Phased-Implementation Study in Japan

Introduction

In healthcare settings, documentation demands have become a significant burden for nurses.¹ A survey showed that over 20% of nurses’ working hours are devoted to documentation,² reducing the time available for direct patient care. Nursing summaries are essential for ensuring continuity of patient care but require substantial time and effort. Nurses must concisely and accurately summarize changes in patients’ conditions, the interventions implemented, and their outcomes. These tasks require clinical judgment and strong documentation skills. With an aging population and growing healthcare demands, improving workflow efficiency has become increasingly urgent.

Artificial Intelligence (AI) applications in medicine now span image diagnosis support,³ drug discovery,⁴ and surgical assistance.^5,6 Large language models (LLMs) and generative AI tools for medical documentation are also advancing rapidly globally.^7–9 Recent applications of LLMs in medical text generation have demonstrated promising results in creating discharge summaries and clinical note assistance. It has been suggested that AI assistance in schedule coordination and documentation tasks may enable more efficient nursing workflows.¹⁰ However, challenges remain in nursing-specific documentation due to the complex, continuous nature of nursing care and the need for integration with nursing care plans and subjective, objective, assessment, plan (SOAP) format documentation.^11,12 Additionally, an on-premises deployment approach is critical for maintaining data security and regulatory compliance in healthcare settings.

This study details the phased development and implementation of an on-premises LLM-based nursing summary system and validates its generated documents and effects.

Materials and Methods

This study was conducted at the Yuai-Kai Foundation and Oda Hospital, a core regional hospital in Japan. The facility implemented an electronic health record (EHR) system, where nurses routinely created documentation and summaries. The development of the AI-based nursing summary creation system began in December 2023, following a phased approach from prototype (Ver. 0.1) to production (Ver. 1.0). The project adopted an iterative development model, well-suited for implementing novel, untested mechanisms. The AI system was integrated into the existing EHR system (CSI Co., Ltd.) through application programming interface (API) connections, allowing direct access to nursing progress records and care plans while maintaining data security within the hospital’s network infrastructure. The system utilized a proprietary LLM developed by OPTiM Corporation, based on transformer architecture with optimized prompts for Japanese medical terminology and nursing documentation patterns.

Nursing summaries were created for randomly selected hospitalized patients either by nurses during routine clinical practice or by AI systems in development. Different patient cases were assigned to either AI-assisted or manual documentation groups to avoid overlap. The nurses who conducted the verification were nurse managers with extensive experience and skills. Ver. 0.2 and 0.3 of the AI-based summary system were used for these cases, and validation was conducted for each version, respectively. Versions 0 and 0.1 were not subjected to verification.

Patient data—including admission length of stay, AI-generated nursing summaries, and the nurses involved—were obtained from medical charts. We recorded the time nurses spent creating summaries manually and revising AI-generated ones. The primary endpoint was the time required for nursing summary creation. Total AI processing time plus revision time was compared with manual summary creation time using the Mann–Whitney U-test. Each nurse measured and recorded the time for task completion in minutes using a personal timepiece. Additionally, five charge nurses evaluated unmodified AI-generated summaries across five criteria—(1) factual accuracy, (2) clarity and readability, (3) appropriateness, (4) comprehensiveness, and (5) overall satisfaction—using a five-point Likert scale.¹³ Scores are reported as medians with quartiles. This study was approved by the facility’s ethics committee (Approval No. 20250313). Regarding safe medical data handling, we adopted an on-premises environment, similar to stringent security measures found in US HIPAA-compliant systems.¹⁴ Analyses were performed using IBM SPSS Ver. 25 (IBM Corp., Armonk, NY, USA).

This study was conducted in accordance with the Declaration of Helsinki and the Ethical Guidelines for Medical and Health Research Involving Human Subjects established by the Ministry of Health, Labour and Welfare and the Ministry of Education, Culture, Sports, Science and Technology of Japan. This study was approved by the Research Ethics Committee of Yuai-Kai Foundation and Oda Hospital (Approval No. 20250313). We disclosed the research information on the hospital’s website and allowed patients to opt out of the study.

In this study, a conflict of interest exists because employees of OPTiM Corporation participated as authors and the developed system was subsequently supplied as a product to Oda Hospital. To ensure objectivity, authors from multiple independent medical institutions collaborated on the research, with the corresponding author, who has no direct contractual relationship with the vendor, overseeing the entire process. Furthermore, during the development process, we adopted a “clinician-in-the-loop” approach that actively incorporated feedback from nurses in actual clinical settings and prioritized the clinical utility of the system to minimize bias from commercial interests.

Results

System Improvement Process (Figure 1)

Prototype Development Stage (Ver. 0 to 0.1)

Prototype development began in December 2023. In the early development stage, assigned nurses briefed technicians (OPTiM Corporation) on the purpose of nursing summaries, progress records, vital sign charts, nursing care plans, and basic patient admission databases. Discussions were held regarding the provision of patient chart information from the chart vendor (CSI Co., Ltd.) to technicians, reporting to the ethics committee, and obtaining patient consent, and decisions were made to submit existing nursing summary formats. By January 2024, CSI provided information from 40 patients who had consented to external use of their data, enabling technicians to generate AI-driven summaries from nursing progress records. During the initial stage, the system was implemented by having LLMs read recent hospitalization records to output summaries. However, this yielded fragmented and impractical documents such as “orthostatic hypotension present, caution during movement” and “infection prevention behaviors are maintained.” These lacked important context and had no clear linkage with corresponding nursing care plans, making them insufficient for understanding patient issues.

Figure 1 System improvement process. Abbreviations: LLM, Large language models; SOAP, Subjective, objective, assessment plan; HER, Electronic health records.

In the final prototype stage (March 2024), a survey of 50 cases revealed that nurses spent a median time of 16 minutes creating summaries without using AI. Subsequently, implementation of Ver. 0.1 began at the end of March 2024, with a system that directly extracted and summarized nursing records from EHRs. However, the AI-generated summaries lacked clarity regarding which nursing care plans corresponded to the implemented records. They were sparse in content, failed to capture the essence of observational records or progress, and were evaluated as “not at a level that could be called a summary,” indicating very low practicality.

In April 2024, nurses revised 50 summaries created by the AI system and provided detailed feedback. This feedback identified challenges, including linking nursing care plans with nursing records, understanding medical terminology, and LLM execution times. Initially, the system attempted summarization using nursing records alone; however, the actual document structure showed that these records were connected to nursing care plans. Furthermore, there was a technical challenge: the initial nursing record-only summarization method required enormous input text, necessitating multiple LLM inference executions (eg, daily summarization of nursing records followed by weekly summarization of those results). This required substantial processing time, which was problematic.

Transition to Ver. 0.2 and Improvements (Ver. 0.1 to 0.2)

To address the challenges of Ver. 0.1, we transitioned to Ver. 0.2. Generally, nursing care plans are recorded with details such as observation, direct care, and education, divided into items such as #1, #2, and #3, and then documented in a SOAP format for each. After examining the feedback received in April, we concluded that a method where LLMs correctly interpreted these recording patterns and documented them by nursing problems would be beneficial. Data acquisition methods and prompt tuning were implemented. This method optimized the character count per execution, thus shortening overall LLM execution time.

However, a new challenge emerged: more care content outlined in the nursing care plans was output than the actual care content provided, and the documentation of implementation records was weak. Therefore, attempts were made to incorporate summaries of nursing content into the nursing care plan items, and the revised status of the prototype version became an agenda item.

Based on these challenges, Ver. 0.2 was released on July 19, 2024. At that time, feedback indicated that summaries focused on care content, making hospitalization progress unclear. We then proposed summarizing weekly nursing care plan evaluations. However, the content became centered on summarizing each item of “nursing care plan,” “progress,” and “nursing care plan evaluation,” while the patient’s condition became difficult to understand. Therefore, Oda Hospital provided OPTiM with materials regarding the SOAP definitions used in the summaries (Supplement 1), and the SOAP definition text was reflected as part of the prompt.

Ver. 0.3 Development and Evaluation (Ver. 0.2 to 0.3)

Based on the challenges faced by Ver. 0.2 and the newly provided SOAP definitions, Ver. 0.3 was released in late August 2024 to strengthen the interpretation of SOAP definitions and optimize the balance of nursing care plans, implementation, and progress. In Ver. 0.3, adjustments were made to emphasize the implementation content to resolve the challenges of the previous version.

In September 2024, we confirmed AI summary usage and usability. These summaries focused on nursing care plan goal achievement and rationales. However, clinical practice requires including patient condition changes, posing challenges in displaying nursing diagnosis names per number and managing lengthy content. At the end of October 2024, workload surveys indicated nurses spent a median time of 10 minutes creating and modifying summaries using the AI system.

Production Version (Ver. 1.0) Release (January 2025)

Following the improvements and evaluations of Ver. 0.3, the official version, Ver. 1.0, was released in January 2025. This final version newly implemented extended functions for medical information providers, significant user interface improvements, and complete integration with EHR system icons, in addition to previous nursing summary functions. Through these enhancements, the system was launched as a formal operation, seamlessly integrating into daily nursing operations.

Effectiveness Comparison with and without AI Nursing Summary Creation System Use

When validating the nursing summaries created by AI, the total scores for the five evaluation items in Ver. 0.2 and Ver. 0.3 showed no difference (Table 1). For individual items, items (1), (3), (4), and (5) demonstrated no changes in median scores between Ver. 0.2 and Ver. 0.3, whereas only item (2), clarity and readability, showed a median of 3 for Ver. 0.2 versus 3.5 for Ver. 0.3. When comparing the total scores for the five items using the Mann–Whitney U-test, no significant difference was found (p = 0.571).

Table 1 Nurse Evaluation of AI-Driven Nursing Summary System Ver. 0.2 and Ver. 0.3

Additionally, using AI significantly reduced the median time for nurses to create and modify summaries to 10 minutes, compared to 16 minutes without AI (p < 0.001, Mann–Whitney U-test; Table 2). The median time for AI-generated nursing summary creation alone was one minute (interquartile range, 1–2 minutes). While the median length of stay for patients in the AI-using group was 22 days (mean: 25.9 days) and 25 days for the non-AI group (mean: 30 days), this difference was not statistically significant (p = 0.432, Mann–Whitney U-test).

Table 2 Median Time and Interquartile Range Comparison for Nursing Summary Creation

Discussion

After approximately one year of phased improvements since prototype development began in December 2023, we successfully overcame initial challenges and completed a practical AI nursing summary system that specifically contributed to nurse workflow efficiency. The quality of summaries created using the AI system by nurses in Ver. 0.2 and Ver. 0.3 showed no significant differences. We anticipate that this development process will serve as a model for future medical AI development, demonstrating the successful integration of practical knowledge with AI technology.

In this study, development of the AI nursing summary system reduced nursing record creation time by approximately 40% (from 16 to 10 minutes). While international studies have reported more dramatic time reductions (up to 90%) in medical documentation,^15,16 direct comparison is challenging due to fundamental methodological differences. Previous studies primarily measured AI generation time alone, without accounting for essential human verification and modification processes that are critical in clinical practice. Our study implemented a comprehensive “human-in-the-loop” approach, measuring total time including AI generation (one minute) plus nurse verification/modification time (eight minutes). Even with this rigorous human oversight requirement, which is essential for patient safety, we achieved a clinically meaningful 40% time reduction.

The phased development approach adopted in this study (evolution from Ver. 0.1 to 1.0) implemented the iterative development model recommended for healthcare AI system development.¹⁷ In particular, the decision to actively incorporate feedback from clinical field nurses aligns with the “clinician-in-the-loop” approach emphasized in US medical AI development—an approach in which healthcare professionals participate from the design stage to enhance clinical utility.¹⁸ This study’s improvement in interpreting nursing care plan number recording patterns offers a notable solution to the technical challenge of understanding medical document-specific structures. While multiple studies emphasize integrating medical domain knowledge for accurate AI medical document creation,^19,20 this study demonstrates its importance in the specific case of understanding the SOAP format and nursing care plan structures.

Regarding quality assessment, our study demonstrated limited improvement between Ver. 0.2 and Ver. 0.3, with only clarity and readability showing enhancement (median 3.0 to 3.5). However, this finding requires careful interpretation within the developmental context. The transition from Ver. 0.1 to Ver. 0.2 showed dramatic qualitative improvements in content coherence and clinical relevance, although these were not systematically evaluated using our standardized assessment criteria. This suggests that Ver. 0.2 may have achieved a functional quality threshold that left limited room for further improvement in subsequent iterations.

From a clinical effectiveness perspective, achieving time reduction is extremely useful. Approaches such as Retrieval-Augmented Generation contribute to improving the accuracy of medical AI,²¹ and appropriately linking nursing care plans with nursing records likely enables more clinically valuable summaries. Additionally, insights regarding the balance of nursing summary content represent important achievements. Nursing summaries are not mere records; they are communication tools to ensure the continuity of patient care. From this perspective, comprehensive information provision—including changes in the patient’s condition—is an essential element for future care providers to understand the patient’s overall condition. The results demonstrate the importance of maintaining “comprehensive and balanced information” in AI-generated nursing summaries, an essential value of these documents. This provides a crucial guideline for balancing the often-conflicting goals of AI efficiency and clinical value.

The study’s limitations include generalizability constraints due to single-facility implementation, data privacy and security challenges, risks of excessive AI dependence, and the need for standardized evaluation indicators. In particular, validating adaptability in facilities with different EHR systems or documentation practices remains a crucial future challenge. Regarding AI-generated content confirmation and modification, it is essential to clarify AI’s role as a tool supporting clinical judgment, necessitating the establishment of effective collaboration models between AI systems and healthcare professionals. Regarding evaluation, although nurses with a certain level of experience and skills conducted the assessments, we were unable to establish unified evaluation criteria through pre-training or assess inter-rater reliability. Blinding was not feasible during the evaluation of nursing summaries. Sample size was not determined in advance, and the number of samples represents what could be collected during the development phase. Potential confounding factors were not evaluated. A critical limitation of our quality evaluation is the absence of direct comparison between AI-generated and human-created summaries for identical patients, making absolute quality assessment challenging. Future studies should incorporate head-to-head comparisons with human-generated summaries and develop objective quality metrics that correlate with patient care outcomes.

Conclusion

This study achieved an approximate 40% reduction in nursing record creation time and produced clinically acceptable summary documents through the phased development and implementation of an AI-driven nursing summary system. This success stemmed from integrating iterative development with medical professional collaboration and AI technology, which aided in understanding nursing record-specific structures. As digital healthcare transformation advances, this study’s phased, collaborative development approach provides crucial guidelines for future medical AI development, serving as a model for building AI systems that meet the needs of complex medical settings. Although this is a single-center study, the phased approach employed in this research suggests the potential generalizability of the system across diverse clinical settings. Furthermore, this work may serve as an important starting point for developing a collaborative human–AI model in which AI effectively supports rather than replaces human clinical judgment. Further validation is needed to improve the quality of this system.