Deep Learning Classification of Rheumatoid Arthritis in Hand Radiographs Interpretability Insights and Web Application

Introduction

Rheumatoid Arthritis (RA) is a chronic systemic autoimmune disease characterized by synovitis as its pathological basis, with predominant clinical manifestations of pain, swelling, and morning stiffness in small joints, particularly the proximal interphalangeal joints, metacarpophalangeal joints, wrists, and knees.¹ Delayed diagnosis and intervention in RA can result in progressive joint cartilage and bone destruction, leading to deformities and functional impairment, thereby adversely affecting long-term prognosis and quality of life.² Medical imaging plays an indispensable role in rheumatoid arthritis (RA) diagnosis, enabling assessment of synovitis, cartilage degradation, and bone erosion. Despite advancements in and accessibility of various imaging modalities, including ultrasound, magnetic resonance imaging (MRI), and conventional radiography, plain radiography remains essential for RA evaluation.^3,4 Particularly, conventional radiography of the hands and wrists remains the most widely utilized imaging modality for assessing structural joint damage in RA, owing to its operational simplicity, cost-effectiveness, and widespread availability.⁵ RA diagnosis depends on synthesizing multiple parameters: clinician expertise, patient symptomatology, imaging findings, and laboratory data. This integrative process is inherently complex and time-intensive, with diagnostic accuracy vulnerable to subjective interpretation and incomplete data synthesis. Such limitations may result in diagnostic errors or oversight. Consequently, developing efficient and objective strategies, particularly those based on the widely available hand radiographs, to enhance RA diagnostic accuracy represents a crucial clinical priority.

The limitations of conventional diagnosis have spurred interest in artificial intelligence (AI)-assisted solutions. Deep learning, particularly convolutional neural networks (CNNs), has emerged as a powerful tool for automating image analysis tasks. CNNs excel at extracting discriminative features from medical images, enabling objective and reproducible disease classification and prediction.^6–8 This potential has been increasingly explored in the context of RA. Several pioneering studies have demonstrated the efficacy of deep learning models in automating the detection of radiographic joint damage and predicting RA progression from hand radiographs.^9–13 For instance, Peng et al conducted a comprehensive evaluation of five mainstream CNN architectures, demonstrating that models like GoogLeNet could achieve an exceptional AUC of 97.80% and a sensitivity of 100.0% for RA recognition from hand radiographs.¹⁰ Beyond utilizing existing architectures, Kesavapillai et al developed a specialized CNN model (RA-XTNet), which outperformed standard pre-trained networks, achieving a classification accuracy of 90% for hand radiographs.¹¹ These studies underscore the feasibility and evolving efficacy of using deep learning to augment RA diagnosis based on hand radiographs.

However, the adoption of “black-box” deep learning models in clinical practice is often hindered by their lack of interpretability. Clinicians need to understand the rationale behind a model’s prediction to trust and act upon it. Techniques like Grad-CAM (Gradient-weighted Class Activation Mapping) address this by generating visual explanations that highlight the image regions most influential to the model’s decision.^14–17 While Grad-CAM has proven invaluable in explaining CNN-based models in other radiological domains, such as identifying abnormalities in wrist and elbow radiographs,¹⁸ localizing rib fractures in computed tomography (CT) scans,¹⁹ and interpreting breast cancer detection in mammograms,²⁰ its application specifically within RA research, particularly for interpreting models trained on hand radiographs, remains relatively unexplored. Similarly, SHapley Additive exPlanations (SHAP) analysis, which quantifies feature contribution and has gained traction in interpreting complex machine learning models in healthcare,²¹ has seen limited use in this niche. Moreover, the entire process from model construction to clinical application is of great significance. Therefore, there is an urgent need to develop an accurate RA diagnostic model, integrate interpretable methods, and build a work platform based on this model to validate its clinical knowledge-based decision-making process.

Accordingly, this study investigates the feasibility of applying VGG networks to radiographic RA diagnosis, intending to facilitate accurate and efficient clinical detection. This study has three characteristics: First, a CNN-based automatic diagnosis model is proposed to achieve accurate and fast diagnosis, including early RA cases. Second, to enhance the model’s reliability and interpretability, Grad CAM and Shapley value analysis were used for comprehensive visualization to demonstrate whether samples from the same category can be effectively clustered. Third, an integrated data model application analysis framework has been established to allow clinical doctors to simply input hand radiographs and immediately obtain RA diagnostic predictions. This streamlined approach not only facilitates clinical implementation but also demonstrates the potential for AI to provide more accessible and efficient services in rheumatology practice and related medical fields.

This study aims to develop and validate a deep learning-based system for automatically distinguishing RA patients from normal adults using hand X-ray images. By integrating Grad CAM for visual localization and SHAP values analysis for feature importance quantification, the “black box” of this model attempts to be revealed. By building an online APP based on Streamlit, it provides transparent and reliable predictive explanations for clinical doctors to ensure that the model’s decisions are based on radiological-related features to promote clinical adoption.

Method

Study Population

A retrospective analysis was conducted on 1655 hand X-ray images collected from patients with rheumatoid arthritis (RA) and healthy controls at Yichang Central People’s Hospital and Yichang Second People’s Hospital between January 2017 and April 2024. All images were randomized and divided into training and testing sets by institution. The planned work has been approved by the Ethics Committee (Human Research) of the First Clinical Medical College of China Three Gorges University and the Second affiliated hospital of China Three Gorges University (Approval No.2024–070-01 and 202433). The study is retrospective in design and uses only anonymized clinical data; patients cannot be contacted; the research is essential for the public interest of medical science; the waiver does not compromise patients’ rights or welfare. All patient data have been anonymized, with identifying information removed, and will be used solely for the purposes of this study. The research strictly adheres to the principles of the Declaration of Helsinki, ensuring data security and protection of patient privacy.

The inclusion criteria comprised: RA patients meeting the 2010 ACR/EULAR classification criteria,^22,23 with completed hand radiographs and complete clinical data; and healthy controls exhibiting neither joint pain nor swelling persisting beyond one week; while exclusion criteria included: RA patients with incomplete or suboptimal quality radiographs; and non-RA individuals lacking wrist/hand joint pain, persistent swelling (>1 week), or history of trauma.

Equipment and Methods

All hand radiographs were obtained by certified radiology technicians using a Wandong VX3733 X-ray scanner under standardized protocols with 60 kV tube voltage, 3 mAs exposure dose, and 100 cm film focus distance, capturing both posteroanterior and lateral views in JPEG format.

X-Ray Image Model Construction

Data Collection

Hand radiographs of RA patients and healthy controls were collected at a 1:1 ratio. To validate model generalizability, we selected 50 normal and 50 RA hand radiographs from these two sources as an independent test set (Figure 1).

Figure 1 Schematic diagram of the training and testing set construction framework. Abbreviations: RA, Rheumatoid Arthritis; Normal, healthy controls.

Image Preprocessing

All collected images underwent standardized preprocessing, including resizing to 256×256 pixels, followed by center-cropping to 224×224 pixels, vectorization, and sym8 wavelet transformation, with global normalization to maintain optimal pixel resolution while preserving diagnostically relevant information. To address common radiographic artifacts (including variable data types, excessive unfilled regions, abnormal angulations, and image darkening), we implemented quality enhancement techniques through strategic cropping and brightness adjustment. The standardized processing pipeline consisted of: conversion to PyTorch tensors, pixel value normalization to [0,1], and channel-wise standardization using mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] values. Data augmentation employed three strategies: randomized rotation only (±45°), randomized brightness adjustment only (±40%, scaling pixel values 0.6–1.4 with 0–255 clipping), and their combination (±45° rotation followed by ±40% brightness adjustment) (Figure 2).

Figure 2  Image processing pipeline. Data Processing. Resize: Resize all images to 256×256 pixels. CenterCrop: Crop a 224 × 224-pixel center patch from each. ToTensor: Convert images to PyTorch tensors and scale pixel values to [0, 1]. Normalize: Standardize each channel with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. Data Augmentation. Rotation only: rotate within ±45. Brightness only: adjust brightness by ±40% (scaled by np.random.uniform (0.6, 1.4)). Rotation + Brightness: combine the two operations above for cumulative augmentation. “R” indicates the right side.

Model Development

A simplified VGG8 architecture has been implemented, which maintains the iconic design principles of VGG, namely the use of small convolution kernels (3x3) to achieve detailed feature extraction with reduced parameters, ultimately achieving a fully connected layer for binary classification. The architecture consists of: five convolutional blocks containing eight convolutional layers for efficient computation while maintaining accuracy, 2×2 max-pooling layers with stride 2 for spatial reduction, and a classifier comprising a 25,088-dimensional input layer followed by two 512-dimensional hidden layers with ReLU activation and dropout (p=0.5), terminating in a 2-node output layer (RA vs non-RA) with Softmax activation (Figure 3).

Figure 3 Schematic diagram of the model architecture. Abbreviations: RA, Rheumatoid Arthritis; Normal, healthy controls.

Interpretability Analysis

To enhance the transparency of our deep learning model’s decision-making process, we implemented a dual interpretability framework combining: Gradient-weighted Class Activation Mapping (Grad-CAM) for spatial visualization of critical image regions influencing predictions, and SHAP analysis to quantitatively assess feature contributions. This multimodal approach enables both heatmap-based localization of pathological features and mathematical quantification of input feature importance through additive feature attribution methods.

Web Application Development Based on Streamlit Framework

Based on the VGG model, X-ray classification tasks were carried out, and a Python and Streamlit framework was used to build a web app to make the model more convenient and clinically applicable.

Statistical Analysis

Model performance was rigorously evaluated using standard classification metrics calculated from confusion matrices of both training and test sets: accuracy = (TP+TN)/(TP+TN+FP+FN), sensitivity (recall/true positive rate) = TP/(TP+FN), specificity = TN/(TN+FP), precision = TP/(TP+FP), and area under the receiver operating characteristic curve (AUC), where TP=true positives, FP=false positives, TN=true negatives, and FN=false negatives.

Results

Training Set Modeling

The training dataset comprised 796 hand radiographs from healthy individuals and 759 from RA patients (n=1,555), with an additional independent test set of 100 images (Table 1). Following comprehensive data augmentation and preprocessing, the model demonstrated robust performance with AUC values of 0.99 (training) and 0.81 (test), alongside accuracy rates of 0.94 (training) and 0.74 (test) (Figures 4 and 5 and Table 2). These results indicate successful feature extraction from hand radiographs, with strong discriminative capability maintained across both training and validation phases. The model was trained for 100 epochs using an Adam SGD optimizer (learning rate=0.05) to optimize training data fitting (Figure 5C).

Table 1 Data Distribution of Training and Testing Sets

Table 2 Performance Metrics of the Model on Training and Testing Sets

Figure 4 Model learning and test performance curve.

Figure 5  Model evaluation metrics: (A) Training set ROC curve, (B) Training set confusion matrix, (C) Testing set ROC curve, (D) Testing set confusion matrix.

Model Interpretability Analysis

The interpretability analysis combines Grad-CAM (Figure 6A) and Shapley value visualization (Figure 6B) to elucidate the model’s decision-making process. As shown in Figure 6A, the heatmap overlaid on RA patient radiographs revealed that the model primarily focused on joint spaces and bony articulations (red/yellow regions, particularly in the wrist joints) - anatomical areas clinically known to demonstrate characteristic RA changes. Conversely, blue/green regions indicated minimal diagnostic contribution.

Figure 6  Interpretability analysis: (A) Grad-CAM visualization showing regions with significant positive contributions to RA classification (red/yellow areas); (B) SHAP analysis demonstrating feature importance (darker colors indicate higher contribution). Abbreviations: RA, rheumatoid arthritis; NORMAL, healthy controls.

Shapley analysis (Figure 6B) quantitatively confirmed these findings, with red-highlighted areas (potential pathological regions) showing strong positive contributions to “RA” classification, while blue regions (normal bone structures) supported “NORMAL” classification. Notably, the model’s attention patterns demonstrate remarkable concordance with established clinical diagnostic priorities, validating its learning of medically relevant features.

Web Application Development Based on Streamlit Framework

Building upon the VGG-based X-ray classification model, Python and the Streamlit framework were used to develop an accessible web application, establishing a comprehensive pipeline from data processing to clinical implementation. As illustrated in Figure 7, the application features three functional modules: user login interface, sample image gallery, and diagnostic prediction interface with integrated Shapley value visualization. This implementation not only operationalizes our interpretable AI system but also significantly reduces the technical barrier for clinical use. The developed framework represents a complete workflow encompassing data preprocessing, model development, application deployment, and performance evaluation, serving as a replicable template for future medical AI research and translation.

Figure 7  Demonstration of the web-based diagnostic application.

Discussion

In rheumatology, AI models have been predominantly applied to X-ray image analysis, with convolutional neural networks (CNNs) establishing themselves as the leading architecture for AI-based radiological image interpretation.²⁴ While CNNs have demonstrated robust capability in identifying key pathological features of RA in hand radiographs, including joint space narrowing, bone erosions, and osteoporosis.^25–27 The VGG architecture, as a classical CNN variant, offers unique advantages for medical image analysis due to its deep convolutional design and exceptional compatibility with imaging data characteristics.²⁸ This study was designed to evaluate the feasibility and applicability of VGG models for computer-aided diagnosis of RA using hand radiographs. A VGG-based convolutional neural network was developed and systematically validated its performance in RA image classification tasks. The experimental results demonstrated robust diagnostic capability, with area under the curve (AUC) values of 0.99 (training set) and 0.81 (test set), accompanied by accuracy rates of 0.94 and 0.74, respectively. These findings substantiate that the VGG architecture achieves both high diagnostic precision (training AUC 0.99) and clinically meaningful generalizability (test AUC 0.81) in discriminating between normal and abnormal hand radiographs in RA, providing empirical evidence for developing reliable RA diagnostic support systems.

Moreover, Grad CAM and Shapley value decomposition were used for in-depth analysis of the decision-making process of the model. The Grad-CAM technique visualizes gradient-weighted feature maps to intuitively reveal the model’s focus areas on pathological regions (Figure 6A), while Shapley value analysis quantifies the contribution of each input feature to the model output from a game-theoretic perspective (Figure 6B). Notably, the model’s attention patterns demonstrated strong concordance with clinically relevant regions traditionally emphasized by rheumatologists. This dual interpretability approach not only significantly enhanced model transparency and explainability,²⁹ but more importantly, uncovered fundamental alignment between the model’s learning mechanism and clinical diagnostic reasoning. These findings provide crucial theoretical support for the clinical translation of VGG-based radiographic diagnosis in RA management.

Furthermore, the Python ecosystem and Streamlit framework were utilized to develop a lightweight web application that enables efficient deployment and user-friendly access to the RA diagnostic model. This implementation provides a practical technical solution for personalized precision medicine, facilitating multi-center collaboration and data sharing in rheumatology practice. The integrated data-model-application-analysis framework established in this study combines the powerful pattern recognition capabilities of deep learning with advanced interpretability analysis. This synergistic approach not only identifies critical diagnostic regions through explainable AI techniques but also effectively assists clinicians in rapidly extracting diagnostic patterns and clinical insights. While specifically designed for RA assessment, the framework’s modular architecture demonstrates strong potential for rapid adaptation to other musculoskeletal disorders and related disease domains.

Notably, AI applications in radiographic diagnosis of RA have demonstrated promising yet heterogeneous outcomes. Early studies utilizing small datasets reported limited accuracy - for instance, a 2020 CNN-based study achieved 73.33% accuracy using 180 hand radiographs for RA/normal classification.¹³ Subsequent improvements through dataset expansion and architectural optimization yielded 94.46% accuracy, with 0.95 sensitivity and 0.82 specificity from 290 radiographs.³⁰ A pivotal advancement came from Üreten et al,³¹ who employed transfer learning with pretrained VGG-16 to circumvent data limitations, achieving 90.7% accuracy, 92.6% sensitivity, 88.7% specificity, 89.3% precision, and 97% AUC. Ma et al³² further escalated the benchmark using a multicenter dataset (9,964 radiographs from 8,533 patients across 7 hospitals), attaining 0.955 AUC for RA detection against normal controls. Peng et al¹⁰ conducted the first comprehensive architecture comparison (AlexNet / VGG / GoogLeNet / ResNet / EfficientNet), reporting >90% AUC and >98% sensitivity for RA detection, and >77% AUC with >80% sensitivity for staging - albeit limited by a small 240-image dataset. Nevertheless, the majority of studies^10,13,30 rely on relatively small-scale datasets and imbalanced data, which may lead to model overfitting and consequently diminish generalizability in real-world clinical settings. Secondly, while certain models demonstrate exceptional performance metrics,^31,32 their clinical utility and physician trustworthiness are substantially constrained by the absence of integrated application deployment frameworks and comprehensive interpretability analyses. Furthermore, although recent comparative studies of multiple architectures¹⁰ report outstanding performance (eg, GoogLeNet achieving 97.80% AUC and 100.0% sensitivity), these results still require validation through larger and more diverse datasets encompassing varied demographic populations and clinical scenarios to ensure robust external validity.

In this study, the lightweight VGG-8 architecture was chosen for several practical and clinical considerations. Due to its interpretability and compatibility, the order and transparent structure of VGG are very suitable for interpretability techniques such as Grad-CAM and SHAP, which are the core of our research objectives. Additionally, considering clinical deployment goals, lighter models can facilitate easier integration into web-based applications and reduce inference time, which is crucial for providing real-time diagnostic support. Furthermore, as shown in our results, the VGG-8 model achieved strong performance on a medium-sized dataset (training AUC=0.99, testing AUC=0.81), demonstrating its adequacy for the task. Compared to several popular models currently, although EfficientNet-B0 performs better in light to medium models, its network structure is complex, and interpreting the results requires more careful verification, indicating that the model may require longer computation time.³³ In addition, there is a strong dependence on pre-trained weights, which requires fine-tuning during domain transfer, which is not conducive to clinical deployment. Similar to the EfficientNet-B0 model, the Vision Transformer (ViT) model requires longer training time due to its transformer-based architecture.³⁴ Moreover, due to the lack of some key “inductive biases” in computer vision tasks, the Vision Transformer model requires massive amounts of data for learning the model.³⁵ Therefore, the VGG-8 model is more suitable for conducting this study.

In CNN model construction, both data quantity and quality are paramount for predictive performance. Adequate dataset size ensures proper model training, enhances performance,³⁶ mitigates overfitting, and improves robustness.^37,38 Additionally, class imbalance—a prevalent challenge in deep learning where certain classes dominate the training set—can introduce prediction bias toward majority classes and compromise accuracy.^39,40 Addressing this imbalance is critical, as it directly enhances model generalizability.^41,42 To overcome these limitations and address prior shortcomings in RA classification research, this study established a multicenter-derived, large-scale dataset with balanced class distribution. This approach minimizes performance evaluation bias caused by data imbalance, ensuring reliable and generalizable classification outcomes. Notably, to address the diagnostic challenge of early-stage RA cases that often show negative findings on X-ray examinations, this study innovatively incorporated clinically confirmed early RA cases with non-characteristic radiographic changes and developed a diagnostic model demonstrating high sensitivity (Table 2) for detecting subtle pathological features in the training set (0.94 accuracy) while maintaining preliminary generalizability in the test set (0.6 sensitivity and 0.74 accuracy), indicating robust feature extraction capability for clinically challenging cases with ambiguous imaging findings. Future studies should focus on expanding the sample dataset and further investigating these subtle imaging features to systematically identify quantitative imaging biomarkers for early RA progression,⁴³ thereby providing critical evidence for clinical early intervention strategies. Moreover, deep learning models are often regarded as “black boxes”^44,45 due to their inherent lack of interpretability, which significantly limits their clinical adoption in medical image analysis. To address this critical limitation, our study implemented comprehensive interpretability analyses using both Grad-CAM and Shapley value methods. These approaches not only enhanced decision transparency but also revealed remarkable alignment between the model’s focus areas and clinically relevant regions, demonstrating that the model effectively learned medically meaningful patterns consistent with established diagnostic principles.

While AI-driven diagnostic support tools show growing potential in rheumatology, their clinical integration remains limited by issues of accuracy, interpretability, and workflow compatibility.⁴⁶ Existing patient-facing tools, such as symptom checkers (eg, Ada) and referral systems (eg, Rheport), rely primarily on symptomatic input and demonstrate modest diagnostic accuracy (52–63%) even in high-prevalence settings.⁴⁷ In contrast, our Streamlit-based web application leverages radiographic data to provide objective, image-based decision support specifically for RA hand X-ray interpretation.

The developed application offers several distinct advantages: it integrates directly with image-based inputs, supporting standard medical imaging formats and providing real-time predictions with interpretable outputs (SHAP and Grad-CAM visualizations); it achieves clinically relevant performance (test AUC: 0.81, accuracy: 74%) with a focus on structural abnormalities such as joint space narrowing and erosions; and it is designed for seamless integration into clinical workflows via an intuitive, clinician-oriented interface that requires no specialized computational skills.

Unlike general-purpose symptom checkers, our tool provides specialized, reproducible, and transparent decision support for radiographic assessment—a critical component of RA diagnosis and monitoring. This represents a meaningful step toward bridging the gap between AI innovation and clinical practice in rheumatology.

Although this study has achieved promising results, some limitations should be acknowledged. Although the multi-center dataset in this study includes a certain number of hand X-rays from RA patients, its size is still insufficient to meet deep learning standards. There is no better validation method, such as k-fold cross-validation, and it is limited to one region geographically/racially, which may limit the universality of the model.⁴⁸ Second, the current binary classification framework (RA vs normal) has not been fully validated for differential diagnosis against other arthritic conditions like osteoarthritis (OA) or ankylosing spondylitis (AS). Most importantly, consistent with the 2010 ACR/EULAR RA classification criteria,^22,23 radiographic findings alone are insufficient for definitive diagnosis - clinical confirmation requires comprehensive evaluation of: joint involvement patterns, serological abnormalities, acute-phase reactants, and symptom duration. Future web-based systems must therefore integrate multimodal data, including advanced imaging analyses, genetic profiles, clinical histories, biomarker patterns, and treatment response predictions, to achieve clinically reliable diagnoses.

Future studies should further expand the scale of datasets and consider cross-institutional data collection and integration to enhance model robust and generalization capability. At the model optimization and extension level, structural and parameter tuning will be performed based on the VGG architecture to improve its diagnostic efficacy for RA hand radiographs. Specifically, we propose to introduce attention mechanisms⁴⁹ to strengthen the model’s ability to extract and classify complex imaging features, while continuing to develop interpretable AI frameworks such as attention heatmaps and saliency maps. These approaches will systematically improve algorithmic transparency and gradually bridge the gap between model performance and clinical acceptability. Inspired by the residual learning framework,⁵⁰ subsequent studies could incorporate identity shortcut connections to mitigate degradation problems encountered when scaling to deeper architectures. In clinical practice, multimodal datasets incorporating hand X-ray, ultrasound, and MRI for diverse diseases such as RA, OA, and AS can be constructed utilizing databases like the China Rheumatism Data Center (CRDC). By developing multi-task classification models⁵¹ and exploring transfer learning-based multimodal fusion diagnostic approaches, complementary information from different imaging modalities can be leveraged to enhance diagnostic accuracy. Meanwhile, transfer learning techniques enable the reuse of pretrained model parameters, accelerating model optimization for RA diagnosis and treatment scenarios, thereby systematically improving differential diagnostic precision in complex clinical settings. Further, we will integrate clinical indicators, genetic data, laboratory test results, and radiomics features to construct a multi-source heterogeneous data fusion model. This approach aims to elucidate the underlying disease mechanisms and enhance predictive performance, thereby supporting personalized precision diagnosis for RA. Concurrently, we plan to conduct large-scale, multicenter prospective clinical trials for external validation of the model, systematically evaluating its real-time diagnostic accuracy and integration efficacy within clinical workflows. These efforts will facilitate the translation of deep learning models from experimental tools into reliable clinical decision-support systems. Ultimately, the fused model will be embedded into a mobile application (APP) framework, establishing an intelligent diagnostic closed-loop system to optimize healthcare delivery efficiency and quality.

Conclusion

In summary, this study demonstrated moderate diagnostic performance by using the VGG8 model to classify and diagnose X-ray images of the hands, including early RA patients. Through Grad-CAM and SHAP analysis, the “black box” of the model was revealed, and an online APP was built based on Streamlit to assist clinical doctors in decision-making, demonstrating the enormous potential of deep learning in medical imaging diagnosis.