AI Grading of Lateral Canthal Lines: Novel Models for Unseen Synthetic Image Generation and Data Augmentation

Introduction

The application of artificial intelligence (AI) in dermatology has been increasing with promising results.^1–4 A large labeled training dataset is usually required to develop accurate and reliable AI algorithms for clinical applications since small datasets are prone to overfitting and cannot handle extreme cases.⁵ However, large-scale clinical images are frequently lacking and difficult to acquire. Human labeling of these images is also time and labor-consuming. Moreover, imbalanced or biased data in training datasets also affects the outcome of AI systems. These constraints may limit the development of AI algorithms for dermatologic applications.

To address these issues, traditional data augmentation methods that involve transforming existing data through operations such as rotation, flipping, scaling, translation, and color transformation have been utilized.⁶ However, traditional data augmentation may encounter problems including excessive deformation leading to distortion and is not suitable for all tasks.⁶ Therefore, a more recent approach to data augmentation involves using Generative Adversarial Networks (GANs).⁷ Synthetic images generated from GANs have been applied to medical imaging, including images of skin cancer, onychomycosis, and acne.^8–11 Recently, Cho et al utilized GAN to generate synthetic data from non-standardized images to train a convolutional neural network (CNN) for classifying melanocytic skin lesions with high accuracy.¹² However, GANs may face challenges such as insufficient generation diversity and difficulty in neural network training.¹³ Diffusion-based image generation models have been developed to address these limitations. They are increasingly applied in medical imaging for tasks such as image segmentation, reconstruction, anomaly detection, and synthetic image generation, among others.^14,15 Although diffusion-based image generation models offer greater controllability compared to GANs, achieving precise localized control of the generated image is still a challenge. To date, diffusion-based image generation models have not been applied to dermatological clinical images. In this study, we introduce a compositional zero-shot and few-shot generation model that incorporates structured compositional labels into the diffusion framework, thereby enhancing image generation controllability for generating clinical dermatologic photographs.

Prominent lateral canthal lines, or crow’s feet, are a common reason for patients seeking aesthetic treatments. The treatment outcome is evaluated using validated grading scales.^16,17 Standardized grading of lateral canthal lines is important for providing objective severity assessment for treatment planning and outcome monitoring, allowing for improved patient satisfaction and increased reproducibility of research in this area. In this study, we aimed to create a novel generative AI algorithm capable of generating high-quality patient images of the periorbital region with modifiable lateral canthal line severity based on standardized clinical photographs. In this paper, we describe the process of generating patient images with canthal wrinkles using the compositional zero-shot and few-shot generation model.⁴ We demonstrate how the generated images can address the issues of insufficient medical imaging samples or data imbalance in a population of East Asians, thereby improving the overall accuracy of the classifier performance. The generated photographs were used to train a CNN classifier for grading lateral canthal line severity. We also trained CNNs for lateral canthal line grading based on original patient photographs or a dataset containing images generated using traditional data augmentation techniques. The classification accuracies of the above three CNNs were compared.

Materials and Methods

Dataset and Image Preprocessing

Photographs of 121 subjects (6 males, 115 females; mean age 43.53 10.6 years) receiving procedures for aesthetic enhancement, including laser treatments and injectables, were included. All patients are of East Asian descent and have Fitzpatrick skin type III or IV. Standardized photographs of the left and right lateral oblique face under resting conditions were obtained using the VISIA^® skin analysis system under standard lighting (7th generation, Canfield Scientific).¹⁸,283 treatment sessions and 566 (left and right) photographs were used for evaluation. The severity of lateral canthal lines of each image was evaluated by two board-certified dermatologists using a five-point rating scale (0, 1, 2, 3, 4) with a grade 0 indicating no wrinkles and higher grades indicating more severe wrinkles (Figure 1a).¹⁷ In cases where discrepancies occurred, the evaluators discussed the case and made the final decision. All photographs used in this study were preprocessed by cropping, resizing, and normalization. The periorbital region containing complete lateral canthal lines was cropped from the original images. The original image resolution was approximately 4K, and the periorbital region containing complete lateral canthal lines was cropped to 1024*1024 pixels. The cropped images were resized to 128*128 pixels and normalized with a mean of 0.5 and a variance of 0.5 (Figure 1a). This normalization aims to stabilize the model training process and reduce the impact of large differences in input data. This study was approved by the Kaohsiung Medical University Hospital Institutional Review Board (KMUHIRB-E(I)-20190418), and the requirement for patient consent was waived as all images used were deidentified. The study was conducted in compliance with the Declaration of Helsinki and followed the CLEAR Derm consensus guidelines¹⁹ for dermatology AI and the complete checklist is provided in Supplementary Table 1.

Figure 1 Original dataset of patient photographs with lateral canthal line grading from grade 0 to 4 after cropping the periorbital areas (a) and synthetic images with corresponding severity of lateral canthal lines using the compositional zero-shot and few-shot generation model (b).

Generative Model: The Diffusion Process

The image generation model was based on a diffusion model architecture.²⁰ Diffusion models consist of two processes: forward diffusion and inverse diffusion. Forward diffusion occurs in the form of a Markov chain progressively adding Gaussian noise with variance to the latent variable to generate a new latent variable . The Forward Diffusion Process can be represented as

This process is akin to continuously adding noise to an image until it becomes completely blurred, resembling a situation where the image is entirely composed of noise, as depicted in Supplementary Figure 1. The inverse diffusion process involves progressively reversing the forward process and is represented by the following equation

This process entails using a trainable neural network to remove the added noise and restore the original data (Supplementary Figure 2). The key to the inverse diffusion process lies in how the model is trained to reconstruct clear images from the observed blurry images.

When the model can accurately predict the noise at each moment, we only need to sample once from the original Gaussian noise and iteratively denoise it to reconstruct a complete image. Through the layers of denoising in the inverse diffusion process, we can generate the desired dataset. Due to the control of added variance, each generated photograph can have slight variations.

With many major companies investing significant resources in building and researching generative models, numerous impressive models have emerged, such as OpenAI DALL-E 2,²¹ Google Images,²² or large-scale image generation models like Stable Diffusion.²³ However, applying these large models trained on massive datasets, often in the millions, to our tasks presents certain challenges. Their datasets encompass a wide range of styles, species, and forms, which differ greatly from the images in our task, making it difficult for subsequent classification models to learn key features and even causing counterproductive effects. How to mitigate this problem will be discussed in the following section.

The Compositional Zero-Shot and Few-Shot Generation Model

To make the above-described diffusion model generate specific images, a control variable c must be added that constitutes a conditional generation model. The forward diffusion model thus becomes

and the inverse diffusion process in represented as

To mitigate the issues concerning existing generative AI models, this study utilizes fully initialized parameters of the generative model in conjunction with a specialized label embedding layer. By training the label embedding layer to induce the model with an understanding of the concepts and corresponding relationships among different compositions, the advantage lies in the likelihood that the distribution of images generated by the model will exhibit a higher correlation with the training data.

Before introducing conditional generative models, it is essential to elucidate the concept of compositionality. Compositionality refers to the cognitive ability of humans to decompose, recombine, and validate compositional concepts. The recombined concepts may be entirely new or reinforce existing ones. For instance, “red apple”, “running lion”, and “beautiful rose” can be decomposed and recombined into “red rose” and “running apple”, where “red rose” is plausible while “running apple” is not. Applied to wrinkle generation, this concept translates to the concepts of images corresponding to wrinkle grades in the dataset, such as the distribution of wrinkle grades from grades 0 to 4. If the generative model can integrate different wrinkle data from the dataset and learn their concepts through neurons, it can subsequently infer and generate a large number of images corresponding to their grades but exhibit slight variations (Figure 2). This process aids significantly in augmenting the dataset and subsequent wrinkle classification.²⁴ A more detailed explanation of the concept of compositional zero-shot image generation is illustrated in the supplementary file (Supplementary Figure 3).

Figure 2 Illustration of applying compositional few-shot image generation to lateral canthal lines.

The model architecture is depicted in Figure 3. The noise fitting in the backward process is predicted by Unet.²⁵ The training process runs a total of 1000 iterations, with each iteration consisting of 100 noise training and parameter update steps. The conditioning control here is represented by compositional class labels and the model learns the corresponding relationships during training. The hyperparameters of the generation model are summarized in Supplementary Table 2. All generated images were manually reviewed, and unsuitable images were discarded.

Figure 3 Architecture of the generation model.

Classification Model

A neural network composed of a CNN feature extractor and a classifier was employed to classify the severity of lateral canthal lines. As shown in Supplementary Figure 4, the feature extractor was based on the ResNet-34 network pre-trained on ImageNet.²⁶ The output of the average pooling layer was a 512-dimensional feature vector. The feature vector was further connected to a classifier composed of three fully connected (Fc) layers and three dropout layers. The activation function of the last output layer was a Softmax function, which generates the conditional probability estimates for each class. The classifier gradually reduced the output dimension from 512, 512, to 256. The final output indicates the probability of the grade of lateral canthal lines of the input image and the highest score is chosen as the model’s final prediction result. The neural network in the present study was built and stimulated on PyTorch. The training details of the classification model are summarized in Supplementary Table 3.

Model Training and Validation

To verify the impact of the compositional zero-shot and few-shot generation model, we compared the lateral canthal line classification accuracy, learning curves, and area under the receiver operating characteristic curve of the CNN trained on three different datasets. The first approach involved using the original dataset and splitting it into training, validation, and test sets in a 64%–16%–20% ratio. The second approach also used the original dataset but incorporates image processing techniques including rotation (−10° to +10°), random sharpening, automatic contrast adjustment, and random flipping (10%). The data split remained the same as in the first approach. These operations were performed using PyTorch functions. The third approach, which is the focus of this study, involved using a compositional zero-shot and few-shot generation model to augment the dataset in addition to traditional data augmentation methods described in the second approach. More specifically, 2100 training images were generated for each grade, resulting in 10,500 images generated across 5 grades. Sixty-four percent of the original data and 80% of the generated images form the new training dataset, while 16% of the original data and 20% of the generated images were used for the validation set, and 20% of the original data are used for the test set. Each model underwent 200 epochs, with validation and calculation of validation accuracy performed after each training iteration. Following training, the model with the highest validation accuracy is selected as the final model. The outcomes of the model trained on the three datasets were evaluated in terms of overall classification accuracy.

Results

Generated Images

A total of 2100 generated images were selected for each class, with the aid of human experts, resulting in 10,500 generated images across 5 grades of lateral canthal lines. The new dataset consisting of 80% of original photographs and generated images was more balanced compared to the original dataset as the proportion of each grade is similar in the new dataset (Supplementary Figure 5). Examples of images generated by using the compositional zero-shot and few-shot generative model are shown in Figure 1b. The overall style of the generated images is similar to the original photographs.

Grading Results: Overall Accuracy Improved After Adding Generated Images to the Dataset

The grading accuracies of the three approaches are summarized in Table 1. In one test that showed the greatest performance difference, the model trained on the original dataset achieved an accuracy of 82%, while the second approach, utilizing traditional data augmentation methods, yielded an accuracy of 85%. The third approach, using the compositional zero-shot and few-shot generation model in addition to the traditional data augmentation methods, achieved an accuracy of 91%. The mean ± standard deviations of classification accuracies are 0.8346 ± 0.0233 for the original dataset, 0.8582 ± 0.0123 with traditional augmentation, and 0.8964 ± 0.0073 when generated images were included. Clearly, the accuracy of the three methods increases progressively, which aligns with our hypothesis that increasing the amount of data can improve the model’s fitting ability, resulting in higher accuracy.

Table 1 Performance of Classification Models Trained on the Dataset with and without Image Augmentation

Comparison of Learning Curves

The orange line in Supplementary Figure 6 represents the accuracy of the training set at different time intervals, while the blue line represents the accuracy of the validation set at different time intervals. From left to right of the graph, it can be observed that the model’s accuracy improves in the rightmost graph (82% to 91%). The larger gap between the training and testing curves in the three graphs also indicates that the rightmost graph has a denser gap, suggesting that using generated images for data augmentation slightly alleviates overfitting issues. While overfitting did not occur in the middle graph, its accuracy plateaued around 85%.

Receiver Operating Characteristics

The areas under the curves (AUC) of the three approaches are summarized in Table 1. The AUC of the classification model trained on the dataset including the generated images was superior compared to the other two methods (0.981 vs 0.950 and 0.934) (Figure 4f). Moreover, within the same method, we also conducted receiver operating characteristic (ROC) comparisons across different output categories. It is noted that our proposed method outperforms the other two methods in four out of five grades, with only one grade (grade 1) slightly lower than the traditional data augmentation approach (Figure 4a–d). This further validates the enhancement of our method in classification effectiveness.

Figure 4 The receiver operating characteristic (ROC) curves for each grade (a–e) of the original dataset (blue line), the traditional data augmentation dataset (orange line), and the dataset with generated images (green line). The area under the curve of the dataset with images generated using our proposed generative model was greater than that of the other two datasets (f).

Confusion Matrices

As observed from the confusion matrices (Supplementary Figure 7), each model generally does not exhibit significant errors, such as misclassifying grade 4 lateral canthal lines as grade 0 or 1. Most errors occur in misclassifications with a difference of one grade. Moreover, models trained on the dataset augmented with the generated images tend to accurately predict the correct answers and confine errors within one grade difference (Supplementary Figure 7c). On the other hand, models trained on the original dataset or dataset augmented with traditional methods sometimes exhibit misclassifications with a difference of two grades or more (Supplementary Figure 7a and b). It can also be noted from the plots that using the dataset augmented with generated images shows significant improvements in the classification of grade 4 categories compared to the other two methods. This also validates the notion that augmenting data with the generated images can alleviate classification errors caused by data imbalance.

Discussion

In this study, we developed an AI image generator based on a compositional zero-shot and few-shot generation model, a conditional image generation algorithm. The innovation of this study lies in developing and training a diffusion model specifically tailored to our dataset, ensuring that the generated images are consistent with the original dataset distribution and characteristics, thereby avoiding the creation of incongruous images. Additionally, we employed a novel classification model training method, generating a large volume of synthetic data from a small initial dataset to enhance the training process. This approach was validated by comparing the classification model trained with these synthetic data to models trained without them, demonstrating superior accuracy and stability in the model’s performance.

There are increasing applications of deep learning on dermatologic images, but large datasets with high-quality images are difficult to obtain.⁵ Increasing training data improves the performance of CNN classifiers.²⁷ Traditional data augmentation techniques, including image rotation, flipping, cropping, noise injection, and color space transformation, have been utilized to increase data size.⁶ However, important information may be discarded during image transformation.⁶ Recently, increasing studies using a generative adversarial network (GAN) for data augmentation have produced reliable results.¹² Nevertheless, GAN models still face many challenges, including insufficient generation diversity and difficulty in neural network training.¹³

Among various image generation models, diffusion models have gained great popularity recently for their ability to generate high-quality, realistic images.²⁰ Initially, currently off-the-shelf diffusion models (eg, Stable Diffusion) were used to generate images for this project. However, with Stable Diffusion, we were not able to generate images that matches the style of our original photographs and could not control the severity of lateral canthal lines of each generated image to the desired grade. By using the compositional zero-shot and few-shot generation model, images similar to the original photographs with controllable lateral canthal line severity were generated (Figure 1b). This approach allows researchers to generate realistic images according to the specific requirements of each task. In this study, the generated images were incorporated into the original dataset of clinical photographs for training a classification model to overcome the limitations of a small dataset. The model trained on the dataset with synthetic images demonstrated both superior classification accuracy (91% vs 85% and 82%) and AUC (0.981 vs 0.950 and 0.934) compared to the original dataset and the dataset augmented using traditional augmentation methods, respectively. This result suggests that the compositional zero-shot and few-shot generation model is an effective approach for data augmentation in AI model training.

In addition to the size of datasets, imbalanced data also affect the outcome of CNN classifiers, resulting in a less accurate classification for significantly underrepresented categories.²⁸ Imbalanced data is commonly encountered in clinical data. In the present study, our original dataset was also imbalanced and contained a higher proportion of grade 1 and grade 2 images and significantly fewer grade 4 photographs (Supplementary Figure 5a). Consequently, the AUCs for grade 4 lateral canthal lines were both below 0.95 for both the classification neural network trained on the original dataset and the dataset using traditional augmentation methods (Figure 4e). With generated images (Figure 4e), however, the AUC for grade 4 lateral canthal lines was improved to 0.99 after training with a more balanced dataset. This result confirmed that generated images allow for a more balanced dataset and improve classification outcomes.

Another important issue in AI model training is the lack of data representativeness and diversity.²⁹ Lack of training data incorporating underrepresented populations may lead to biased results and reinforcement of social inequalities.²⁹ In dermatology, the underrepresentation of darker skin tones in training datasets contributes to reduced AI model accuracy in patients with these skin types.³⁰ We believe that the compositional zero-shot and few-shot model may also be applied to overcome this problem, as it is able to generate images with limited examples. Incorporating generated images of underrepresented populations for AI model training will be a valuable topic for future research.

This study has several limitations. First, there is limited diversity of the study population as most of the subjects enrolled are females of East Asian descent with Fitzpatrick skin types III–IV. The results should be interpreted with caution when applied to other populations. However, as the compositional zero-shot and few-shot generation model is able to generate images with limited dataset, it may also be used to overcome this issue by incorporating examples of underrepresented populations. More importantly, this study aims to demonstrate that incorporating images generated by the newly developed model can enhance the training of a classification model, and our results support this objective. Secondly, the quality of the generated images was not evaluated by objective metrics. However, all generated images were examined by human experts to ensure only adequate images were included for training. We believe that this mechanism allows us to integrate and leverage knowledge from both human judgment and data-driven processes. Lastly, as this is a single-center study, further external validation will be valuable to confirm the generalizability of our results.

Conclusion

The results of this study suggest that the compositional zero-shot and few-shot generation model can be used to produce clinical images similar to real patients with different characteristics. Incorporating these synthetic images could help build larger and more balanced datasets for AI algorithms training, thereby improving classification outcomes. While future projects focusing on AI applications using dermatologic photographs could potentially benefit from this image generation method, further validation is needed to determine its reliability in a more diverse setting.