1.1. Related Work

Before ML can be safely integrated into diagnostic IoT devices, robust model training and analysis must be conducted to ensure accurate inference. Regarding dental smile analysis, previous research efforts have focused on automating the recognition of dental issues through advanced computer vision (CV) techniques and ML algorithms. There is a multitude of existing ML techniques, and it is imperative that the most suitable ones are applied to particular problems. Recent years have seen a surge of a subfield of ML called deep learning (DL). DL is empowered by interconnected layers of neural networks (NN) and has been adapted to many applications in the scope of healthcare and CV [26]. There is a range of model architectures within DL for CV with their respective trade-offs in performance, resources, and latency.

Previous studies that employed DL in dental analysis have adapted variable architectures with varying degrees of success. One notable study [27] chose a combination of detection and segmentation, employing an architecture within DL called Mask Region-based Convolutional Neural Network (R-CNN). Mask R-CNN builds upon Faster R-CNN by introducing an additional branch dedicated to predicting masks at the pixel level, which are calculated based on rigorous labeling shown in Figure 2, for discerning and effectively segmenting relevant objects [28]. Due to the absence of publicly available datasets, [27] the study collected and annotated only 100 images, achieving a pixel accuracy of tooth segmentation between 90.1% and 97.4% for natural teeth and lower for dentures. The observed high accuracy is consistent with the nature of Mask R-CNN methods, which prioritize performance over speed. An analysis of a single frame using Mask R-CNN typically takes a few seconds, depending on the data to be analyzed and the number of inferences to be made [29]. This delay will add up with image preprocessing and routing, and it will be multiplied by the number of frames to be analyzed. For example, if three images are taken for averaged results or different angles for other applications, the delay can be 10-30 seconds. In pursuit of scalable real-time results and a smooth user experience, another study [30] employed an extension of You Only Look Once (YOLO), the YOLACT++ instance segmentation model, analyzing a dataset comprising 5500 images of faces distributed across diverse classes. While the model exhibited high accuracy in detecting facial features like the nose, eyebrows, and eyes, a significant decline in precision was observed when segmenting the gingiva and buccal corridor. This decrease in precision may be attributed to a relatively lower number of annotated instances per class for these specific regions in the training dataset, highlighting the importance of a well-balanced dataset in DL. Another study [31] employed routine clinical practices to acquire 1250 X-ray images for their objective of tooth detection and identification with faster R-CNN, citing the high cost of instance segmentation labeling as their rationale.

The common thread in all these studies is the use of relatively small datasets, which often prevents unlocking the full potential of NN and leads to poor generalization [32]. While the growth of big data has undeniably improved the capabilities of NN, particularly in the realm of CV, notable data gaps persist. In our case, this is evident due to the absence of an adequate dataset capturing dental smiles. Models trained on small datasets have a tendency to result in overfitting or not being able to generalize on unseen testing data, performing with high accuracy during training but low accuracy during testing. There is a multitude of techniques under data augmentation that compensate for the lack of data by artificially expanding the dataset. Data augmentation in CV makes for better DL models by making multiple copies of the same image with diverse geometric transformations, color augmentations, kernel filters, random erasing, and the application of generative artificial intelligence (GenAI) [33]. The latter, unlike the other listed methods, synthetically generates new data points by capturing and changing distinctive objects within the image and generating new images with additional noise to increase performance on unseen images [34]. GenAI, in particular, is promising in CV, especially with the advancements in text-to-image generation capabilities in recent years [33,35].

Numerous studies have been taking advantage of generative adversarial networks (GANs), a subclass of GenAI to generate plausible data and augment small or imbalanced datasets. One study used 13 small medical datasets to show that the optimal proportion of GAN augmentation significantly enhances the performance of all ML classifiers [36]. Similarly, another study built an emotion classification model and used GAN augmentation to increase samples of less common classes such as disgust, which increased classification accuracy by 5%–10% [37]. The study [32] observed a similar trend, employing GAN data augmentation to amplify an X-ray dataset and improving the CNN model for pneumonia and COVID-19 detection. Other generative techniques have been used to solve similar problems in different areas. For instance, [38] used autoencoders to successfully generate multivariate data for fire scenarios, taking slope, vegetation, and other factors into account. This shows the wide range of problems generative models could be leveraged in to aid classical ML in analyzing complex data.

While there is notable progress in data augmentation in the field of medical image analysis for tooth identification and segmentation, the issue of data scarcity remains a challenge for smile classification. This challenge is even more pronounced given that a comprehensive dental smile dataset is not yet available, limiting advancements in dental smile analysis. To address this challenge and fill this research gap, presents a novel and accurate framework for dental smile analysis empowered by the combination of DL methods and GenAI-empowered data augmentation. To the best of our knowledge, this is the first study to explore the application of ML in dental smile analysis, which has the potential to be applied not only in digital dental esthetic technology in clinics and hospitals but also in standalone home devices for accurate and efficient smile analysis.

1.2. Contribution and Vision

In this paper, we propose a novel diagnostic smile analysis tool for excessive gingiva detection, leveraging a CNN model and GenAI data augmentation. We employ a sequential topology architecture for its simplicity and adaptability. As with previously outlined studies, a persistent challenge in our research stems from the insufficient number of data available to generalize to unseen content. To resolve this issue, we curate a dataset of a total of 512 dental smile images from available open-source images and text-to-image AI-generated samples. The inclusion of AI-generated data from Adobe Firefly’s diffusion model [39] significantly enhances classification accuracy, demonstrating the efficacy of our methods. The usage of text-to-image generative models in data augmentation is still an early practice, but we show that it has potential in dentistry. It enables us to attain higher quality results, for example, detecting excessive gingival display correctly on previously unseen images with 81.607% accuracy, without incurring the costs associated with annotating segmentation data.

Furthermore, we integrated the trained CNN model into a minimalistic user interface (UI) that guides the user to the correct position and captures a single image. The image is subsequently preprocessed according to specifications and passed through the pre-trained CNN model. The image is then analyzed in the backend, and the UI displays the result of the smile analysis obtained from the model in real-time, forming an end-to-end application.

This application could be stand alone or integrated into an innovative IoT smart mirror technology, termed as the Internet-of-Mirrors (IoM) [40]. This visionary ecosystem of interconnected smart mirrors could be used to integrate the smile analysis application, as well as other digital health and beauty applications. Telehealth services, such as teledentistry, are time and cost-effective ways of health-related communication [41]. The motivation lies in connecting patients with healthcare professionals or products in a timely manner, as depicted in Figure 3. In this system, each mirror would be equipped with sensors, such as a camera, to acquire user input. In the context of smile analysis, the mirror would take the dental smile image, crop it according to the specifications, and use pre-trained CNN model to produce an output and inform the patient of their smile condition. The results of potential dental issues and personalized suggestions would be displayed on the UI. Smart mirrors placed in homes would serve users by promptly delivering tailored products or healthcare recommendations to align with their individual needs, and effectively facilitating connections between patients and professionals. Furthermore, smart mirrors placed in clinics could resolve the smile analysis subjectivity issue by providing a consistent basis for diagnosis.

To summarize, the main contributions of the paper are the following:

• Novel Diagnostic Smile Analysis Tool: We introduce a CNN model trained on real and AI-generated images of dental smiles to detect excessive gingival display, providing a non-invasive digital diagnostic tool.

• Dental Smile Dataset: A dataset of 512 dental smile images with a mix of real and AI-generated images from Adobe Firefly’s diffusion model, leading to a significant improvement of classification accuracy and addressing the challenge of limited data points for dental image analysis.

• Application Integration: Integration of the proposed solution into an application, offering a smooth human-computer interaction experience and instantaneously displaying the results of the diagnosis and suggestions.

2.1. Dataset

To train CNN models to an acceptable degree, a large dataset is required. There are many resources of fully curated and often readily preprocessed datasets of images for various CV uses online. In the context of dental smiles, the absence of a suitable dataset in open-source repositories posed a significant challenge. Consequently, the ethical collection of a diverse array of dental images encompassing various ethnicities, genders, and degrees of gingival display became a persistent endeavor.

In total, two sets of 256 (counting 512 combined) images were collected and evenly distributed into “gummy" and “normal" classes. The first set was manually curated from publicly available images scrapped from online frontal face images of people smiling. This set included samples showing highly positive excessive gingival displays as well as more discrete displays. The second set was obtained from text-to-image AI-generated images using Adobe Firefly’s diffusion model. The text prompts for the “normal” class ranged from “frontal portrait of a person smiling with teeth; this person does not have a gummy smile or braces” to simply selecting relevant images in “frontal portrait of a person smiling”. It is important to note that smile analysis with braces was excluded from the training dataset, as individuals undergoing treatment do not typically undergo aesthetic interventions until treatment is completed.

Generating excessive gingival display proved to be more challenging. The text-to-image diffusion model excels in comprehending daily human speech, but its grasp of medical terminology remains somewhat restricted from our observations. Changing terms like “excessive gingival display" to more commonly used phrases like “gummy smile” resulted in images of people chewing gum for several images. Providing special tags such as “three millimeters between the teeth and the upper lip” worked better, though not all candidates qualified. In the end, the prompts varied throughout the data collection, ensuring only relevant images were added. Despite requiring some filtering, this approach still consumed less time and resources compared to manually annotating teeth with tools like LabelMe or VGG Annotator. This is due to the fact that segmentation labels require 20–30 points to precisely trace the pixels of the teeth to be applied to the image. On the contrary, labeling classes only requires one identifying label per image.

Both sets contributed 56 randomly chosen images each to the testing folder, setting aside 112 of the original 512 images for testing. The remaining 400 images (200 from real, 200 AI-generated by the diffusion model) were randomly split with 80% assigned to training and 20% to validation. In total, three separate datasets were curated: real images (256 images), AI-generated images (256 images), and combined images (512 images). Compared to other CV classification datasets, the model trained on this dataset was more susceptible to overfitting. Therefore, it was vital that the dataset’s quality was superlative for the model to classify unseen images accurately. This was ensured by balancing a diverse range of individual frontal smile images evenly between the two classes and validating the correctness of the labels with a professional dentist.

2.2. Preprocessing

Proactive measures were taken in the preprocessing step to mitigate the risk of overfitting. Initially, all images were normalized and converted to grayscale using the OpenCV library. Following this, a pre-trained facial landmark detector from the Dlib library was employed to identify the coordinates of the mouth in each image. Subsequently, every image was cropped to a size of 28 by 28 pixels, centered around the mouth. This systematic preprocessing step was consistently applied to both training, validation, and testing data, ensuring the overall quality and consistency of the dataset. Due to the nature of this preprocessing approach, images of users’ faces were not saved but were funneled through the pipeline up to the cropping stage. This approach mitigated many privacy and ethical concerns related to data retention and destruction.

Before feeding these images into the model, multiple augmentation layers were applied to artificially expand the dataset. In addition to the generated images, each training image underwent a horizontal flip, three random zoom factors between −0.2 and 0.2, random contrast with a factor between 0.25 and 0.75, and random brightness with a factor between −0.5 and 0.5. This apprach increased the size of each dataset by a factor of seven (the original image and six augmentation layers), resulting in atotal of 3584 images.

The entire process pipeline is illustrated in Figure 4. which presents a flowchart from receiving raw images to preprocessing and training, using an example Adobe Firefly image.

2.3. Model Architecture

A CNN classifier model was built using Tensorflow and Keras to classify previously unseen images into either “gummy” or “normal” smile classes. A sequential model architecture was chosen for its simplicity, offering a linear topology that enables the customization of layer stacking while maintaining one input tensor and one output tensor. This choice was driven by the primary goal of minimizing the risk of overtraining, as small datasets tend not to generalize well to unseen testing data. The overall architecture of the model follows a standard layer sequence. The pattern of a 2D convolution layer followed by a 2D max pooling is used twice. The rectified linear unit (ReLU) function, represented by R(z) = max(0, z), served as the activation parameter. The ReLU activation function was also used in the first of the three densely-connected NN layers. The second dense layer used the softmax s(xi) function, which takes a vector xi and out- puts the probability distribution of each class for N classes, denoted by:

The final dense layer converted the output into a probability distribution across the classes. Each model, distinguished by real, AI-generated, and combined datasets, was trained using the Adam optimizer. Through hyperpa- rameter tuning, the optimal number of epochs was deter- mined to be 150, and the batch size was set to 32. All other parameters remained constant throughout training sessions.

2.4. Mirror User Interface

The envisaged IoM calls for the integration of an interactive dashboard. The interface of the dashboard was originally designed in-house, as shown in Figure 5 by the CSI group at JWSE at the University of Glasgow. This served as a base for a prototype UI to connect to the pre-trained smile analysis model. A basic layout of applications with corresponding functionalities was created using modular programming, allowing for flexible addition of buttons with corresponding function calls to accommodate the evolving functionalities of the application. For instance, ML models trained to detect and classify skin diseases could be paired with the Skin Analysis button in the backend function call without altering the overall structure of the code.

Upon pressing the Smile Analysis button, the system initiates video capture from the Intel RealSense Depth Camera D435, accompanied by real-time red bounding box tracking of the user’s face on the screen. Simultaneously, pre-trained Haar cascades for face and smile detection operate in the background. The bounding box changes color to green when the user maintains a smile for two seconds, at which point an image is captured. A demo of the entire process can be viewed here https://youtu.be/yj_FavXCL2I. The captured image then enters the standardized preprocessing pipeline and undergoes analysis within the classification model, yielding the resulting class instantaneously on the screen.

3.1. Gingival Display CNN Model

LetterSpace=-2.0The research yielded several key findings. First, we evaluated the performance of the CNN model for detection of excessive gingival display. The model employing diffusion model data augmentation in addition to geometric transformations achieved an average accuracy of 81.607% with a standard deviation of 1.938. This statistic reflects the average accuracy across 10 trials, each involving training the model over 150 epochs and evaluating its performance on 56 previously unseen images to ascertain the accuracy of label predictions. The standard deviation was then calculated using the standard formula, where the sum of the squared differences is divided by the number of data points denoted by n:

Figure 6 illustrates the fluctuation in accuracy percentage throughout the progression of epochs for a representative model, showing an upward logarithmic trajectory of training accuracy. The validation accuracy starts around 0.50 and closely mirrors the training accuracy trend, though slightly lower. The training and validation loss of the same session follows the opposite trend, as depicted in Figure 6. Training beyond 150 epochs proved ineffective, as validation accuracy and loss diverged from training resulting in a plateauing training accuracy and decreasing validation accuracy.

The GenAI-empowered CNN model was compared to the two byproduct models, one trained on 200 real images and the other on 200 AI-generated images. For consistency, each model’s testing accuracy was averaged over 10 trials. The box plots of each model are shown in Figure 7. As indicated by the error bars on the graph and the p value greater than 0.1, there was no significant difference between the performance of the model trained on real human images and the model trained solely on generative images. However, the median accuracy of the model trained on both real and generated data was significantly higher than that of the models trained on either real or GenAI images alone (p value < 0.01) and higher than the baseline of 50% (p value $<<$ 0.01).

Additionally, the confusion matrix of a representative model with an accuracy of 82.14% and an F1 score of 0.82 highlights the importance of a balanced dataset in achieving roughly even results in both ’gummy’ and ’normal’ classes. Figure 8 shows the squares on the right diagonal represent mismatches between true labels and predicted labels, indicating false positives and false negatives. These mismatches are generally outnumbered by the true positives in the first row and column and true negatives in the second row and column, representing a high numbers of correctly labeled testing data points.