Diabetes is a disease with increasing prevalence, and diabetic retinopathy (DR) is one of the most common complications (World Health Organization, 2022). DR is characterized by retinal abnormalities, which damage the eye and can lead to blindness. Its severity depends on the type and amount of retinal abnormalities: HE, microaneurysms (MA), hard exudates (EX), soft exudates (SE), intraretinal microvascular abnormalities (IRMA), and neovascularization (NV). DR is graded from $0$ to $4$ (no DR, mild, moderate, and severe nonproliferative DR, and proliferative DR), as described by Wilkinson et al. (2003). DR grading of fundus images is traditionally a manual process requiring medical expertise. Figure 1 provides examples of fundus images of eyes with increasing severity of DR including the ground truth segmentations of the six retinal abnormalities. These images underline the challenge of identifying relevant medical abnormalities without specialist training.

Deep neural networks have shown impressive results for objectively predicting levels of DR from fundus images (Lakshminarayanan et al., 2021). However, these models are complex and difficult to interpret, which is regarded as an obstacle for clinical implementation of such systems (Vellido, 2020). If ophthalmologists do not understand why the model made the specific predictions, they might refuse to use it. Moreover, if algorithms make decisions that can affect the life of the patient to a large extent, such as in medical diagnoses, the patient has the right to get an explanation about why the decision was made. In other words, being able to explain the decision process of deep neural networks for medical applications is crucial.

XAI aims to explain machine learning models and their predictions. Previous work on explainable artificial intelligence (XAI) for fundus image analysis have mainly focused on heatmap methods, which highlight the pixels in the image regarded as important during inference (van der Velden et al., 2022). As an example, Ayhan et al. (2022) performed a quantitative analysis of heatmaps produced by a wide selection of XAI methods to explain deep learning-based DR detection in fundus images, showing that the quality of heatmaps vary greatly. Despite their popularity, heatmap methods have some limitations (Arun et al., 2021). Heatmaps do not explain why an area in the image was highlighted (Kovalerchuk et al., 2021), or quantify how important the highlighted area is. Moreover, one heatmap is generated for each observation, making it challenging to get a global overview of how the model interprets images.

Concept explanation methods are attractive for medical applications because they measure how much the deep neural networks are influenced by high-level concepts representing clinical findings (Salahuddin et al., 2022). A concept can be described as a theme or topic, e.g., ’stripes’ and ’dots’ for natural images, or diagnostic findings such as ’hemorrhages’ and ’microaneurysms’ for fundus images. The six different diagnostic concepts used for this work is shown with the ground truth segmentation masks in Figure 1. TCAV (Kim et al., 2018) and Concept Bottleneck Models (Koh et al., 2020) are two concept-based methods that have several advantages above heatmap methods. Both allow the user to define the concepts, which ensures relevant and meaningful concepts that are interpretable for the end-users. How the concepts are used varies between the two explanation methods and is outlined in Sections 2.1 and 2.2. Moreover, the relative importance of the concepts can be quantitatively measured. For Testing with Concept Activation Vectors (TCAV), concept scores can be generated for a group of images, e.g. images belonging to the same class, allowing the user to investigate whether the model has learnt aspects coherent to domain knowledge and diagnostic guidelines. CBMs, on the other hand, allow the user to directly modify the model’s intermediate concept predictions at test time and observe how this affects the final model prediction. This way of manipulating the model after training is attractive in the medical field, e.g. if the clinician wants to increase the emphasis of a concept in the image that the model missed. While TCAV explains models post-hoc, i.e., predicting the concepts after the classification model has been trained, CBMs provide ad-hoc explanations, where the prediction of the concepts are trained together with the classification model. Even though concept explanations can be more intuitive than heatmaps for medical doctors, neither TCAV or CBMs have been extensively studied in the field of DR grading. In this work, we thus investigate and compare TCAV and CBMs for explaining deep neural networks trained to grade DR in fundus images.

Seven open access datasets were used in the current study: APTOS (Karthik and Dane, 2019), DR Detection (Dugas et al., 2015; Cuadros and Bresnick, 2009), Messidor-2 (Decencière et al., 2014; Abràmoff et al., 2013), FGADR (Zhou et al., 2021), DDR (Li et al., 2019), DIARETDB1 (Kauppi et al., 2007) and IDRiD (Porwal et al., 2018). An overview of the data is provided in Table 1. All datasets including image-level annotations of DR grade were used to train the deep neural networks for DR grading. Several data sources are combined to ensure generalizable models, as the fundus images were captured at different locations, by different healthcare personnel, and using different devices, making the training data diverse. For concept generation, fundus images segmented with medical findings relevant for diagnosis of DR were used. FGADR was applied for both DR grading and concept generation. The distribution of DR levels are highly skewed, with the majority of the images representing eyes with no signs of DR, and annotations of IRMA and NV are only available in FGADR. The four datasets used for developing the DR grading models were split into $80\%$ for training, $10\%$ for validation, and $10\%$ testing. Images from the same patient were placed in the same split of the dataset. Additionally, the black background was removed from all images. Further on, Contrast Limited Adapted Histogram Equalization (CLAHE) was applied to enhance the image quality by making the blood vessels and retinal findings more visible (Zuiderveld, 1994). For the training set, several image augmentation techniques, such as random flipping, blurring, and change of perspective, were also applied. The source code for all the experiments, including concept explanations, is publicly available online¹¹1https://github.com/AndreaStoraas/ConceptExplanations_DR_grading.

The models based on the Densenet-121 architecture outperformed the Inception V3-based models for both TCAV and CBMs models. The performance metrics on DR grading for the models based on Densenet-121 on the combined and FGADR test sets are presented in Table 2. Because concepts were not used when training the models explained by TCAV, the performance metrics reported for the ‘TCAV models’ can be regarded as baseline results for DR level classification. We observe from Table 2 that the model used for TCAV had the highest performance on the combined test set. The CBM trained on six concepts generalized poorly from FGADR to the combined test set, but performed best on the FGADR test set. This is not surprising, as the FGADR training set is much smaller and less diverse than the combined training set. For the model trained on four concepts, the performance was not significantly different between the two test sets because of the mixed training data. Regarding the concept detection task, the CBM trained on six concepts correctly identified 86.2% of the diagnostic concepts, compared to 82.5% for the model based on four concepts. The balanced accuracy also increased from 80.9% to 85.9% when increasing the number of concepts. These results were computed on the FGADR test set. The CBM trained on six concepts was used in further experiments since this model performed best on the FGADR data for both DR grading and concept detection.

Due to memory limitations, TCAV scores were calculated on a representative subset of the combined test set consisting of $50$ randomly picked images from each DR level. The performance for DR prediction on the representative test set did not differ significantly from the full test set. As mentioned in Section 2.1, the TCAV concepts were either based on the full images or only the image area surrounding the medical finding(s) of interest. Masking the concept images did not generate significantly different results, thus the full concept images were used. Figure 3 shows both TCAV scores for each concept and the concept predictions with the CBM at different DR levels. Increasing DR severity is associated with higher TCAV scores for more concepts and higher concept counts. Note that the TCAV scores and CBM concept counts are not directly comparable. Different datasets were used, and the TCAV scores reflect relative importance between the concepts while the CBM counts are merely the predicted presence of concepts.

Test time intervention on an increasing number of concepts was performed for the CBM trained on six concepts. The intervention concerned concepts that were wrongly predicted by the CBM. As previously mentioned, the order of which concepts to intervene on was determined by the corresponding balanced accuracy in DR classification following the intervention on a single concept. Consequently, the concept resulting in the best balanced accuracy on the DR grading was included first, while the concept with the worst performance was included last. The left hand side of Figure 4 shows the DR classification performance metrics on the entire fine-grained annotations diabetic retinopathy (FGADR) test set when concept intervention was performed incrementally. We observe that the intervention had best effect when five out of six concepts were corrected (correcting the HE concept did not show further improvement). Even though the performance did not increase dramatically, test time intervention improved the balanced accuracy and precision compared to no intervention. Next, test time intervention was performed only on the misclassified images in the test set. As seen on the right hand side of Figure 4, the performance peaked when all six concepts were corrected. The balanced accuracy increased from 56.0% with no corrections to 69.4%. Qualitative examples on how test time intervention affected the DR level predictions are provided in Figure 5.

According to the diagnostic criteria for DR level 1 (mild DR), MA should be the only abnormality present (Wilkinson et al., 2003). This corresponds well to both TCAV and CBM results in Figure 3, highlighting MA as the most important concept for DR level 1. The IRMA, SE, and MA concepts were ranked highest by TCAV for DR level 2, and all concepts but NV was identified by the CBM. This is consistent with the diagnostic criteria for MA, HE, and SE. Interestingly, the IRMA concept was ranked highest by TCAV, even though this finding is mainly associated with DR level 3. By inspecting the datasets more closely, several DR level 2 images actually contained IRMA. It is therefore reasonable that the model identifies IRMA as important when making predictions on DR level 2 images. Additionally, several DR level 3 images were predicted to belong to DR level 2. This can also partly be explained by the high TCAV scores for the IRMA concept. The TCAV scores for DR level 3 gave high importance to MA, HE, EX, SE, and IRMA, and are coherent with the diagnostic criteria for this DR level, as well as the presence of all concepts except NV for the CBM. The presence of HE, EX, and SE was increased compared to level 2, which is also expected. Finally, for DR level 4, TCAV ranked NV highest, followed by EX and HE, which was expected as this is the only DR level where NV is present. NV was also identified by the CBM. Taken together, the concept-based explanations seem to align with established medical knowledge about DR. This is encouraging in terms of applying TCAV and CBMs in the clinic for explaining deep neural networks for DR grading. The next step would be to validate the usability of these concept-based XAI methods with feedback from ophthalmologists. In this current study, we compare the explanations with widely accepted guidelines for grading of DR, but there might be other clinically relevant aspects that were not considered.

The overall ranking of the concepts were not different for the concepts based on full images and concepts based on masked images that focused on the area containing the medical findings of interest. This could mean that even though some findings are small, they are still sufficiently learnt from the full images. On the other hand, similar results could also indicate that the masking technique was not efficient enough. In order to avoid extreme deviations in the image sizes for the masked concept images, a lower limit of 520 $\times$ 520 pixels was applied. For images of low resolutions, this restriction implied that a very small part of the image was removed and that the masked and full image versions were almost the same. However, since the model was trained on images input size 620 $\times$ 620 pixels, input images with few pixels were regarded as less likely to provide useful results. Consequently, the lower pixel limit was considered the best alternative when preparing the masked concept images. The small variations between the results from full concept images and masked concept images indicate that it is sufficient to use the original images for concept generation. Further on, time is saved because we do not have to mask out the relevant findings from the images in order to get meaningful concepts.

IRMA and NV are typical findings for severe DR. The presence or absence of these findings is therefore expected as useful information when learning to grade DR. As observed in Table 2, the CBM trained on four concepts (MA, HE, EX and SE) performed worse when grading DR on the FGADR test set. Even though the model trained on four concepts received more training data and was better at predicting these four concepts than the model trained on all six concepts, missing information about IRMA and NV seems to negatively affect the DR grading. This highlights the importance of high quality concept annotations for these XAI methods.

Test time intervention showed to be a great advantage of the CBM and resulted in more accurate model predictions. When the concepts for incorrect predictions are inspected, the user can get important information about why an image was misclassified. As an example, the DR level $4$ image in Figure 5 was misclassified as DR level $3$, because the bottleneck model missed the NV concept. By correcting this, the image was correctly classified.

Despite including $>33800$ images for training the combined DR classification model, the model did not outperform earlier deep neural networks (Lakshminarayanan et al., 2021). In this work, we combined four different datasets where the fundus images were captured using various devices and arriving from patient populations in different geographical areas. The high image diversity probably makes it challenging for a model to capture representative patterns in the data compared to more homogenous datasets. On the other hand, the diverse collection of training data could make the model robust to variations and noise in data from a real-world setting and increase its ability to generalize to new datasets. We could not identify any previous studies using the same combination of data as us, meaning that our results are not directly comparable to previously reported performance metrics. Because this study focuses on the explanation methods, the performance is regarded as sufficiently high.

Concept-based explanations are promising for increasing the understanding of DR grading with deep neural networks. While CBMs allow for test time intervention on the concepts, these models are limited by the lack of publicly available medical datasets annotated with both concepts and target labels. For TCAV, concepts can be defined using other data sources, meaning that the training data does not need additional concept annotations. Consequently, the model explained by TCAV outperformed the CBMs for DR grading. Our results highlight a major drawback of the CBMs: Because the development dataset must be annotated with concepts and DR level, the amount of available data is small. The requirement of training a modified CBM also complicates direct comparison and combination with other XAI methods. Before larger medical datasets annotated with concepts and target labels are available, TCAV gives the best trade-off between model performance and explainability for DR grading.