Congenital heart disease (CHD) is the most common group of congenital malformations (Bennasar et al., 2010)(Yeo et al., 2018)(van Velzen et al., 2016). CHD is a defect in the structure of the heart or great vessels that is present at birth. Approximately $6-11$ per $1000$ newborns are affected. $20-30\%$ of these heart defects require surgery within the first year of life (Yeo et al., 2018). In order to detect the disease, the most common approach is the standard anomaly ultrasound scan at approximately $20$ weeks of gestation (e.g. 18+0 to 20+6 weeks in the UK). In contemporary screening pathways, i.e., 2D ultrasound at GA 12 and 24, the prenatal detection rate of CHD is in a range of $39-59\%$. (Pinto et al., 2012) (van Velzen et al., 2016) In (Yeo et al., 2018), algorithmic support has been used to find diagnostically informative fetal cardiac views. With this aid, clinical experts have been shown to discriminate healthy controls from CHD cases with $98\%$ sensitivity and $90\%$ specificity in 4D ultrasound. However, 4D ultrasound is not commonly used during fetal screening and in the proposed teleradiology setup still all images have to be manually assessed by highly experienced experts to achieve such a high performance.

In this work we focus on a subtype of CHD, Hypoplastic Left Heart Syndrome (HLHS). Examples of HLHS in comparison with healthy fetal hearts are presented in Figure 1. HLHS is rare, but is one of the most prominent pathologies in our cohort. In HLHS the four chamber view is usually grossly abnormal, allowing the identification of CHD (although not necessarily a detailed diagnosis) from a single image plane. A condition that is identifiable on a single view plane provides a clear case study for our proposed method. If HLHS is identified during pregnancy, provisions for the appropriate timing and location of delivery can be made, allowing immediate treatment of the affected infant to be instigated after birth. Postnatal palliative surgery is possible for HLHS, and the antenatal diagnosis of CHD in general has been shown to result in a reduced mortality compared to those infants diagnosed with CHD only after birth (Holland et al., ). However, the detection of this pathology during routine screening still remains challenging. Screening scans are performed by front-line-of-care sonographers with varying degrees of experience and the examination is influenced by factors such as fetal motion and the small size of the fetal heart.

One-class classification is a case of multi-class classification where the data is from a single class. The main goal is to learn either a representation or a classifier (or a combination of both) in order to distinguish and recognise out-of-distribution samples during inference. Discriminative as well as generative methods have been proposed utilizing deep learning, for example one class CNN (Oza and Patel, 2019) and Deep SVDD (Ruff et al., 2018). Usually these methods utilise loss functions, similar to those of OC-SVM (Schölkopf et al., 2001) and SVDD (Tax and Duin, 2004) or use regularisation techniques to make conventional neural networks compatible to one-class classification models (Perera et al., 2021). Generative models are mostly based on autoencoders or Generative Adversarial Networks. In this work we mainly focus on the application of generative adversarial networks for anomaly detection in medical imaging.

Generative adversarial networks for anomaly detection were first proposed by (Schlegl et al., 2017). In (Schlegl et al., 2017), a deep convolutional generative adversarial network, inspired by DCGAN as proposed by (Radford et al., 2016), is used as AnoGAN. During the training phase, only healthy samples are used. This approach consists of two models. A generator, which generates an image from random noise and a discriminator, which classifies real or fake samples as common in GANs. More specifically, the generator learns the mapping from the uniformly distributed input noise sampled from the latent space to the 2D image space of healthy data. The output of the discriminator is a single value, which is interpreted as the probability of an image to be real or generated by the generator network. In their work, a residual loss is introduced, which is defined as the $l1$ norm between the real images and the generated image. This enforces the visual similarity between the initial image and the generated one. Furthermore, in order to cope with GAN instability, instead of optimizing the parameters of the generator via maximizing the discriminator’s output on generated examples, the generator is forced to generate data whose intermediate feature representation of the discriminator ($D_{H}$) is similar to those of real images. This is defined as the $l1$ norm between intermediate feature representations of the discriminator given as input the real image and the generate image respectively. In AnoGAN, an anomaly score is defined as the loss function at the last iteration, i.e., the residual error plus the discrimination error. AnoGAN has been tested on a high-resolution SD-OCT dataset. For evaluation purposes, the authors report receiver operating characteristic (ROC) curves of the corresponding anomaly detection performance on image level. Based on their results, using the residual loss alone already yields good results for anomaly detection. The combination with the discriminator loss improves the overall performance slightly. During testing, an iterative search in the latent space is used in order to find the closest latent vector that reconstructs the real test image better. This is a time consuming procedure and this optimisation process can get stuck in local minima.

Similar to AnoGAN, a faster approach, f-AnoGAN has been proposed in (Schlegl et al., 2019). In this work, the authors train a GAN on normal images, however instead of the DCGAN model a Wasserstein GAN (WGAN) (Arjovsky et al., 2017)(Gulrajani et al., 2017) has been used. Initially, a WGAN is trained in order to learn a non-linear mapping from latent space to the image space domain. Generator and discriminator are optimised simultaneously. Samples that follow the data distribution are generated through the generator, given input noise sampled from the latent space. Then an encoder (convolutional autoencoder) is training to learn a map from image space to latent space. For the training of the encoder, different approaches are followed, i.e training an encoder with generated images (z-to-z approach-ziz), training an encoder with real images (an image-to-image mapping approach -izi) and training a discriminator guided izi encoder ($izi_{f}$). As anomaly score, image reconstruction residual plus the residual of the discriminator’s feature representation ($D_{H}$) is used. The method is evaluated on optical coherence tomography imaging data of the retina. Both (Schlegl et al., 2017) as well as (Schlegl et al., 2019) use image patches for training and are modular methods which are not trained in an end-to-end fashion.

Another GAN-based method applied to OCT data has been proposed by (Zhou et al., 2020), in which authors propose a Sparsity-constrained Generative Adversarial Network (Sparse-GAN), a network based on an Image-to-Image GAN (Isola et al., 2017). Sparse-GAN consists of a generator, following the same approach as in (Isola et al., 2017), and a discriminator. Features in the latent space are constrained using a Sparsity Regularizer Net. The model is optimized with a reconstruction loss combined with an adversarial loss. The anomaly score is computed in the latent space and not in image space. Furthermore, an Anomaly Activation Map (AAM) is proposed to visualise lesions.

Subsequently, AnoVAEGAN (Baur et al., 2018) has been proposed, in which the authors discuss a spatial variational autoencoder and a discriminator. It is applied to high resolution MRI images for unsupervised lesion segmentation. AnoVAEGAN uses a variational autoencoder and tries to model the normal data distribution that will lead the model to fully reconstruct the healthy data while it is expected to fail reconstructing abnormal samples. The discriminator classifies the inputs as real or reconstructed data. As anomaly score the $l1$ norm of the original image and the reconstructed image is used.

Opposite to reconstruction-based anomaly detection methods as they are discussed above, in (Shen et al., 2020) adGAN, an alternative framework based on GANs, is proposed. The authors introduce two key components: fake pool generation and concentration loss. adGAN follows the structure of WGAN and consists of a generator and discriminator. The WGAN is first trained with gradient penalty using healthy images only and after a number of iterations a pool of fake images is collected from the current generator. Then a discriminator is retrained using the initial set of healthy data as well as the generated images in the fake pool with a concentration loss function. Concentration loss is a combination of the traditional WGAN loss function with a concentration term which aims to decrease the within-class distance of normal data. The output of the discriminator is considered as anomaly score. The method is applied to skin lesion detection and brain lesion detection. Two other methods that utilise discriminator outputs as anomaly score, however not tested for medical imaging, are ALOOC (Sabokrou et al., 2018) and fenceGAN (Ngo et al., 2019). In ALOOC (Sabokrou et al., 2018), the discriminator’s probabilistic output is utilised as abnormality score. In their work an encoder-decoder is used for reconstruction while the discriminator tries to differentiate the reconstructed images from the original ones. An extension of the ALOOC algorithm, is the Old is Gold (OGN) algorithm which is presented in (Zaheer et al., 2020). After training a framework similar to ALOOC, the authors finetune the network using two different types of fake images which are bad quality images and pseudo anomaly images. In this way they try to boost the ability of the discriminator to differentiate normal images from abnormal ones.

In (Ngo et al., 2019) the authors propose an encirclement loss that places the generated images at the boundary of the distribution and then use the discriminator in order to distinguish anomalous images. They propose this loss with the idea that a conventional GAN objective encourages the distribution of generated images to overlap with real images.

In  (Gong et al., 2020) an approach based on the ALOOC algorithm is proposed for the detection of fetal congenital heart disease. However, during training both normal and abnormal samples are available, which is one of the key differences compared to our approach where only healthy subjects are utilised. Furthermore, additional to the encoder-decoder and discriminator networks which are used in ALOOC, they use two additional noise models of the same architecture where the input is an image plus Gaussian noise ($\tilde{x}$) in order to make their encoder-decoder networks more robust to distortions. In (Perera et al., 2019) a one-class generative adversarial network (OCGAN) is proposed for anomaly detection. OCGAN consists of two discriminators, a visual and a latent discriminator, a reconstruction network (denoising autoencoder) and a classifier. The latent discriminator learns to discriminate encoded real images and generated images randomly sampled from $\mathcal{U}\sim(-1,1)$, while the visual discriminator distinguishes real from fake images. Their classifier is trained using binary cross entropy loss and learns to recognise real images from fake images. Finally, in (Pidhorskyi et al., 2018) a probabilistic framework is proposed which is based on a model similar to $\alpha$-GAN. The latent space is forced to be similar to standard normal distribution through an extra discriminator network, called latent discriminator similar to (Rosca et al., 2017). A parameterized data manifold is defined (using adversarial autoencoder) which captures the underlying structure of the inlier distribution (normal data) and a test sample is considered as abnormal if its probability with respect to the inlier distribution is below a threshold. The probability is factorised with respect to local coordinates of the manifold tangent space.
A summary of the key features for the works above is given in Table 1.
To establish consistency between different related works we define $x$ as a test image, $\hat{x}$ as a reconstructed image, $D$ as a discriminator network, ($D_{H}$ as (intermediate) feature representation of a Discriminator network), $E$ as an encoder network (image space $\rightarrow$ latent space), $D\text{e}$ as a decoder network (latent space back to image space), $G$ as a generator network (where input is a noise vector), $z$ as latent space representation and $\lambda$ as a fixed learning rate.

In order to predict an anomaly score $s$, three different strategies are utilised. For an unseen image $x_{unseen}$ and its reconstructed image $\hat{x}_{unseen}$, we utilise as baseline the reconstruction error which is defined as the $l2$ norm, i.e., $s_{rec}=\|x_{unseen}-\hat{x}_{unseen}\|_{2}^{2}$ between image and reconstructed image (residual).

The second candidate for $s$ is the output of the discriminator. $D$ should give high scores for reconstructions of original, normal images, but low scores for abnormal images, $s_{discr}=1-D(x_{unseen})$. Finally, we compute an anomaly score using a gradient-based method, GradCam++,  (Chattopadhay et al., 2018). Inspired by (Kimura et al., 2020)  (Venkataramanan et al., 2019) (Liu et al., 2020) we apply GradCam++ to the score of the discriminator with regards to the last rectified convolutional layer of the discriminator. This produces attention maps and is also valuable for the localisation of the pathology. The intuition of using attention maps for computing anomaly scores, is based on the hypothesis that after training the discriminator not only learns to discriminate between normal and abnormal samples but also learns to focus on relevant features in the image. Thus, specifically for HLHS, where the left artery is missing or is occluded compared to normal samples, a discriminator should identify and locate this difference. The GradCam++ is computed following:

Let $y$ be the logits of the last layer as they are derived from the discriminator network $D(x_{unseen})$. For the same operators $(i,j)$ and $(a,b)$ applied to the feature map $A^{k}$ we compute weights:

where the gradient weights $a_{ij}^{k}$ can be computed as:

and the saliency map (SM) is computed as a linear combination of the forward activation maps followed by a ReLU layer:

We then computed the sum of the attention maps of image $x_{unseen}$ and its reconstruction from the Generator network, $\hat{x}_{unseen}$:

and finally computed the anomaly score $s_{attn}$ as

To compute the anomaly score we encapsulate the information of reconstruction (Kimura et al., 2020). Reconstruction of a normal image should be crisper compared to reconstructions from an anomalous observation. Finally, we attempt to combine anomaly scores, such as $s_{rec}$ with $s_{discr}$. However, the anomaly detection performance does not improve noteworthily.

The available dataset contains 2D ultrasound images of four-chamber cardiac views. These are standard diagnostic views according to (NHS, 2015). The images contain labelled examples from normal fetal hearts and hearts with Hypoplastic Left Heart Syndrome (HLHS) (HLHS, 2019) from the same clinic, using exclusively an Aplio i800 GI system for both groups to avoid systematic domain differences. HLHS is a birth defect that affects normal blood flow through the heart. It affects a number of structures on the left side of the heart that do not fully develop.

Our dataset consists of $2317$ 4-chamber view images for which $2224$ cases are normal and $93$ are abnormal cases. Healthy control view planes have been automatically extracted from examination screen capture videos using a Sononet network (Baumgartner et al., 2017) and manual cleaning from visually trivial classification errors. A set of HLHS view planes that would resemble a 4-chamber view in healthy subjects has been extracted with the same automated Sononet pipeline. Another set has been manually extracted from the examination videos by a fetal cardiologist and 38 cases that are not within 19+0 - 20+6 weeks or show a mix of pathologies have been rejected.

For training, $2131$ 4-chamber view images, which are considered as normal cases are used. During training, only images from normal fetuses are used. For testing, two different datasets are derived for three different testing scenarios:

For $\mathbf{dataset_{1}}$(Figure  2) we use 4-chamber views from all available HLHS cases, extracted by Sononet and cleaned from gross classification errors; in total $93$ cases. Further $93$ normal cases have been randomly selected from the remaining test split of the healthy controls and added to this dataset. HLHS cases are challenging for Sononet, which has been trained only on healthy views. Thus, in HLHS cases, it will only select views that are close to the feature distribution of healthy 4-chamber views, which are not necessarily the views a clinician would have chosen. For $\mathbf{dataset_{2}}$(Figure  2), we use the $93$ normal cases from $dataset_{1}$ and the expert-curated HLHS images from the remaining, nonexcluded $53$ cases. For each of these cases $1$ to $4$ different view planes have been identified as clinically conclusive. With this dataset we perform two different subject-level experiments: a) selecting one of the four frames randomly and b) using all of the 177 clinically selected views in these 53 subjects and fusing the individual abnormality scores to gain a subject-level assessment. We also evaluate per-frame anomaly results.

The images are rescaled to $64\times 64$ and normalised to a $[0,1]$ value range. No image augmentation is used.

For evaluation purposes, the anomaly score is computed as described in Section 3.1. For $\alpha$-GAN and VAE-GAN we use $s_{attn}$, $s_{rec}$ and $s_{discr}$ as anomaly scores as they are presented in Section 3.1.

For comparison with the state-of-the-art we train four algorithms: convolutional autoencoder (CAE) (Makhzani and Frey, 2015; Masci et al., 2011), One-class Deep Support Vector Data Description (DeepSVDD) (Ruff et al., 2018) and f-AnoGAN (Schlegl et al., 2019).

Deep Convolutional autoencoder (DCAE)  (Makhzani and Frey, 2015; Masci et al., 2011) is also trained as a baseline. For training, MSE loss is utilised. For DCAE and One-class DeepSVDD we use the same architectures as the ones used for the CIFAR10 dataset in the original work (Ruff et al., 2018). Reconstruction error, i.e., $\|x-De(E(x))\|_{2}$, is defined as anomaly score ($s_{DCAE}$).

Deep Support Vector Data Description (DeepSVDD) (Ruff et al., 2018) computes the hypersphere of minimum volume that contains every point in the training set. By minimising the sphere’s volume, the chance of including points that do not belong to the target class distribution is minimised. Since in our case all the training data belongs to one class (negative class-healthy data) we focus on (Ruff et al., 2018) . Let $f$ be the network function of the deep neural network with $L$ layers and $\theta^{l}$ the weights’s parameters of the $l_{th}$ layer. We denote the center of the hypersphere as $o$. The objective of the network is to minimize the loss which is defined as:

The center $o$ is set to be the mean of outputs which is obtained at the initial forward pass. The anomaly score ($s_{svdd}$) is then defined at inference stage as the distance between a new test sample to the center of the hyper-sphere, i.e., $\|f(x)-o\|^{2}$

f-AnoGAN (Schlegl et al., 2019) is described in Section 2.2. We were not able to successfully train f-AnoGAN using the same networks as we used for $\alpha$-GAN, hence we utilise similar networks and an identical training framework as described in (Schlegl et al., 2019). We follow the $izi_{f}$ training procedure for the encoder network. As anomaly detection score ($s_{anogan}$) a combination of $L2$ residual loss between the image and its reconstruction and the $L2$ norm of the discriminator’s features of an intermediate layer is utilised as it is defined in Table 1.

In all algorithms the latent dimension is chosen as $128$. We run all experiments $5$ times using different random seeds (Mario Lucic et al., 2018). We report the average precision, recall at the Youden index of the receiver operating characteristic (ROC) curves as well as the average corresponding area under curve (AUC) of the $5$ runs of each experiment. Furthermore, we apply the DeLong’s test (DeLong et al., 1988) to obtain z-scores and p-values in order to test how statistically different the AUC curve of the proposed model compared to the corresponding curves of the state-of-the-art models (CAE and DeepSVDD, f-AnoGAN and VAE-GAN) is. We perform four different experiments:

Experiment 1 uses $dataset_{1}$ and aims to evaluate general, frame-level outlier detection performance, including erroneous classifications and fetuses below the expected age range. In Table 2, the best performing model based on AUC score is the $\alpha$-GAN method using $s_{attn}$ as anomaly score which achieves an average of $0.82\pm 0.012$ AUC. The $\alpha$-GAN model achieves the best precision score. However, regarding F1 score and Recall VAE-GAN outperforms $\alpha$-GAN with $0.88$ and $0.78$ respectively. DeepSVDD shows the best specificity at $0.76$. Figure 3 shows the ROC for the best performing (AUC, F1) initialisation and the distribution of normal and abnormal scores for the best model of $\alpha$-GAN at the Youden index. We present confusion matrices for the $\alpha$-GAN and the VAE-GAN models in Figure LABEL:fig:conf and Figure LABEL:fig:confvae. For normal cases both models achieve similar classification performance. However, for identifying abnormal cases $\alpha$-GAN seems to have an advantage.

Based on the DeLong’s test, for Exp. 1, for the average scores (of five experiments), $\alpha$-GAN compared to f-AnoGAN yields $z=-5.22$ and $p=1.80e-07$. Similarly, the values for $\alpha$-GAN compared to CAE are $z=-4.82$ and $p=1.37e-06$. Finally, comparing $\alpha$-GAN and DeepSVDD results in $z=-6.49$ and $p=8.52e-11$. Since $p<0.01$ for all comparisons, we can assume that $\alpha$-GAN performs significantly better than the state-of-the-art when applied to fetal cardiac ultrasound screening for HLHS. Comparing $\alpha$-GAN with VAE-GAN the values are $z=-1.21$ and $p=0.22$ which does not indicate a significant difference between AUC curves. As can be seen from the results, the GAN-based methods achieve better performance for detecting HLHS.

Experiment 2 uses $dataset_{2}$ for specific disease detection capabilities with expert-curated, clinically conclusive 4-chamber views for 53 HLHS cases. We choose one of the relevant views per subject randomly. Table 3 summarises these results. VAE-GAN has the highest AUC, F1, precision and specificity scores using $s_{attn}$ as anomaly score. Also, we note from Figure LABEL:fig:conf1 and Figure LABEL:fig:conf11 that the VAE-GAN method misclassified less HLHS cases while achieving better performance for confirming normal cases. Average F1 score is $0.89$. Figure 4 shows ROC, anomaly score distribution and confusion matrices at the Youden index of this experiment.

Experiment 3 uses $dataset_{2}$ and is similar to Exp. 2 except that we take all clinically identified views for each subject into account. We average the individual anomaly scores for each frame, depending on the number of frames that are available per subject. VAE-GAN achieves a better AUC score with $0.86$ compared to $0.84$ of $\alpha$-GAN as can be seen in Table 4. However, as can be seen from the confusion matrices (best performing initialisation), $\alpha$-GAN shows a better true positive rate at the cost of a higher number of false positives (Figure LABEL:fig:conf2). This configuration might be preferred in a clinical setting since it reduces the number of missed cases at the cost of a slightly higher number of false referrals.