Before starting continual training, the task module is pre-trained on a labelled data set $\mathcal{L}=\{\langle i_{1},l_{1}\rangle,\dots,\langle i_{L},l_{L}\rangle\}$ of image-label pairs $\langle i,l\rangle$ obtained on a particular scanner (base training). This base training is a conventional epoch based training procedure assuming a static, fully labelled data set. In this training phase samples can be revisited in a supervised training scheme without restriction of the labelling budget. After base training is finished, continual training is started from a model which performs well on data of a single scanner. Continual active training follows the scheme depicted in Figure 2 and outlined in Algorithm 1. First, an input-mini-batch $\mathcal{B}=\{x_{1},\dots,x_{B}\}$ is drawn from $\mathcal{S}$ and the pseudo-domain module evaluates the style embedding (see Section 3.2) of each image $x\in\mathcal{B}$. Based on this embedding, a decision is taken to store $x$ in one of the memories ($\mathcal{O}$ or $\mathcal{M}$) or discard $x$. The fixed sized training memory $\mathcal{M}=\{\langle m_{1},n_{1},d_{1}\rangle\dots,\langle m_{M},n_{M},d_{M}\rangle\}$, where $m$ is the image, $n$ the corresponding label and $d$ the assigned pseudo-domain, holds samples the task network can be trained on. Labels $n$ can only be generated by querying the oracle $\mathbf{o}(x)$, and are subject to the limited labelling budget $\beta$. $\mathcal{M}$ is initialized with a random subset of $\mathcal{L}$ before starting continual training. Pseudo-domain detection is performed within the outlier memory $\mathcal{O}=\{\langle o_{1},c_{1}\rangle\dots,\langle o_{n},c_{n}\rangle\}$, which holds a set of unlabelled images. $o$ represents the image and $c$ is a counter, how long the image is part of $\mathcal{O}$. Details about the outlier memory are given in Section 3.5. Given that training has not saturated on all pseudo-domains, a training step is performed by sampling training-mini-batches $\mathcal{T}=\{\langle t_{1},{u}_{1}\rangle,\dots,\langle t_{T},u_{T}\rangle\}$ of size $T$ from $\mathcal{M}$, and training the task module (Section 3.3) for one step. This process is continued by drawing the next mini-batch $\mathcal{B}$ from $\mathcal{S}$.

CASA does not assume direct knowledge about domains (e.g. scanner vendor, scanning protocol). Therefore, in the pseudo-domain module the style of each image $x$ is evaluated and $x$ is assigned to a pseudo-domain. Pseudo-domains represent groups of images which exhibit similar style. A set of pseudo-domains $\mathcal{D}=\{\langle c_{1},d_{1},\bar{p}_{1}\rangle\dots\langle c_{D},d_{D},\bar{p}_{D}\rangle\}$ is defined by their style embedding center $c_{j}$ and the maximum distance $d_{j}$ from $c_{j}$ that an image is considered belonging to pseudo-domain $j$. In addition, a running average $\bar{p}_{j}$ of the performance on $j$ for each pseudo-domain $j\in\{1,\dots,D\}$ is stored.

The task module is responsible for learning the target task (e.g. cardiac segmentation), where the main component of this module is the task network ($\mathbf{t}(x)\mapsto y$), mapping from input image $x$ to target label $y$. During base training, this module is trained on a labelled data set $\mathcal{L}$. During continual active training, the module is updated in every step by drawing $n$ training-input-batches $\mathcal{T}$ from the memory $\mathcal{M}$ and performing a training step on each of the batches. The aim of CASA is to train a task module performing well on images of all image acquisition settings available in $\mathcal{S}$ without suffering catastrophic forgetting.

The $M$ sized training memory $\mathcal{M}$ is balanced between the pseudo-domains currently in $\mathcal{D}$. Each of the $D$ pseudo-domains can occupy up to $\frac{M}{D}$ elements in the memory. If a new pseudo-domain is added to $\mathcal{D}$ (see Section 3.5) a random subset of elements of all previous domains is flagged for deletion, so that only $\frac{M}{D}$ elements are kept protected in $\mathcal{M}$. If a new element $e=\langle m_{k},n_{k},d_{k}\rangle$ is inserted to $\mathcal{M}$ and $\frac{M}{D}$ is not reached, an element currently flagged for deletion is replaced by $e$. Otherwise the element will replace the one in $\mathcal{M}$, which is of the same pseudo-domain and minimizes the distance between the style embeddings. Formally, the element with index $\xi$ is replaced:

The outlier memory $\mathcal{O}$ holds candidate examples that do not fit an already identified pseudo-domain, and might form a new pseudo-domain by themselves. Whether they form a pseudo-domain is determined based on their proximity in the style embedding space. Examples are stored until they are assigned a new pseudo-domain, or if a fixed number of training steps $z$ is reached. If no new pseudo-domain is discovered for an image within $z$ steps, it is considered a ’real’ outlier and removed from the outlier memory. Within $\mathcal{O}$, new pseudo-domains are discovered, and subsequently added to $\mathcal{D}$. The discovery process is started when $\lvert\mathcal{O}\rvert=o$, where $o$ is a fixed threshold of minimum elements in $\mathcal{O}$. To detect a dense region in the style embedding space of samples in the outlier memory, the pairwise euclidean distances of all elements in $\mathcal{O}$ are calculated. If there is a group of images for which all pair-wise distances are below a threshold $t$, a new pseudo-domain is established by these images. For all elements belonging to the new pseudo-domain, labels are queried from the oracle and they are transferred from $\mathcal{O}$ to $\mathcal{M}$.

Throughout the experiments, five methods were evaluated and compared:

1. 1.

   Joint model (JM): a model trained in a standard, epoch-based approach on samples from all scanners in the experiment jointly.

2. 2.

   Domain specific models (DSM): a separate model is trained for each domain in the experiment with standard epoch-based training. The evaluation for a domain is done for each domain setting separately.

3. 3.

   Naive AL (NAL): a naive continuously trained, active learning approach of labelling every $n$-th label from the data stream, where $n$ depends on the labelling budget $\beta$.

4. 4.

   Uncertainty AL (UAL): Is a common type of active learning which labels samples where the task network is uncertain about the output (Budd et al., 2019). Here, uncertainty is calculated using dropout at inference as an approximation for Bayesian inference (Gal and Ghahramani, 2016).

5. 5.

   CASA (proposed method): The method described in this work.

Joint models and DSM require the whole training data set to be labelled, and thus are an upper limit to which the continual learning methods are compared to. CASA, UAL and NAL use an oracle to label specific samples only. The comparison to NAL and UAL evaluates if the detection of pseudo-domains and labelling based on them is beneficial in an active learning setting. Note, that the aim of our experiments is to show the gains of CASA compared to other active learning methods, not to develop new state-of-the-art methods for either of the three tasks evaluated.

We evaluate different aspects of CASA:

1. 1.

   Performance across domains: For all tasks, we evaluate the performance across domains at the end of training, and highlight specific properties of CASA in comparison to the baseline methods. Furthermore, we evaluate the ability of continual learning to improve accuracy on existing domains by adding new domains backward transfer (BWT), and the contribution of previous domains in the training data to improving the accuracy on new domains forward transfer (FWT) (Lopez-Paz and Ranzato, 2017). BWT measure how learning a new domain influences the performance on previous tasks, FWT quantifies the influence on future tasks. Negative BWT values indicate catastrophic forgetting, thus avoiding negative BWT is especially important for continual learning.

2. 2.

   Influence of labelling budget $\beta$: For cardiac segmentation, the influence of the $\beta$ is studied. The labelling budget is an important parameter in clinical practice, since labelling new samples is expensive. We express $\beta$ as a fraction of the continual data set. Different settings of $\beta$ are analysed $\beta=\frac{1}{5}$, $\beta=\frac{1}{8}$, $\beta=\frac{1}{10}$ and $\beta=\frac{1}{20}$. To solely study the influence of $\beta$, the memory size in this experiments is fixed $M=128$ for all settings.

3. 3.

   Influence of memory size $M$: For cardiac segmentation different settings for the memory size $M$ are evaluated. The memory size is the number of samples that are stored for rehearsal, and might be limited due to privacy concerns and/or storage space. Here, $M$ is evaluated for $[64,128,256,512,1024]$ and a fixed $\beta=\frac{1}{10}$. We assume that the diversity of the data addressed by CASA is a result of the set of scanners, not the number of images in the dataset, therefore $M$ is fixed to specific numbers rather than a fraction of the dataset.

4. 4.

   Memory composition and pseudo-domains: We study if our proposed method of detecting pseudo-domains is keeping samples in memory that are representative of the whole training set for cardiac segmentation. In addition, we evaluate how the detected pseudo-domains are connected to the real domains determined by the scanner types.

5. 5.

   Learning on a random stream: We study how CASA is performing on a random stream of data, where images of different acquisition settings are appearing randomly in the data stream. In contrast to the standard setting, where these acquisition settings appear subsequently with a phase of transition in between.

Here, the quantitative results at the end of the continual training are compared for a memory size $M=128$ and a labelling budget of $\beta=\frac{1}{10}$. Different settings for $M$ and $\beta$ are evaluated in Section 5.2 and 5.3 respectively.

The influence of $\beta$ on cardiac segmentation performance is shown in Figure 3. For the first scanners C1 and C2, a similar performance can be observed for all methods and values of $\beta$. This was due to the fact that all methods have a sufficient amount of budget to adapt from C1 to C2. For C3, the performance for CASA was slightly higher compared to UAL and NAL. The most striking difference between the methods can be seen for scanner C4. There, CASA performed significantly better for $\beta=\frac{1}{5}$ and $\beta=\frac{1}{8}$. For $\beta=\frac{1}{10}$ CASA outperformed UAL and NAL on average however, a large deviation between the individual five runs is observable. Investigating this further revealed that CASA ran out of budget before C4 data appeared in the stream for one of the random seeds. Thus it was not able to adapt to C4 for this random seed. For $\beta=\frac{1}{20}$, CASA consumed the whole labelling budget before C4 data occured in the stream. Thus, it was not able to adapt to C4 data properly. UAL only had little budget left when C4 data came in and runs out afterwards thus, the performance was significantly worse to the results with more labelling budget. NAL performed the best for the setting with the lowest budget $\beta=\frac{1}{20}$, due to the fact that NAL labels every 20th step and thus did not run out of budget until the end of the stream is reached.

For cardiac segmentation, we investigate the influence of the training memory size $M$.

The memory size $M$ influences the the adaption to new domains the the continuous data stream. In Figure 4, CASA, UAL and NAL are compared for $M=\langle 64,128,256,512,1024\rangle$. For the first scanners C1 and C2 in the stream, the performance was similar across AL methods and settings of $M$. For $M=64$, CASA could not gain any improvement in comparison to UAL and NAL, meaning that the detection of pseudo-domains and balancing based on them is more useful for reasonable large memory sizes. All methods performed best for $M=64$ however, the memory on the end of the stream was primarily filled with C3 and C4 data, which would lead to forgetting effects if training continues. In addition, a performance drop for $M>=128$ compared to $M=64$ for C4 data can be observed. This is a sign that the higher the memory size, the longer it takes for all methods to adapt to new domains. At the end of the data stream training for C4 data has not saturated for a rehearsal memory of size 128 and larger. The large variation in performance of CASA on C4 data across different sizes of $M$ might be due to the early consumption of the whole labelling budget. For each independent test run, the ordering of the continuous data stream is randomly changed, some of those orderings led to CASA running out of budget before the stream ended, thus the adaptation to C4 data was not completed.

We analyzed the balancing of our memory at the end of training (with $M=128,\beta=\frac{1}{10}$) and the detection of different pseudo-domains by extracting the style embedding for all samples in the training set (combined base and continual set). Those embeddings were mapped to two dimensions using t-SNE (Maaten and Hinton, 2008) for plotting. In Figure 5 (a), it is observable that different scanners are located in different areas of the embedding space. Especially scanner C4 forms a compact cluster separated from the other scanners. Furthermore, a comparison of the distribution of the samples in the rehearsal memory at the end of training (Figure 5(b)) shows that for CASA, the samples distributed over the whole embedding including scanner C4. For UAL and NAL, most samples focused on scanner C1 samples (base training scanner), and a lower number of images of later scanners were kept in memory. Note, that this does not mean that UAL and NAL labelled primarily scanner C1 samples but that those methods did not balance the rehearsal memory. So, labelled images from C2-C4 might be lost in the process of choosing what samples to keep. As shown in Appendix Figure 8, those observations were stable over differnt test runs. Figure 5 (c) confirms the finding. Here, the memory distribution for the compared methods over five independent runs (with different random seeds) demonstrated the capability of CASA to balance the memory across scanner, although the real domains are not known during training.

Pseudo-domain discovery in CASA (with $M=128$, $\beta=\frac{1}{10}$) resulted in 6-8 pseudo-domains, showing that the definition of pseudo-domains does not exactly capture the definitions of domains in terms of scanners. In addition, the detection was influenced by the order of the continuous data stream. Figure 6 shows the distribution of images from certain scanners in the pseudo-domains for one training run of CASA (results for all $n=5$ independent runs are shown in Appendix Figure 9). The first two pseudo-domains were dominated by samples from scanner C1, while the last pseudo-domain consisted mainly of scanner C4. The pseudo-domains 3-6 represented a mix of C2 and C3 data. This is consistent with Figure 5 where we see that the distributions of C2 and C3 overlap while C1 and especially C4 data is more separated.

To analyze the influence of the sequential nature of the stream, we show how CASA performs on a random stream of data, with no sequential order of the scanners used. Results are given in Table 5. Note, that in this comparison adding randomness to the stream results in eliminating the domain shifts between scanners and making the samples within the stream approximately independent and identically distributed (i.i.d.). CASA performed well on a random stream however, for scanners with few samples in the training set (Scanner C2, C4), a drop in performance was observed. This is due to the fact that the pseudo-domain detection based on the outlier memory was not as effective as on a continuous stream, where we expect a accumulation of outliers as new domains start to occur in the stream.

NAL performed well on a random stream, outperforming NAL on a ordered stream, as well as CASA. Due to the randomness in the stream and the sampling strategy of NAL (taking every $n$-th sample), NAL learned on a diverse set of samples and managed to reach a more balanced training. Similar observations hold for uncertainty based active learning. Mixing up the order of samples in the stream leads to an earlier occurrence of data from C3 and C4, thus a better rehearsal set can be constructed with UAL. This highlights the ability of CASA to learn under the influence of domain shifts. If no domain shifts occur in the data the specific design of the approach does not provide a benefit.