We now explore how the learned dataset-specific scale and shift parameters of the fully-trained network can be used to identify relationships between annotation styles across different datasets. Understanding these relationships would not only improve data collection methods and make for more fair performance evaluations, but would enable strategic pooling of compatible styles. This type of strategic pooling is especially important for research projects with limited datasets where each style may only have a few labelled samples.

To identify relationships, we calculate the cosine similarity (normalized dot product) of the parameters between all datasets for all CIN layers in the network in order to determine the relevant relationships and where they occur within the network. Specifically, the cosine similarity between two sources, $s$ and $\tilde{s}$, is computed as follows:

where $n$ is the specific CIN layer in the network, and $s$ and $\tilde{s}$ represent different sources ³³3Since the scale parameters are initialized at 1, and a scale of 1 is representative of the parameter having no effect on the activation, we subtract 1 from all scale vectors before performing the cosine similarity calculation. This effectively relocates the origin to the initialization point of all scale parameters allowing for a more fair evaluation of the parameter changes that occurred during training..

By analysing the cosine similarity between dataset-specific parameters, we can quantify high dimensional direction-based relationships between the scale and shift parameter vectors of the different datasets. An additional analysis, that considers magnitude, is performed by comparing the euclidean norm of these vectors. We attempt to look at the norms to detect any noticeable clusters or trends that may go undetected in the cosine similarity analysis.

Beyond modeling dataset-specific biases, we propose an image conditioning mechanism to model image-related detection biases. To do so, we propose an image conditioning mechanism based on FiLM Perez et al. (2018) consisting of two key components: (1) An image encoder to generate a latent representation of the image; (2) A linear layer for each CIN layer to transform the latent representation of the image into a set of affine instance normalization parameters (per channel scale and shift). By doing this, the model is able to adapt the style of the output segmentation based on (global) image characteristic(s). The full architecture can be found in Figure  9.

We train a modified nnUNet (Isensee et al., 2018) for the task of multiple sclerosis (MS) T2-weighted lesion segmentation. The primary modifications include the addition of dropout, and replacing the standard Instance Normalization layer in each convolutional block with a CIN layer. This architecture is shown in detail in Figure 3. All models in this paper are trained using Binary Cross Entropy (BCE) Loss and Adam optimizer (Kingma and Ba, 2014). Random affine and random contrast augmentations are also used to prevent over fitting.

In this work, we leverage a large, proprietary dataset consisting of images acquired from MS patients from six different multi-centre, multi-scanner clinical trials. In this context, each trial serves as a different dataset. Each trial contains multi-modal MR sequences of patients with one of three different MS subtypes: Secondary-Progressive (SPMS), Primary-Progressive (PPMS), or Relapsing Remitting (RRMS). RRMS patients suffer from recurrent attacks of neurological dysfunction, usually followed by at least partial recovery (Hurwitz, 2009). SPMS may eventually develop in patients with RRMS. SPMS is characterized by progressive clinical disability in the absence of relapses. The transition from RRMS to SPMS is gradual and not clearly defined. PPMS is characterized progression in the absence of relapses from the onset of the disease, i.e., without a relapsing remitting stage. SPMS and PPMS subjects are about 10 years older than RRMS patients on average. SPMS patients have larger lesion volumes than RRMS and PPMS patients (Lublin et al., 2014).

For all trials, each patient sample consists of 5 MR sequences acquired at 1mm $\times$ 1mm $\times$ 3mm resolution: T1-weighted, T1-weighted with gadolinium contrast, T2-weighted, Fluid Attenuated Inverse Recovery (FLAIR), and Proton Density. In each trial, images were collected across multiple different sites, and multiple different scanners ($\scriptstyle\sim$80 per trial), therefore there were no consistent acquisition differences between trials. T2 lesion labels were generated at the end of each clinical trial, and were produced by an external company, where trained expert annotators manually corrected the output of a proprietary automated segmentation method. Since the trials were completed at different times, labels may have been generated with different versions of the automated segmentation method. Although different expert raters corrected the labels, there was overlap between raters across trials and all raters were trained to follow a similar labelling protocol. Each image has one associated label (i.e. one label style, unlike in inter-rater bias studies). Furthermore, within one trial, the same labelling process was followed in order to keep the labels within the trial consistent. All imaging data was re-processed using a single pre-processing pipeline, resulting in nearly indistinguishable intensity distributions across trials. While image-based biases were effectively normalized out after re-processing, the image effects would have influenced the labels during the original labelling process through the semi-automated algorithm. In this study, the need to model trial-specific biases stems primarily from subtle differences that accumulate throughout the labelling process. How these differences translate into the resulting annotation styles has not been extensively studied, despite the impact this can have on research into related problems (e.g. domain adaptation).

Automatic T2 lesion segmentation methods are evaluated with voxel-level segmentation metrics. To evaluate performance, we use DICE (Segmentation F1 score) (Powers (2008)).⁴⁴4Operating point (threshold) is chosen based on the best segmentation DICE. We also present Precision Recall Area Under Curve (PR-AUC) for all experiments in the Appendix to provide results independent of threshold selection. For relevant experiments, detection F1 is also provided. Detection F1 is obtained by performing a connected component analysis on the segmentation mask to group lesion voxels in an 18-connected neighbourhood to identify individual lesions Nair et al. (2020).

In this section, experiments are conducted to examine the CIN model and its benefits in situations with variable annotation styles. This series of experiments is conducted with the described 6 clinical trial datasets. To maintain balance, an equal number of patients were taken from each clinical trial, resulting in 390 patients per trial, totalling 2340 patients in the aggregated dataset. This dataset was then then divided (on a per-trial basis) into non-overlapping training (60%), validation (20%), and testing (20%) sets. In these experiments, the CIN model is conditioned on the trial, termed the trial-conditioned model. We also train single-trial models where only one dataset is used in training, and a naive-pooling model where all datasets are used but the model is blind to the source of the data. Each model is individually hyperparameter tuned to optimize performance on the validation sets, thus the naive-pooling, single-trial, and conditioned models are all tuned separately to enable fair comparisons.

Table 1(a) demonstrates the problem of different annotation styles learned by single-trial models (see Appendix A for more results). These models show a clear performance degradation of up to 10% when applied to different trial test sets. This explicitly shows the issue of comparing models trained on one annotation style to ground truths generated in a conflicting annotation style. Since we only have one annotation style per trial, it is essentially impossible to accurately and fairly evaluate a single-trial model on another trial with a different annotation style.

Comparisons of the DICE segmentation scores for the single-trial models, naive-pooling model, and the proposed trial-conditioned model are shown in Table 1(b). The lower overall performance of the naive-pooling model (as is common practice) demonstrates another consequence of variable annotation styles. Specifically, despite a 500% increase in the size of the training set, the naive-pooling model under performs the single trial baselines in nearly all cases, effectively eliminating the expected benefit of using more data. This is likely because the naive-pooling model has no information regarding the annotation style that is expected for any given sample and when tested on each trial, is unable produce a label in the required style.. On the other hand, provided with context on the source of each patient sample, the trial-conditioned model shows performance on par with the single-trial models, illustrating CIN’s ability to learn trial-specific annotation styles. The final trial-conditioned model is also capable of producing any annotation style for any input, thus providing researchers with multiple “opinions” on a given sample, as shown in Figure 4.

Figure 4 shows the results for a test sample from one trial (RRMS-B) as segmented by the trial-conditioning model using CIN parameters from several different trials. The results clearly demonstrate unique segmentation styles across trials. One observation that can be made is that conditioning on the SPMS-B, PPMS-A and PPMS-B trials results in a noticeable relative over-segmentation on some of the larger lesions in all three cases. Using these three label styles also results in completely missing the lesion in the posterior part of the white matter tract located in the brain’s midline. The similarity in the results across the SPMS-B, PPMS-B, and PPMS-A segmentation maps suggests that they form a potential annotation style subgroup. On the other hand, the RRMS-A and SPMS-A segmentation styles are similar to that of RRMS-B and therefore result in very accurate results according to the RRMS-B label. This suggests that these three styles make up another subgroup. While these groupings are noticeable in the qualitative results, we will show later in Section 5.2 that these groupings are also revealed by our CIN parameter analysis.

In addition to qualitative results, the relationship between these sets of trials (SPMS-B, PPMS-B, PPMS-A) and (RRMS-A, SPMS-A, RRMS-B) is also demonstrated in the quantitative results presented in Table 2. This table shows the DICE performance on the test set for the conditioned model using the different trial styles. These results show that the best SPMS-B performance is obtained not only by the SPMS-B style, but also by the PPMS-A style. The PPMS-B style results in a SPMS-B performance that is intermediate between that of the SPMS-B/PPMS-A style and the RRMS-A/RRMS-B/SPMS-A style, suggesting that the PPMS-B style could be considered somewhat unique, depending on how the groupings are defined. The best RRMS-B performance is achieved by the RRMS-B, SPMS-A, and RRMS-A styles. The best PPMS-A performance is achieved by the SPMS-B style. These results further confirm a strong relationship between the trial labelling processes. The differences between the labelling processes are distinct but complex, and despite this, the scaling/shifting achieved by CIN parameters are able to capture these unique styles.

We further explore and identify relationships between the trial styles by exploiting relationships in the learned CIN parameters (the trial-conditioned parameters in the CIN layer) from the trained trial-conditioned model discussed in Section 5.1. Recall that there are a set of learned CIN parameters for each trial in the training set. The CIN parameters consist of a scale and shift parameter, each of size [1, channels, 1, 1, 1], and there are two CIN layers per block in the network, rendering analysis challenging. In order to effectively evaluate relationships between high dimensional per-trial parameters, we take two approaches: 1) cosine similarity analysis, and 2) euclidean norm analysis.

In the cosine similarity analysis, we calculate the metric as described in Section 3.1. This analysis allows us to identify similarities between the directions of the CIN parameters, with $+1$ indicating the vectors are in the same direction in high dimensional space and $-1$ meaning they are in the opposite direction. This method allows us to gauge similarities between scales and shifts per layer, between each pair of trials. For the vector norm analysis, we plot the euclidean norm of the scale against the euclidean norm of the shift parameter on one plot for each layer, per-trial. This analysis results in 14 scatter plots where each point is the euclidean norm of the (scale, shift) for a different trial. This qualitative analysis allows for visualization of the different relationships considering both magnitudes. For the purpose of brevity, only one scatter plot per section of the network (encoder, center, decoder) is provided.

Figure 5 provides a visualization of the (pairwise) cosine similarity analysis for SPMS-A with respect to all other trials. This result reveals the subgroup trends noted in Section 5.1, where SPMS-A shows distinct similarity in annotation style with RRMS-A and RRMS-B. The analysis results for all other trials are shown in Appendix A. The same relationships are confirmed in all other trial comparisons, where two distinct style subgroups are discovered: (1) SPMS-B, PPMS-A, and PPMS-B, and (2) RRMS-A, RRMS-B, and SPMS-A. The main commonality between the groups is that the trials were labelled using different versions of the label generation pipeline, indicating that the label generation pipeline was the main contributor to annotation bias between these particular trials. This relationship is further shown in the scatter plots in Figure 6 where the two groups form visible clusters. Figure 6 shows proximity between the PPMS-A/PPMS-B/SPMS-B group with some distance from the cluster formed by the RRMS-A/RRMS-B/SPMS-A group.

Other than the annotation style group identification, the cosine similarity analysis also uncovered other trends. Interestingly, the shifts are more similar across all trial comparisons than the scales, suggesting high discriminatory importance of the scale parameter in the CIN formula. The center layers were more similar across trials relative to the rest of the network layers, especially in the bias parameters. This could mean that there may not be a huge distinction between trials when they are encoded into a (high-level) latent space.

Lastly, the PPMS-B trial results in Figure 13 shows very little strong relation or opposition to most of the trials. Although the similarity with PPMS-A and SPMS-B still noticeable, it is much less extreme than the similarity between just PPMS-A and SPMS-B. Figure 6 also shows that the PPMS-B point strays further from PPMS-A and SPMS-B. Figure 6 b) especially shows PPMS-B very astray from all other trial parameters. This trial’s unique annotation could be due to a number of reasons, including unique changes or updates to the labelling or preprocessing pipeline, or a specific labelling protocol required for a particular study. These possible changes could all result in PPMS-B’s unique CIN parameters. Although similar to SPMS-B and PPMS-A, these trends lead us to believe the PPMS-B trial could be considered its own group.

These identified groupings contradict some of the clinical similarities between trials, specifically the disease subtype. While one might think that naively pooling trials of the same disease type would be appropriate, our analysis shows the importance of considering other factors in the labelling process. In the case of this study, the likely cause of the differences between trials was a difference in the overall labelling process and in particular, the corresponding semi-automated labelling algorithm. The significant difference in the labelling process highlights the underlying ambiguity in MS lesion boarders. As previously noted, the non-lesional DAWM which can be found adjacent to both focal lesions and healthy tissue makes delineating lesion boarders or identifying smaller lesions somewhat subjective. The intensity distribution between DAWM and the focal lesions overlap, thus forcing both annotation software and human raters to decide on an arbitrary threshold to discretize what is in fact a continuous healthy-to-pathology transition (Seewann et al., 2009). Making assumptions about annotation styles due to prior knowledge may therefore mislead researchers and result in unfair analyses or comparisons between different datasets.

The previous section outlined a strategy to explore annotation style relationships between trials by analysing similarities in the CIN parameters. The experimental results of the CIN parameter analysis indicated a strong relationship between the RRMS-A, RRMS-B, and SPMS-A trials. SPMS-B and PPMS-A also showed high CIN parameter similarities, and PPMS-B was identified as unique. To validate the conclusions obtained from this method, another series of experiments using the identified groups are conducted in this section. Here, a single model is trained on each identified group (Group-pooling) without any conditioning. Another model is trained with conditioning on the identified trial groupings. This Group-Conditioned model was designed such that the trials within an identified group all share CIN parameters through the entire network during training. This results in one set of CIN parameters for SPMS-A, RRMS-A, and RRMS-B, and another set of parameters for SPMS-B and PPMS-A, and lastly a set for PPMS-B.

When naively pooling trials within the same identified group (group-pooling), the model does not suffer from the same performance degradation previously demonstrated in the all-trial naive-pooling experiments. In Table 3(a), the resulting performance on the Group-Pooled model trained on SPMS-A, RRMS-A, RRMS-B group matches and slightly outperforms their single-trial baselines (see Table 1(a)). The same is true for the SPMS-B and PPMS-A group-pooled model. These results confirm that the groups identified from the CIN parameter analysis do in fact have significant similarities between their annotation styles. The group-conditioned model also matches performance of the trial-conditioned model, further confirming the groups identified from the parameter analysis. These annotation style groups are especially beneficial for cases where each dataset is very small such that training on each independent dataset does not yield adequate performance in the desired annotation style.

Table 3(b) also demonstrates that the grouped-conditioned model is successfully able to learn distinct styles even though several trials share CIN parameter sets. When using different annotation styles (conditioning on the “wrong” group), DICE scores suffer up to 5% performance degradation. This performance degradation further stresses the importance of understanding sources of annotation styles and their relationships. If researchers make assumptions about the sources of annotation styles, for example if they assume a disease-subtype cohort bias is the main cause of the annotation style, they can make incorrect groupings and combine datasets with incompatible annotation styles. For example, in this case, SPMS-B and PPMS-A are not the same disease subtype, but they have compatible annotation styles. Contrarily, PPMS-B is the same subtype as PPMS-A, but as shown in Table 3(b), they have somewhat incompatible annotation styles resulting in the aforementioned performance degradation.

These experiments also further argue the point that evaluating on subjective annotation styles is not truly fair. Although there is notable performance degradation when using the wrong conditioning/style, the model is clearly adequate as it is capable of performing well when it instructed to generate the desired annotation style. This finding emphasizes the importance of reconsidering how supervised models are evaluated in new settings.

This set of experiments examines whether CIN is able to learn an isolated non-linear annotation style within a single dataset. In this experiment, we ensured that the source biases arose solely from different labelling protocols, while keeping all other factors, including disease stage, patient cohort, and time of collection, constant. To that end, only the SPMS-A dataset was used. For this experiment, the dataset consisted of 500 SPMS-A patient images. Half of the dataset was modified by changing the segmentation labels such that all small lesions (10 voxels or less) were set to the background class, essentially removing them from the label. This can be thought of as being equivalent to a labeling protocol that purposefully ignores small lesions for clinical purposes (Trial-MSL). The labels of the remaining half of the dataset were not modified in any way (Trial-Orig). These datasets were then each split into non-overlapping training (60%), validation (20%), and testing (20%) sets. Similarly to the previous experiments, we train Single-Trial models, a Naive-Pooling model, and a CIN Trial-Conditioned model.

Table 4 shows the test results for this set of experiments on a non-modified Trial-C test set (Trial-Orig). We show detection results specific to small lesion detection in order to demonstrate CINs ability to learn an annotation style that ignores small lesions. Results show that the Single Trial-MSL model, which is trained on data with a significantly different annotation style than the test data, exhibits poor small lesion detection performance. The degradation in performance is expected as Trial-MSL has small lesions labeled as background, while the test set (Trial-Orig) has small lesions marked as lesions. The poor detection performance is not because the trained model is “bad”, as one may conclude from the results, but because the definition of “ground truth” is different between the training set and the test set. The Naively-Pooled model, which is trained on both Trial-Orig and Trial-MSL was unable to learn the MSL style and suffered significantly in segmentation performance due to its inability to accommodate two conflicting annotation styles. The model trained using the CIN approach was able to adapt to the difference in annotation styles and exhibit good lesion-level detection and voxel-level segmentation performance when conditioned on the appropriate dataset. Looking at the Trial-conditioned model conditioned on Trial-MSL, we see that CIN is able to learn the Trial-MSL label bias quite effectively, and is able to ignore small lesions while maintaining voxel-level segmentation performance. This experiment shows the importance of providing context to the model regarding the desired annotation style in the test set.

We can also study the impact of a pure annotation-based bias on the model by examining the CIN parameters as in Section 5.2. Figure 8 clearly shows notable differences between the MSL and Orig trials, particularly in the scale parameter. These results demonstrate the impact annotation styles can have on network parameters, and further show CINs capacity to model complex differences in the label space. Interestingly, even though the bias was only in the annotations, there are very low cosine similarity values throughout the encoder. This is significant as it demonstrates that in order to model changes in the annotation style, the model makes use of capacity throughout the entire network. This also indicates that patients with different annotations styles (although potentially very similar in image appearance) can have completely different latent representations, thus highlighting the issue with some domain invariant methods that don’t account for annotation style differences.

We now evaluate CINs ability to learn new annotation styles by mimicking a clinical situation where a new trial is introduced. In this experiment, we had two trials of “known” bias, SPMS-B and RRMS-A, and we are provided with a few samples of a new “unknown” trial, SPMS-A. We took two pre-trained models: one naively-pooling with SPMS-B and RRMS-A, and another CIN Trial-Conditioned model trained on only SPMS-B and RRMS-A. The affine parameters of the normalization layers were then fine-tuned using only 10 labeled samples from the SPMS-A dataset. Segmentations were done on a held-out test set from SPMS-A. In these experiments, we used 1000 SPMS-B samples and 1000 RRMS-A samples split into non-overlapping training (60%), validation (20%), and testing (20%) sets. For SPMS-A, 10 samples were used for training, 100 were used for validation, and 100 for testing.

Table 5 depicts the results from this set of experiments. The performance of the naively-pooled model improves when the instance normalization parameters of the model are fine-tuned compared to no fine-tuning. Furthermore, a CIN-pooled model shows good SPMS-A performance when conditioned on RRMS-A, which makes sense given the proven relationship between RRMS-A and SPMS-A. By fine-tuning the trial-specific CIN layer parameters of this model on 10 samples of SPMS-A, we are able to then condition the model on SPMS-A during test time. This leads to the highest performance improvement over all models. This experiment series shows how CIN can be leveraged in the clinic to quickly learn and accommodate the annotation style needed for the desired task. Additionally, it shows the need to consider annotation style in the evaluation or implementation of models, as poor results on a test set with a different annotation style can mislead researchers, who may otherwise assume that the performance drop stems from a difference in the image distribution (e.g. different scanners, etc.). As previously established, the RRMS-A style is compatible with the SPMS-A style, thus resulting in the relative good performance of the trial-conditioned model in RRMS-A-mode. Without considering subjective annotation styles, researchers may interpret this result as meaning the trial-conditioned model in RRMS-A-mode is better; however, it is simply because the RRMS-A style is more similar to the SPMS-A styled labels used in the test evaluation.

Figure 8 shows the CIN Cosine Similarity analysis on the fine-tuning experiments ⁵⁵5Note that these experiments were done on an older version of our dataset, where the trials were preprocessed differently and not normalized as vigorously, which may account for trends in the CIN parameter analysis that do not corroborate the results presented in 5.2. This analysis provides interesting insights about what CIN can learn in a few-shot fine-tuning problem. The fine-tuned SPMS-A parameters do not show particular relationship with either SPMS-B or RRMS-A. There are slightly more negative values in the SPMS-A $\cdot$ SPMS-B Scale parameters than there are in the SPMS-A $\cdot$ RRMS-A Scales, but this is not a clear cut relationship. This pattern does confirm some of the results in Table 5 as the RRMS-A CIN parameters yield better performance on the MAESTRO test set than do the SPMS-B. These figures may indicate that 10 samples is not enough for the algorithm to fully learn the annotation style, but the performance results do suggest that 10 is enough for the model to obtain reasonable performance. With only 10 samples, the algorithm is still able to perform better than it did than with only the SPMS-B and RRMS-A training samples. This information can come in the form of a few labelled samples in desired style, or in the form of clinically relevant factors. A hypothetical case for the latter would be if we knew that the RRMS-A style data was better at detection, and the SPMS-B style for lesion volume estimation, and the task of SPMS-A was lesion volume estimation, we would then make the call to use the style of the most relevant dataset, SPMS-B.

In previous sections, we focused on learning and analyzing trial-specific biases. These biases were primarily in the label-space, and were primarily a consequence of known changes to the automatic segmentation method used across datasets. In this section, we examine whether the Image Conditioning mechanism proposed in Section 3.2 can model synthetic detection biases that can be derived from the image itself. To that end, we consider a scenario where the goal of a hypothetical study is to evaluate the effect of a particular drug on MS lesion activity in brain MRI. In this hypothetical scenario, we would be concerned with two key factors in order to evaluate the effect of treatment, principally: (1) Changes in T2 lesion volume between scans; and (2) The presence of gadolinium-enhancing lesions in brain MRI. Typically, significant increases in lesion volume or the presence of gadolinium-enhancing lesions indicates treatment failure Freedman et al. (2020). Practically speaking, gadolinium-enhancing lesions can be identified in the T1C sequence at a glance and do not require computing and comparing lesion volumes across multiple scans. In a resource constrained scenario, it is conceivable that a rater, if gadolinium-enhancing lesions are clearly present, would not bother to make corrections to the T2 segmentation label produced by the automated segmentation algorithm, as the presence of a gadolinium-enhancing lesion is sufficient to indicate treatment failure. In this scenario, subjects that exhibit gadolinium-enhancing lesions would exhibit a different bias in the “ground truth” T2 lesion labels than those subjects that did not exhibit gadolinium-enhancing lesions. The patients belonging to the gadolinium-enhancing cohort would therefore require much more careful corrections to the T2 segmentation labels produced by the automated segmentation algorithm in order to determine whether the treatment was successful in suppressing the accumulation of additional T2 lesions. This example demonstrates a cohort-specific detection bias which, despite being correlated with the image, cannot be easily modeled given that dependencies are long range (Gadolinium enhancing lesion can appear anywhere in the brain).

Experimentally, to evaluate the ability of this mechanism to model synthetic detection biases, we induce a synthetic label bias to simulate the scenario we describe above. Specifically:

• •

  If a patient has gadolinium-enhancing lesions, we dilate the T2 lesion labels.

• •

  If a patient does not have gadolinium-enhancing lesions, the T2 lesion labels are untouched.

Dilation is chosen since this emulates the scenario we describe above, where the rater would not bother to diligently delineate the edges of lesions in patients with gadolinium-enhancing lesions⁶⁶6Generally, raters correct the output of an automated method by removing false positives (rather than identifying false negatives).. This synthetic dataset is generated using 1000 RRMS-A samples split into non-overlapping training (60%), validation (20%), and testing (20%) sets. Overall, the proportion of patients that had gadolinium-enhancing lesions was approximately 34%.

Table 6 depicts the results from this set of experiments. The performance of the naive-pooling model underperforms both single-trial baselines, with performance on the w/o GAD dataset suffering a dramatic performance hit. Clearly, the naive-pooling model is unable to learn the relationship between the presence of Gadolinium-enhancing lesions and the dilation bias while also maintaining performance on the un-altered labels for patient’s without GAD. Indeed, the naive-pooling model’s output appears to be biased toward the dilated dataset. On the other hand, the Image-Conditioned model is clearly able to learn the relationship between the presence of Gadolinium-enhancing lesions and the label bias as evidenced by the across the board performance gain relative to the naive-pooling model. In particular, the image-conditioned model outperforms the naive pooling model by 15.6% on the w/o GAD dataset and by 1.8% on the w/ GAD dataset. Notwithstanding, the image-conditioned model does underperform the single trial baseline models, suggesting that the image conditioning module is unable to detect GAD with perfect accuracy, which is not completely unexpected given that GAD lesions, in many cases, are as small as 3 voxels in size.

Each network was optimized separately. Values for learning rate and schedulers, epochs, dropout, and data augmentation were varied for each network. For the single-trial baselines, the hyperparameters that resulted in the best validation performance on the training trial were chosen, and this model was tested against the other trial sets. For the multi-trial models, the hyperparameters that resulted in the best overall performance across all trials were used. The range for the hyperparameter search was as follows:

• •

  Epochs: 150-250

• •

  Learning Rate: 1e-4 to 3e-4, with 1e-5 weight decay

• •

  Scheduler:

  • –

    Epochs: [25, 50, 100, 150, 200] to [200]

  • –

    Gamma: 1/3 to 1/2

In this study, the threshold was selected on a per-trial basis to result in the highest F1. Specifically, for any given model throughout the paper, the results for a test set of a certain trial are presented at the threshold that yields the best F1 for that trial.

emptyempty

emptyempty