BraTS Meningioma Challenge image data was contributed from 6 different United States academic medical centers: Duke University, Yale University, Thomas Jefferson University, University of California San Francisco, Missouri University, and University of Pennsylvania. Image data consisted of T1, T2, FLAIR, and T1Gd brain MRI sequences from patients with radiographic or pathologic diagnosis of intracranial meningioma. All data preprocessing was conducted using the FeTS tool, (Pati et al., 2022) and included conversion to NIfTI file format, co-registration, $1\,\text{mm}^{3}$ isotropic resampling to the SRI24 atlas space, and automated skull stripping (Schwarz et al., 2019; Thakur et al., 2019, 2020; Juluru et al., 2020; Smith, 2002). The skull stripping algorithm, part of the FeTS preprocessing workflow, was integral in removing non-brain tissue, including the skull and scalp, to isolate the intracranial structures. The brain extraction tool, widely used in neuroimaging pipelines, relies on deformable models and intensity-based thresholding to separate the brain from surrounding tissues (Smith, 2002). In this challenge, skull stripping was critical for preserving patient anonymity by preventing potential face reconstruction from MRI data and to standardize data preparation across institutions. However, it should be noted that meningiomas often extend through the skull and skull-base foramina, and any extra-cranial portions of the tumors were implicitly excluded by this process (Smith, 2002). Despite this limitation, skull stripping was applied to ensure consistency with other BraTS 2023 challenges and to minimize the inclusion of non-brain tissue. After image data pre-processing, tumor compartment labels were created using a comprehensive pre-segmentation, manual correction, and expert revision process, including enhancing tumor, non-enhancing tumor, and surrounding non-enhancing FLAIR hyperintensity (SNFH) as seen in Figure 2, which is an unmodified figure by LaBella et al. (LaBella et al., 2023, 2024). The initial pre-segmentation was performed using a deep convolutional neural network-based model (Isensee et al., 2021). Subsequently, 39 annotators manually reviewed and refined the segmentations. These annotators had varying levels of experience, from medical stu1dents to fellowship-trained neuroradiologists. Each annotator was given instructions on how to use the ITKSnap annotation software (Yushkevich et al., 2006) as well as a document on common errors of meningioma pre-segmentation as discussed in the dataset resource paper by LaBella et al (LaBella et al., 2024). After each annotator completed their manual corrections, the labels were sent to a final board-certified fellowship trained neuroradiologist (EC) for approval. This multi-step approach ensured the accuracy and consistency of the segmentation labels, incorporating multiple rounds of revision as needed to achieve high-quality final segmentations. All participating institutions received Institutional Review Board and Data Transfer Agreement approvals before contributing data, ensuring compliance with relevant regulatory authorities.

The BraTS 2023 Intracranial Meningioma Segmentation Challenge was hosted on the Synapse platform using the BraTS Pre-operative Meningioma Dataset (bra, 2023). To access the challenge dataset and to be eligible for submission of automated segmentation models, participants were required to register as a participating team on the Synapse platform. Registered teams developed automated segmentation algorithms that trained on multi-sequence MRI of pre-treatment intracranial meningioma with associated ground truth labels that were released to the participating teams in May 2023.

In June 2023, each of the participating teams had access to additional validation data consisting of multi-sequence MRI cases. For validation data, teams were able to assess segmentation performance of their models by submitting predicted labels through the Synapse platform, but individual ground truth segmentations were not made publicly available. From July until August 2023, participating teams utilized the validation dataset to fine tune their segmentation models and compose short paper manuscripts. At the end of the validation phase, each participating team uploaded their optimal automated segmentation model and respective manuscript as an MLCube container, which was used for evaluation in the testing phase. During the testing phase, the BraTS organizing committee internally evaluated each of the participating team’s automated segmentation models on the hidden test set of pre-operative meningioma cases with ground truth labels.

During the testing phase, the BraTS organizing committee evaluated metrics on three regions of interest including ET, TC, and WT. ET was solely the enhancing tumor compartment label. TC was the combination of enhancing tumor and non-enhancing tumor compartment labels. WT was the combination of enhancing tumor, non-enhancing tumor, and SNFH compartment labels. Note that the term “whole tumor” was used across the BraTS 2023 cluster of challenges for consistency; however, this term is not entirely accurate for meningioma, where SNFH typically does not contain any tumor but rather represents associated vasogenic edema. Metrics used for evaluation included DSC and the 95HD and were evaluated on a lesion-wise level. The DSC is a measure used to quantify the similarity between two samples, which, in this context, refers to the overlap between automated segmentation and the expert annotated ground truth labels for each respective tumor compartment. The 95HD observed in the segmentation results. The 95HD was used in lieu of the standard 100% Hausdorff Distance to account for smaller lesions that may suffer from overestimates of the standard 100% Hausdorff Distance. For previous BraTS challenges, a global DSC was used for challenge rankings. However, lesion-wise metrics were adopted for the 2023 challenge as there was greater potential for multiple distinct lesions in a single patient image (most notably for the metastasis and meningioma sub-challenges). Distinct lesions were identified by performing a 1 voxel symmetric dilation on the ground truth WT masks, and then evaluating a 26-connectivity 3D connected component analysis to determine if overlap between distinct lesions exists (Rudie, 2023). A case’s lesion-wise DSC and 95HD scores are calculated based on equations (1) and (2) respectively, where L is the number of ground truth lesions and (true positive (TP) + false negative (FN)) is equal to L (Saluja, 2023). A predicted lesion is counted as a TP if at least 1 predicted voxel overlaps with the respective ground truth’s respective region of interest mask. A lesion is counted as a FN if the model does not predict any voxels within the ground truth’s respective region of interest mask. A predicted lesion is counted as a false positive (FP) if the model predicts a distinct lesion that does not overlap with any ground truth lesions’ voxels. The lesion-wise scoring system assigned a specific lesion’s region of interest a DSC score of 0 and a 95HD score of 374 for FP or FN. These equations effectively calculate the average DSC or 95HD values across all of the predicted lesions for a given case. The scoring system also excluded evaluation for ground truth lesions smaller than 50 voxels to avoid evaluation of false ground truth lesions missed in dataset review. This threshold was discussed and decided by fellowship trained neuroradiologists after ground truth label review (Saluja, 2023; Rudie, 2023). Evaluation of submissions was performed on MLCommons’ MedPerf, an open federated AI/ML evaluation platform (Karargyris et al., 2023). MedPerf automated the evaluating pipeline by running the participants’ models on the testing datasets of each contributing site’s data and calculating evaluation metrics on the resulting predictions. Finally, the Synapse platform retrieved the metrics results from the MedPerf server and ranked them to determine the winner (MLCommons Association, 2024; Pati et al., 2023; Karargyris et al., 2023).

The BraTS organizing committee internally evaluated each of the participating team’s automated segmentation models on the hidden test set of pre-operative meningioma cases to determine lesion-wise metrics for both DSC and 95HD for each of the three regions of interest. The participants were ranked against each other for each region of interest’s lesion-wise metric independently. A total of 6 independent rankings were calculated to reflect the two metrics, DSC and 95HD, for each of the ET, TC, and WT regions of interest. Then a “BraTS segmentation score” was calculated based on the average of each independent lesion-wise region of interest metric rankings. For example, if a team had the 3rd best ET DSC, 2nd best TC DSC, 3rd best WT DSC, 3rd best ET 95HD, 2nd best TC 95HD, and 4th best WT 95HD, then that team would have an average ranking of (3+2+3+3+2+4) / 6 = 2.83 as their BraTS segmentation score. The BraTS segmentation score was used to determine the final participant rankings relative to one another. The three top-ranked teams were invited to present their findings at the BraTS workshop at the 2023 MICCAI Annual Meeting held in Vancouver, Canada; although final rank was hidden until the workshop. At the BraTS workshop, the BraTS organizing committee announced the final placement of the three top-ranked teams. Monetary awards of $1400, $1000, and $800 were presented to the three top-ranked teams, respectively.

Overall participant and individual team statistical analysis of DSC and 95HD lesion-wise performance was performed using Python and Microsoft Excel (Excel). Analysis included calculation of participant average, standard deviation, and median DSC and 95HD for each region of interest; overall average and median DSC and 95HD across all participants for each region of interest; volume calculations of each lesion; and number of abutting voxels of lesions compared to the pre-processed brain MRI.

Given the extra-axial location of meningiomas, we sought to evaluate the proportion of meningiomas that were potentially cropped or excluded by the automated skull stripping process. To determine the volume of each compartment label and the number of tumor compartment voxels that were directly abutting the edge of the skull-stripped images for each case, the NumberOfEdgeNeighbors.py script was internally run by BraTS organizers, which evaluated each of the 1483 meningioma MRI cases (LaBella, 2024). This analysis was performed internally due to restricted access to the hidden test dataset. To determine significance of association of WT volume compared to abutting voxels, the Pearson Correlation Coefficient was calculated with associated p-value with a significance level of 0.05.

The top median DSC across all participants for a specific case was a perfect score of 1.00 for enhancing tumor and tumor core as shown in Figure 5. Note that there was a total of 33 enhancing tumor voxels, and 23 were abutting the edge of the MR image. The cases with the next highest overall median and average DSC are shown in Figure 6. Note that the ET volumes are qualitatively much greater in these particularly high DSC cases.

There were two meningioma cases with somewhat unusual imaging appearance as shown in Figures 7 and 8 which had poor performance for test performance metrics across all participants. Notably, they had a majority of non-enhancing tumor making up the TC and WT. These lesions correspond to heavily calcified lesions with little or no visible enhancement and low signal intensity on all provided sequences.

Each region of interest’s DSC was compared to WT volume for each respective case as shown in Figure 8. There was a nonsignificant positive linear correlation between DSC vs WT volume for each of the three regions of interest, ET, TC, and WT; with p values of 0.696, 0.689, and 0.741, respectively. There was a nonsignificant logarithmic correlation between DSC vs WT volume for each of the three regions of interest, ET, TC, and WT; with p values of 0.102, 0.093, and 0.200, respectively (not shown in figure). Notably, Figure 9 also demonstrates a significant number of cases with a lesion-wise DSC of approximately 0.5 for each of the regions of interest. This is due to the lesion-wise metrics penalizing false positives with a value of 0 for the respective prediction and false negatives with a value of 0 for missed ground truth lesions; combined with a very strong performance for another ground truth lesion.

Skull-stripping resulted in 908 of 1000 training cases, 129 of 141 validation cases, and 257 of 283 test cases’ meningioma labels (1286 of 1424, 90.3% overall) having at least 1 compartment voxel abutting the edge of the skull-stripped image edge as seen in the example case in Figure 10. Of the 257 aforementioned cases, the average and median number of abutting voxels was 628.7 and 394 voxels respectively. Figure 11 shows the relationship between the number of abutting voxels and the WT volume ($\text{R}^{2}$ = 0.190 and p = 0.002).

While global metrics (i.e. those used in previous BraTS challenges) were not used for ranking of the 2024 challenge, we have included Figure 12 to show the relationship between the complete tumor volume and each of the global metrics in cases with only a single lesion for each of the regions of interest. Note that there is a noticeable trend towards improved global DSC, global 95HD, and global sensitivity as complete volume increases.

The BraTS 2023 Intracranial Meningioma Segmentation Challenge provided unprecedented insight into the state-of-the-art performance in pre-operative meningioma segmentation, leveraging the largest multi-institutional systematically expert annotated multilabel meningioma MRI dataset to date (Calabrese and LaBella, 2023; LaBella et al., 2024). The challenge saw remarkable performances, particularly from the NVAUTO team, in DSC and 95HD across ET, TC, and WT region of interest segmentation tasks. It was notable that DSC scores were, on average, higher for the meningioma challenge compared to all other BraTS 2023 segmentation sub-challenges, which may be due to the relatively lower complexity of meningiomas compared to other tumor types and/or the relatively high quality and consistency of the meningioma dataset. These lesion-wise DSC and 95HD scores should be considered as benchmarks for future pre-operative meningioma segmentation evaluation. Note that lesion-wise metrics are essential for segmentation tasks with potential for more than one distinct lesion as global metrics may still remain relatively high even if a single, smaller lesion is completely missed by the segmentation algorithm.

The superior performance of NVAUTO, with lesion-wise median DSC values of 0.976, 0.976, and 0.964 lesion-wise average DSC values of 0.899, 0.904, and 0.871, for ET, TC, and WT respectively, signifies a notable advancement in automated meningioma segmentation algorithms. These results not only demonstrate the feasibility of achieving high accuracy in meningioma segmentation, but also suggest that deep learning models can effectively adapt to the diverse morphology and anatomical locations of meningiomas. NVAUTO’s base algorithm, known as AutoSeg3D, an open-source and Pytorch-based framework, is particularly adaptable to a variety of medical imaging automated segmentation challenges (Myronenko et al., 2023). Auto3DSeg allows for auto-scaling to available GPUs; 5-fold training with SegResNet, DiNTS, and SwinUNETR models; and performing inference and ensembling using each of the multiple trained models. For the challenges, NVAUTO found the SegResNet model to be the most accurate and was ultimately the optimal model architecture selected by Auto3DSeg for each of the automated segmentation challenges (Myronenko et al., 2023). Additionally, Auto3DSeg, allows for use of a variety of image modalities and is not limited to just MRI. Auto3DSeg is advertised to run with a GPU RAM $\geq$ 8GB. However, for the 2023 BraTS challenges, NVAUTO used 8 x 16GB NVIDIA V100 machines (Myronenko et al., 2023). Future studies should assess the ability to use Auto3DSeg on more widely available, consumer grade GPUs and non-NVIDIA branded GPUs.

The challenge results revealed a broad range of performances among the 9 participating teams, with DSC scores for the ET ranging from 0.899 to 0.007. Such variability underscores the challenge’s complexity and the diverse approaches to segmentation. When comparing NVAUTO’s meningioma segmentation performance compared to their performance on other challenges, it was found that they performed the best on meningioma’s DSC for ET and TC regions of interest but performed only second best for DSC for WT as shown in Table 6. NVAUTO’s algorithm showed that Sub-Saharan Africa gliomas had a higher DSC for WT than all other tumor types. Since meningioma tend to have a smaller overall whole tumor volume and a higher ET or TC to WT ratio compared to glioblastoma, it can be hypothesized that there is less available training information to create as accurate SNFH compartment labels; and thereafter, the WT region of interest (Ogasawara et al., 2021; Gilard et al., 2021; Khandwala et al., 2021).

Our analysis revealed that all participating teams performed relatively poorly on heavily calcified meningiomas with little or no enhancing tumor. For example, teams had an average TC DSC of only 0.156 for one particular non-enhancing meningioma case. This relatively poor performance is presumably related to the relative rarity of this imaging appearance of meningioma, and the fact that only a small number of such cases were included in the training dataset. Future datasets should include more cases of exclusively non-enhancing tumor to allow for more generalizable automated segmentation models. It is important to consider that radiotherapy plans only consider a single GTV represented by the TC, and whether these indeterminate TC regions are labeled as non-enhancing vs enhancing regions would not impact the resulting treatment volumes (Rogers et al., 2017, 2020).

Note that in Figure 9, the lines of best fit for ET, TC, and WT trend towards improved DSC with increased WT volume. In an automated segmentation challenge, this could cause a higher perceived test set overall DSC score if a larger proportion of larger tumors are included in the test set compared to the overall population. Therefore, it is important to ensure balance within the training, validation, and test sets regarding tumor size, which was not explicitly done for this iteration of the challenge.

Another notable limitation of our study is the absence of explicit testing on out-of-distribution (OOD) cases. While our model demonstrated strong performance on the provided BraTS 2023 meningioma dataset, all data were derived from a limited number of institutions, and no evaluation was performed on data from external sources or significantly different MRI acquisition protocols. Consequently, the generalization ability of the model to cases from different populations, MRI machines, or acquisition settings remains untested. Future work should focus on assessing the model’s performance on OOD data to ensure its robustness and applicability in real-world clinical environments. Techniques such as domain adaptation and cross-institutional validation will be crucial to improve the model’s generalization capabilities and reliability across diverse clinical settings.

Future studies involving meningioma automated segmentation should focus on the most important clinical volumes, particularly the tumor core which comprises the radiotherapy gross tumor volume (GTV). Additionally, future studies should focus on the segmentation of meningioma along the intra-axial and extra-axial border with various face anonymization pre-processing techniques, due to the high frequency for meningioma to be excluded by skull-stripping as demonstrated by our results (Watts et al., 2014; Schwarz et al., 2019, 2021).

This study performed an analysis of the propensity of meningioma to abut the skull-stripped image, thereby having the potential to exclude portions of the meningioma within the skull-stripped image. Due to skull-stripping resulting in 1286 of 1483 meningioma having at least 1 compartment voxel abutting the boundary of the skull-stripped image, future studies should evaluate a different pre-processing anonymization technique that will allow for inclusion of more volume of intracranial meningioma. Schwarz et al. describes a promising mri_reface technique that performs face anonymization by modifying the MR head image face and ear regions to represent an average human face and ear, thereby preserving the vast majority of the MR head image (Schwarz et al., 2019, 2021). Bischoff et al. describes mri_deface, a defacing tool that removes facial features by assigns a probability of voxel being “face” or “brain” and removes voxels that have non-zero probability of being “face” but zero probability of being “brain” (Theyers et al., 2021; Bischoff-Grethe et al., 2007).

Furthermore, while current segmentation approaches predominantly utilize MRI, atypical meningiomas—such as those that are completely non-enhancing or heavily calcified—remain challenging to automate segmentation due to low signal intensity and limited contrast. In these cases, integrating additional imaging modalities could prove highly beneficial. PET imaging, especially with tracers like ${}^{68}\mathrm{Ga}$-DOTATATE, provides metabolic information that can help differentiate viable tumor tissue from calcified or fibrotic areas (Prasad et al., 2022). Likewise, CT offers superior spatial resolution for delineating calcifications and osseous involvement, which is critical when meningiomas extend into or invade bone (Salah et al., 2019). Indeed, studies have demonstrated that PET and CT can outperform MRI in identifying bony invasion and calcification in meningiomas (Galldiks et al., 2023; Salah et al., 2019). Thus, the incorporation of multi-modal imaging may lead to more robust segmentation algorithms capable of addressing the full spectrum of meningioma presentations.

RTOG 0539, a phase II trial of observation for low-risk meningiomas and of radiotherapy for intermediate- and high-risk meningiomas, describes radiation treatment planning and target volume protocols that should be used for meningiomas (Rogers et al., 2017, 2020). For radiation planning, they only required use of pre-operative and post-operative contrast-enhanced MRI. RTOG 0539 defines the GTV to encompass the tumor bed on postoperative-enhanced MRI and to include any residual nodular enhancement. They also state that trailing linear dural tail and cerebral edema should not be specifically included within the GTV, since there is no evidence that recurrence is more likely within the dural tail (Rogers et al., 2017, 2020).

Therefore, future studies that focus on automated segmentation of meningioma for radiation therapy planning should place emphasis on evaluation of the enhancing tumor and post-op resection bed volumes on post-operative T1Gd treatment planning MRI, while reducing emphasis on the surrounding non-enhancing FLAIR hyperintensity compartment and the small trailing linear dural tail enhancement.

However, in the post-treatment follow-up setting, RTOG 0539 still required the use of T1, T2, FLAIR, and T1Gd series, which is similar to the imaging used in the BraTS 2023 Challenge: Intracranial Meningioma. For this challenge’s pre-operative meningioma dataset, the enhancing and non-enhancing tumor labels compose the tumor-core. The SNFH label representing edema, which was included in the WT region of interest is not typically included within radiation therapy meningioma target volumes.