QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation – Analysis of Ranking Scores and Benchmarking Results

Raghav Mehta1, Angelos Filos2, Ujjwal Baid3,4,5, Chiharu Sako3,4, Richard McKinley6, Michael Rebsamen6, Katrin Dätwyler6,7, Raphael Meier8, Piotr Radojewski6, Gowtham Krishnan Murugesan9, Sahil Nalawade9, Chandan Ganesh9, Ben Wagner9, Fang F. Yu9, Baowei Fei10, Ananth J. Madhuranthakam9,11, Joseph A. Maldjian9,11, Laura Daza12, Catalina Gómez12, Pablo Arbeláez12, Chengliang Dai13, Shuo Wang13, Hadrien Reynaud13, Yuanhan Mo13, Elsa Angelini14, Yike Guo13, Wenjia Bai13,15, Subhashis Banerjee16,17,18, Linmin Pei19, Murat AK19, Sarahi Rosas-González20, Ilyess Zemmoura20,21, Clovis Tauber20, Minh H. Vu22, Tufve Nyholm22, Tommy Löfstedt23, Laura Mora Ballestar24, Veronica Vilaplana24, Hugh McHugh25,26, Gonzalo Maso Talou27, Alan Wang25,27, Jay Patel28,29, Ken Chang28,29, Katharina Hoebel28,29, Mishka Gidwani28, Nishanth Arun28, Sharut Gupta28, Mehak Aggarwal28, Praveer Singh28, Elizabeth R. Gerstner28, Jayashree Kalpathy-Cramer28, Nicolas Boutry30, Alexis Huard30, Lasitha Vidyaratne31, Md Monibor Rahman31, Khan M. Iftekharuddin31, Joseph Chazalon32, Elodie Puybareau32, Guillaume Tochon32, Jun Ma33, Mariano Cabezas34, Xavier Llado34, Arnau Oliver34, Liliana Valencia34, Sergi Valverde34, Mehdi Amian35, Mohammadreza Soltaninejad36, Andriy Myronenko37, Ali Hatamizadeh37, Xue Feng38, Quan Dou38, Nicholas Tustison39, Craig Meyer38,39, Nisarg A. Shah40, Sanjay Talbar41, Marc-André Weber42, Abhishek Mahajan43, Andras Jakab44, Roland Wiest6,45, Hassan M. Fathallah-Shaykh46, Arash Nazeri47, Mikhail Milchenko47,48, Daniel Marcus47,48, Aikaterini Kotrotsou49, Rivka Colen49, John Freymann50,51, Justin Kirby50,51, Christos Davatzikos3,4, Bjoern Menze52,53, Spyridon Bakas3,4,5, Yarin Gal2, Tal Arbel1,54
1: Centre for Intelligent Machines (CIM), McGill University, Montreal, QC, Canada, 2: Oxford Applied and Theoretical Machine Learning (OATML) Group, University of Oxford, Oxford, England, 3: Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA, 4: Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA, 5: Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA, 6: Support Center for Advanced Neuroimaging (SCAN), University Institute of Diagnostic and Interventional Neuroradiology, University of Bern, Inselspital, Bern University Hospital, Bern, Switzerland, 7: Human Performance Lab, Schulthess Clinic, Zurich, Switzerland, 8: armasuisse S+T, Thun, Switzerland, 9: Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX, USA, 10: Department of Bioengineering, University of Texas at Dallas, Texas, USA, 11: Advanced Imaging Research Center, University of Texas Southwestern Medical Center, Dallas, TX, USA, 12: Universidad de los Andes, Bogotá, Colombia, 13: Data Science Institute, Imperial College London, London, UK, 14: NIHR Imperial BRC, ITMAT Data Science Group, Imperial College London, London, UK, 15: Department of Brain Sciences, Imperial College London, London, UK, 16: Machine Intelligence Unit, Indian Statistical Institute, Kolkata, India, 17: Department of CSE, University of Calcutta, Kolkata, India, 18: Division of Visual Information and Interaction (Vi2), Department of Information Technology, Uppsala University, Uppsala, Sweden, 19: Department of Diagnostic Radiology, The University of Pittsburgh Medical Center, Pittsburgh, PA, USA, 20: UMR U1253 iBrain, Université de Tours, Inserm, Tours, France, 21: Neurosurgery department, CHRU de Tours, Tours, France, 22: Department of Radiation Sciences, Umeå University, Umeå, Sweden, 23: Department of Computing Science, Umeå University, Umeå, Sweden, 24: Signal Theory and Communications Department, Universitat Politècnica de Catalunya, BarcelonaTech, Barcelona, Spain, 25: Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand, 26: Radiology Department, Auckland City Hospital, Auckland, New Zealand, 27: Auckland Bioengineering Institute, University of Auckland, New Zealand, 28: Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, MA, USA, 29: Massachusetts Institute of Technology, Cambridge, MA, USA, 30: EPITA Research and Development Laboratory (LRDE), France, 31: Vision Lab, Electrical and Computer Engineering, Old Dominion University, Norfolk, VA 23529, USA, 32: EPITA Research and Development Laboratory (LRDE), Le Kremlin-Bicêtre, France, 33: School of Science, Nanjing University of Science and Technology, 34: Research Institute of Computer Vision and Robotics, University of Girona, Spain, 35: Department of Electrical and Computer Engineering, University of Tehran, Iran, 36: School of Computer Science, University of Nottingham, UK, 37: NVIDIA, Santa Clara, CA, US, 38: Biomedical Engineering, University of Virginia, Charlottesville, USA, 39: Radiology and Medical Imaging, University of Virginia, Charlottesville, USA, 40: Department of Electrical Engineering, Indian Institute of Technology - Jodhpur, Jodhpur, India, 41: SGGS Institute of Engineering and Technology, Nanded, India, 42: Institute of Diagnostic and Interventional Radiology, Pediatric Radiology and Neuroradiology, University Medical Center Rostock, Rostock, German, 43: Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, India, 44: Center for MR-Research, University Children’s Hospital Zurich, Zurich, Switzerland, 45: Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland, 46: Department of Neurology, The University of Alabama at Birmingham, Birmingham, AL, USA, 47: Department of Radiology, Washington University, St. Louis, MO, USA, 48: Neuroimaging Informatics and Analysis Center, Washington University, St. Louis, MO, USA, 49: Department of Diagnostic Radiology, University of Texas MD Anderson Cancer Center, Houston, TX, USA, 50: Leidos Biomedical Research, Inc, Frederick National Laboratory for Cancer Research, Frederick, MD, USA, 51: Cancer Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA, 52: Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland, 53: Department of Informatics, Technical University of Munich, Munich, Germany, 54: MILA - Quebec Artificial Intelligence Institute, Montreal, QC, Canada
Publication date: 2022/08/26
https://doi.org/10.59275/j.melba.2022-354b
PDF · Code · Video · arXiv

Abstract

Deep learning (DL) models have provided the state-of-the-art performance in a wide variety of medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder the translation of DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties, could enable clinical review of the most uncertain regions, thereby building trust and paving the way towards clinical translation. Recently, a number of uncertainty estimation methods have been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019-2020 task on uncertainty quantification (QU-BraTS), and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions, and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentages of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, and hence highlight the need for uncertainty quantification in medical image analyses. Our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraTS

Keywords

Uncertainty Quantification · Trustworthiness · Segmentation · Brain Tumors · Deep Learning · Neuro-Oncology · Glioma · Glioblastoma

Bibtex @article{melba:2022:026:mehta, title = "QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation – Analysis of Ranking Scores and Benchmarking Results", author = "Mehta, Raghav and Filos, Angelos and Baid, Ujjwal and Sako, Chiharu and McKinley, Richard and Rebsamen, Michael and Dätwyler, Katrin and Meier, Raphael and Radojewski, Piotr and Murugesan, Gowtham Krishnan and Nalawade, Sahil and Ganesh, Chandan and Wagner, Ben and Yu, Fang F. and Fei, Baowei and Madhuranthakam, Ananth J. and Maldjian, Joseph A. and Daza, Laura and Gómez, Catalina and Arbeláez, Pablo and Dai, Chengliang and Wang, Shuo and Reynaud, Hadrien and Mo, Yuanhan and Angelini, Elsa and Guo, Yike and Bai, Wenjia and Banerjee, Subhashis and Pei, Linmin and AK, Murat and Rosas-González, Sarahi and Zemmoura, Ilyess and Tauber, Clovis and Vu, Minh H. and Nyholm, Tufve and Löfstedt, Tommy and Ballestar, Laura Mora and Vilaplana, Veronica and McHugh, Hugh and Maso Talou, Gonzalo and Wang, Alan and Patel, Jay and Chang, Ken and Hoebel, Katharina and Gidwani, Mishka and Arun, Nishanth and Gupta, Sharut and Aggarwal, Mehak and Singh, Praveer and Gerstner, Elizabeth R. and Kalpathy-Cramer, Jayashree and Boutry, Nicolas and Huard, Alexis and Vidyaratne, Lasitha and Rahman, Md Monibor and Iftekharuddin, Khan M. and Chazalon, Joseph and Puybareau, Elodie and Tochon, Guillaume and Ma, Jun and Cabezas, Mariano and Llado, Xavier and Oliver, Arnau and Valencia, Liliana and Valverde, Sergi and Amian, Mehdi and Soltaninejad, Mohammadreza and Myronenko, Andriy and Hatamizadeh, Ali and Feng, Xue and Dou, Quan and Tustison, Nicholas and Meyer, Craig and Shah, Nisarg A. and Talbar, Sanjay and Weber, Marc-André and Mahajan, Abhishek and Jakab, Andras and Wiest, Roland and Fathallah-Shaykh, Hassan M. and Nazeri, Arash and Milchenko, Mikhail and Marcus, Daniel and Kotrotsou, Aikaterini and Colen, Rivka and Freymann, John and Kirby, Justin and Davatzikos, Christos and Menze, Bjoern and Bakas, Spyridon and Gal, Yarin and Arbel, Tal", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "August 2022 issue", year = "2022", pages = "1--54", issn = "2766-905X", doi = "https://doi.org/10.59275/j.melba.2022-354b", url = "https://melba-journal.org/2022:026" }
RISTY - JOUR AU - Mehta, Raghav AU - Filos, Angelos AU - Baid, Ujjwal AU - Sako, Chiharu AU - McKinley, Richard AU - Rebsamen, Michael AU - Dätwyler, Katrin AU - Meier, Raphael AU - Radojewski, Piotr AU - Murugesan, Gowtham Krishnan AU - Nalawade, Sahil AU - Ganesh, Chandan AU - Wagner, Ben AU - Yu, Fang F. AU - Fei, Baowei AU - Madhuranthakam, Ananth J. AU - Maldjian, Joseph A. AU - Daza, Laura AU - Gómez, Catalina AU - Arbeláez, Pablo AU - Dai, Chengliang AU - Wang, Shuo AU - Reynaud, Hadrien AU - Mo, Yuanhan AU - Angelini, Elsa AU - Guo, Yike AU - Bai, Wenjia AU - Banerjee, Subhashis AU - Pei, Linmin AU - AK, Murat AU - Rosas-González, Sarahi AU - Zemmoura, Ilyess AU - Tauber, Clovis AU - Vu, Minh H. AU - Nyholm, Tufve AU - Löfstedt, Tommy AU - Ballestar, Laura Mora AU - Vilaplana, Veronica AU - McHugh, Hugh AU - Maso Talou, Gonzalo AU - Wang, Alan AU - Patel, Jay AU - Chang, Ken AU - Hoebel, Katharina AU - Gidwani, Mishka AU - Arun, Nishanth AU - Gupta, Sharut AU - Aggarwal, Mehak AU - Singh, Praveer AU - Gerstner, Elizabeth R. AU - Kalpathy-Cramer, Jayashree AU - Boutry, Nicolas AU - Huard, Alexis AU - Vidyaratne, Lasitha AU - Rahman, Md Monibor AU - Iftekharuddin, Khan M. AU - Chazalon, Joseph AU - Puybareau, Elodie AU - Tochon, Guillaume AU - Ma, Jun AU - Cabezas, Mariano AU - Llado, Xavier AU - Oliver, Arnau AU - Valencia, Liliana AU - Valverde, Sergi AU - Amian, Mehdi AU - Soltaninejad, Mohammadreza AU - Myronenko, Andriy AU - Hatamizadeh, Ali AU - Feng, Xue AU - Dou, Quan AU - Tustison, Nicholas AU - Meyer, Craig AU - Shah, Nisarg A. AU - Talbar, Sanjay AU - Weber, Marc-André AU - Mahajan, Abhishek AU - Jakab, Andras AU - Wiest, Roland AU - Fathallah-Shaykh, Hassan M. AU - Nazeri, Arash AU - Milchenko, Mikhail AU - Marcus, Daniel AU - Kotrotsou, Aikaterini AU - Colen, Rivka AU - Freymann, John AU - Kirby, Justin AU - Davatzikos, Christos AU - Menze, Bjoern AU - Bakas, Spyridon AU - Gal, Yarin AU - Arbel, Tal PY - 2022 TI - QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation – Analysis of Ranking Scores and Benchmarking Results T2 - Machine Learning for Biomedical Imaging VL - 1 IS - August 2022 issue SP - 1 EP - 54 SN - 2766-905X DO - https://doi.org/10.59275/j.melba.2022-354b UR - https://melba-journal.org/2022:026 ER -
2022:026 cover