The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up

Razvan V. Marinescu1,2Orcid, Neil P. Oxtoby1Orcid, Alexandra L. Young1,3, Esther E. Bron4Orcid, Arthur W. Toga5Orcid, Michael W. Weiner6Orcid, Frederik Barkhof1,7,8Orcid, Nick C. Fox8Orcid, Arman Eshaghi9,1Orcid, Tina Toni10, Marcin Salaterski10, Veronika Lunina10, Manon Ansart11, Stanley Durrleman11, Pascal Lu11, Samuel Iddi12,13, Dan Li12, Wesley K. Thompson14, Michael C. Donohue12, Aviv Nahon15, Yarden Levy15, Dan Halbersberg15, Mariya Cohen15, Huiling Liao16, Tengfei Li16, Kaixian Yu16, Hongtu Zhu16, José G. Tamez-Peña17, Aya Ismail18, Timothy Wood18, Hector Corrada Bravo18, Minh Nguyen19, Nanbo Sun19, Jiashi Feng19, B.T. Thomas Yeo19, Gang Chen20, Ke Qi21, Shiyang Chen21,22, Deqiang Qiu21,22, Ionut Buciuman23, Alex Kelner23, Raluca Pop23, Denisa Rimocea23, Mostafa M. Ghazi24,25,26,1, Mads Nielsen24,25,26, Sebastien Ourselin27,1, Lauge Sørensen24,25,26, Vikram Venkatraghavan4, Keli Liu28, Christina Rabe28, Paul Manser28, Steven M. Hill29, James Howlett29, Zhiyue Huang29, Steven Kiddle29, Sach Mukherjee30, Anaïs Rouanet29, Bernd Taschler30, Brian D. M. Tom29, Simon R. White29, Noel Faux31, Suman Sedai31, Javier de Velasco Oriol17, Edgar E. V. Clemente17, Karol Estrada32,33, Leon Aksman1, Andre Altmann1, Cynthia M. Stonnington34, Yalin Wang35, Jianfeng Wu35, Vivek Devadas36, Clementine Fourrier11, Lars Lau Raket37,38, Aristeidis Sotiras39, Guray Erus39, Jimit Doshi39, Christos Davatzikos39, Jacob Vogel40, Andrew Doyle40, Angela Tam40, Alex Diaz-Papkovich40, Emmanuel Jammeh41, Igor Koval11, Paul Moore42, Terry J. Lyons42, John Gallacher43, Jussi Tohka44, Robert Ciszek44, Bruno Jedynak45, Kruti Pandya45, Murat Bilgel46, William Engels45, Joseph Cole45, Polina Golland2, Stefan Klein4Orcid, Daniel C. Alexander1Orcid, The EuroPOND Consortium 10, The Alzheimer's Disease Neuroimaging Initiative 10
1: Centre for Medical Image Computing, University College London, UK, 2: Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, USA, 3: Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK, 4: Biomedical Imaging Group Rotterdam, Department of Radiology and Nuclear Medicine, Erasmus MC, Netherlands, 5: Laboratory of NeuroImaging, University of Southern California, USA, 6: Center for Imaging of Neurodegenerative Diseases, University of California San Francisco, USA, 7: Department of Radiology and Nuclear Medicine, VU Medical Centre, Netherlands, 8: Dementia Research Centre and the UK Dementia Research Institute, UCL Queen Square Institute of Neurology, UK, 9: Queen Square Multiple Sclerosis Centre, UCL Queen Square Institute of Neurology, UK, 10: Author not affiliated with any research institution, 11: Institut du Cerveau et de la Moelle épinière, Paris, France, 12: Alzheimer's Therapeutic Research Institute, University of Southern California, USA, 13: Department of Statistics and Actuarial Science, University of Ghana, Ghana, 14: Department of Family Medicine and Public Health, University of California San Diego, USA, 15: Ben Gurion University of the Negev, Beersheba, Israel, 16: The University of Texas Health Science Center at Houston, Houston, USA, 17: Instituto Tecnológico y de Estudios Superiores de Monterrey, Monterrey, Mexico, 18: University of Maryland, College Park, USA, 19: National University of Singapore, Singapore, Singapore, 20: Medical College of Wisconsin, Milwaukee, USA, 21: Emory University, Atlanta, USA, 22: Georgia Institute of Technology, Atlanta, USA, 23: Vasile Lucaciu National College, Baia Mare, Romania, 24: Biomediq A/S, Denmark, 25: Cerebriu A/S, Denmark, 26: University of Copenhagen, Denmark, 27: School of Biomedical Engineering and Imaging Sciences, King's College London, UK, 28: Genentech, USA, 29: MRC Biostatistics Unit, University of Cambridge, UK, 30: German Center for Neurodegenerative Diseases, Bonn, Germany, 31: IBM Research Australia, Melbourne, Australia, 32: Brandeis University, Waltham, USA, 33: Department of Statistical Genetics, Biomarin, San Rafael, USA, 34: Mayo Clinic, Scottsdale, AZ, USA, 35: School of Computing, Informatics and Decision Systems Engineering, Arizona State University, Tempe, USA, 36: Banner Alzheimer's Institute, Phoenix, USA, 37: H. Lundbeck A/S, Denmark, 38: Clinical Memory Research Unit, Department of Clinical Sciences Malmo, Lund University, Lund, Sweden, 39: Center for Biomedical Image Computing and Analytics, University of Pennsylvania, 40: McGill University, Montreal, Canada, 41: University of Plymouth, UK, 42: Mathematical Institute, University of Oxford, UK, 43: Department of Psychiatry, University of Oxford, UK, 44: A.I. Virtanen Institute for Molecular Sciences, University of Eastern Finland, Finland, 45: Portland State University, Portland, USA, 46: Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD, USA
Publication date: 2021/12/31
https://doi.org/10.59275/j.melba.2021-2dcc
PDF · Source code · TADPOLE-SHARE · Video · Website · arXiv

Abstract

Accurate prediction of progression in subjects at risk of Alzheimer's disease is crucial for enrolling the right subjects in clinical trials. However, a prospective comparison of state-of-the-art algorithms for predicting disease onset and progression is currently lacking. We present the findings of "The Alzheimer's Disease Prediction Of Longitudinal Evolution" (TADPOLE) Challenge, which compared the performance of 92 algorithms from 33 international teams at predicting the future trajectory of 219 individuals at risk of Alzheimer's disease. Challenge participants were required to make a prediction, for each month of a 5-year future time period, of three key outcomes: clinical diagnosis, Alzheimer's Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13), and total volume of the ventricles. The methods used by challenge participants included multivariate linear regression, machine learning methods such as support vector machines and deep neural networks, as well as disease progression models. No single submission was best at predicting all three outcomes. For clinical diagnosis and ventricle volume prediction, the best algorithms strongly outperform simple baselines in predictive ability. However, for ADAS-Cog13 no single submitted prediction method was significantly better than random guesswork. Two ensemble methods based on taking the mean and median over all predictions, obtained top scores on almost all tasks. Better than average performance at diagnosis prediction was generally associated with the additional inclusion of features from cerebrospinal fluid (CSF) samples and diffusion tensor imaging (DTI). On the other hand, better performance at ventricle volume prediction was associated with inclusion of summary statistics, such as the slope or maxima/minima of patient-specific biomarkers. On a limited, cross-sectional subset of the data emulating clinical trials, performance of the best algorithms at predicting clinical diagnosis decreased only slightly (2 percentage points) compared to the full longitudinal dataset. The submission system remains open via the website https://tadpole.grand-challenge.org, while TADPOLE SHARE (https://tadpole-share.github.io/) collates code for submissions. TADPOLE's unique results suggest that current prediction algorithms provide sufficient accuracy to exploit biomarkers related to clinical diagnosis and ventricle volume, for cohort refinement in clinical trials for Alzheimer's disease. However, results call into question the usage of cognitive test scores for patient selection and as a primary endpoint in clinical trials.

Keywords

statistical modelling · machine learning · benchmark · alzheimer's disease prediction

Bibtex @article{melba:2021:019:marinescu, title = "The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up", author = "Marinescu, Razvan V. and Oxtoby, Neil P. and Young, Alexandra L. and Bron, Esther E. and Toga, Arthur W. and Weiner, Michael W. and Barkhof, Frederik and Fox, Nick C. and Eshaghi, Arman and Toni, Tina and Salaterski, Marcin and Lunina, Veronika and Ansart, Manon and Durrleman, Stanley and Lu, Pascal and Iddi, Samuel and Li, Dan and Thompson, Wesley K. and Donohue, Michael C. and Nahon, Aviv and Levy, Yarden and Halbersberg, Dan and Cohen, Mariya and Liao, Huiling and Li, Tengfei and Yu, Kaixian and Zhu, Hongtu and Tamez-Peña, José G. and Ismail, Aya and Wood, Timothy and Bravo, Hector Corrada and Nguyen, Minh and Sun, Nanbo and Feng, Jiashi and Yeo, B.T. Thomas and Chen, Gang and Qi, Ke and Chen, Shiyang and Qiu, Deqiang and Buciuman, Ionut and Kelner, Alex and Pop, Raluca and Rimocea, Denisa and Ghazi, Mostafa M. and Nielsen, Mads and Ourselin, Sebastien and Sørensen, Lauge and Venkatraghavan, Vikram and Liu, Keli and Rabe, Christina and Manser, Paul and Hill, Steven M. and Howlett, James and Huang, Zhiyue and Kiddle, Steven and Mukherjee, Sach and Rouanet, Anaïs and Taschler, Bernd and Tom, Brian D. M. and White, Simon R. and Faux, Noel and Sedai, Suman and de Velasco Oriol, Javier and Clemente, Edgar E. V. and Estrada, Karol and Aksman, Leon and Altmann, Andre and Stonnington, Cynthia M. and Wang, Yalin and Wu, Jianfeng and Devadas, Vivek and Fourrier, Clementine and Raket, Lars Lau and Sotiras, Aristeidis and Erus, Guray and Doshi, Jimit and Davatzikos, Christos and Vogel, Jacob and Doyle, Andrew and Tam, Angela and Diaz-Papkovich, Alex and Jammeh, Emmanuel and Koval, Igor and Moore, Paul and Lyons, Terry J. and Gallacher, John and Tohka, Jussi and Ciszek, Robert and Jedynak, Bruno and Pandya, Kruti and Bilgel, Murat and Engels, William and Cole, Joseph and Golland, Polina and Klein, Stefan and Alexander, Daniel C. and , The EuroPOND Consortium and , The Alzheimer's Disease Neuroimaging Initiative", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "December 2021 issue", year = "2021", pages = "1--60", issn = "2766-905X", doi = "https://doi.org/10.59275/j.melba.2021-2dcc", url = "https://melba-journal.org/2021:019" }
RISTY - JOUR AU - Marinescu, Razvan V. AU - Oxtoby, Neil P. AU - Young, Alexandra L. AU - Bron, Esther E. AU - Toga, Arthur W. AU - Weiner, Michael W. AU - Barkhof, Frederik AU - Fox, Nick C. AU - Eshaghi, Arman AU - Toni, Tina AU - Salaterski, Marcin AU - Lunina, Veronika AU - Ansart, Manon AU - Durrleman, Stanley AU - Lu, Pascal AU - Iddi, Samuel AU - Li, Dan AU - Thompson, Wesley K. AU - Donohue, Michael C. AU - Nahon, Aviv AU - Levy, Yarden AU - Halbersberg, Dan AU - Cohen, Mariya AU - Liao, Huiling AU - Li, Tengfei AU - Yu, Kaixian AU - Zhu, Hongtu AU - Tamez-Peña, José G. AU - Ismail, Aya AU - Wood, Timothy AU - Bravo, Hector Corrada AU - Nguyen, Minh AU - Sun, Nanbo AU - Feng, Jiashi AU - Yeo, B.T. Thomas AU - Chen, Gang AU - Qi, Ke AU - Chen, Shiyang AU - Qiu, Deqiang AU - Buciuman, Ionut AU - Kelner, Alex AU - Pop, Raluca AU - Rimocea, Denisa AU - Ghazi, Mostafa M. AU - Nielsen, Mads AU - Ourselin, Sebastien AU - Sørensen, Lauge AU - Venkatraghavan, Vikram AU - Liu, Keli AU - Rabe, Christina AU - Manser, Paul AU - Hill, Steven M. AU - Howlett, James AU - Huang, Zhiyue AU - Kiddle, Steven AU - Mukherjee, Sach AU - Rouanet, Anaïs AU - Taschler, Bernd AU - Tom, Brian D. M. AU - White, Simon R. AU - Faux, Noel AU - Sedai, Suman AU - de Velasco Oriol, Javier AU - Clemente, Edgar E. V. AU - Estrada, Karol AU - Aksman, Leon AU - Altmann, Andre AU - Stonnington, Cynthia M. AU - Wang, Yalin AU - Wu, Jianfeng AU - Devadas, Vivek AU - Fourrier, Clementine AU - Raket, Lars Lau AU - Sotiras, Aristeidis AU - Erus, Guray AU - Doshi, Jimit AU - Davatzikos, Christos AU - Vogel, Jacob AU - Doyle, Andrew AU - Tam, Angela AU - Diaz-Papkovich, Alex AU - Jammeh, Emmanuel AU - Koval, Igor AU - Moore, Paul AU - Lyons, Terry J. AU - Gallacher, John AU - Tohka, Jussi AU - Ciszek, Robert AU - Jedynak, Bruno AU - Pandya, Kruti AU - Bilgel, Murat AU - Engels, William AU - Cole, Joseph AU - Golland, Polina AU - Klein, Stefan AU - Alexander, Daniel C. AU - , The EuroPOND Consortium AU - , The Alzheimer's Disease Neuroimaging Initiative PY - 2021 TI - The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up T2 - Machine Learning for Biomedical Imaging VL - 1 IS - December 2021 issue SP - 1 EP - 60 SN - 2766-905X DO - https://doi.org/10.59275/j.melba.2021-2dcc UR - https://melba-journal.org/2021:019 ER -
2021:019 cover