TADPOLE Challenge asked participants to forecast three key biomarkers: (1) clinical diagnosis, which can be either cognitively normal (CN), mild cognitive impairment (MCI), or probable AD; (2) Alzheimer’s Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13) score; and (3) ventricle volume (divided by intra-cranial volume) from MRI. The exact time of future data acquisitions for any given individual was unknown at forecast time, so participants submitted month-by-month predictions for every individual. Predictions of clinical status comprise relative likelihoods of each option (CN, MCI, and AD) for each individual at each month. Predictions of ADAS-Cog13 and ventricle volume comprise a best-guess estimate as well as a 50% confidence interval for each individual at each month. Full details on challenge design are given in the TADPOLE white paper marinescu2018tadpole .

We had a total of 33 participating teams, who submitted a total of 58 predictions from the longitudinal prediction set (D2), 34 predictions from the cross-sectional prediction set (D3), and 6 predictions from custom prediction sets (see Online Methods section 5.1 for description of D2/D3 datasets). A total of 8 D2/D3 submissions from 6 teams did not have predictions for all three target variables, so we computed the performance metrics for only the submitted target variables. Another 3 submissions lacked confidence intervals for either ADAS-Cog13 or ventricle volume, which we imputed using default low-width confidence ranges of 2 for ADAS-Cog13 and 0.002 for Ventricles normalised by intracranial volume (ICV).

Table 1 summarises the methods used in the submissions in terms of feature selection, handling of missing data, predictive models for clinical diagnosis and ADAS/Ventricles biomarkers, as well as training and prediction times. A detailed description of each method is in Online Methods Section 5.4. In particular, some entries constructed augmented features (i.e. summary statistics), which are extra features such as slope, min/max or moments that are derived from existing features.

In addition to the forecasts submitted by participants, we also evaluated four benchmark methods, which were made available to participants during the submission phase of the challenge: (i) BenchmaskLastVisit uses the measurement of each target from the last available clinical visit as the forecast, (ii) BenchmarkMixedEffects uses a mixed effects model with age as predictor variable for ADAS and Ventricle predictions, and Gaussian likelihood model for diagnosis prediction, (iii) BenchmarkMixedEffectsAPOE is as (ii) but adds APOE status as a covariate and (iv) BenchmarkSVM uses an out-of-the-box support vector machine (SVM) classifier and regressor (SVR) to provide forecasts. More details on these methods can be found in Online Methods section 5.4. We also evaluated two ensemble methods based on taking the mean (ConsensusMean) and median (ConsensusMedian) of the forecasted variables over all submissions.

To control for potentially spurious strong performance arising from multiple comparisons, we also evaluated 100 random predictions by adding Gaussian noise to the forecasts of the simplest benchmark model (BenchmarkLastVisit). In the subsequent results tables we will show, for each performance metric, only the best score obtained by any of these 100 random predictions (RandomisedBest) – See end of Online Methods section 5.4 for more information on RandomisedBest.

Table 2 compiles all metrics for all TADPOLE submitted forecasts, as well as benchmarks and ensemble forecasts, from the longitudinal D2 prediction set. For details on datasets D2 and D3, see Online Methods section 5.1, while for details on performance metrics see Online Methods section 5.2. Box-plots showing the distribution of scores, computed on 50 bootstraps of the test set, are shown in Supplementary Fig. 2, while the distribution of ranks is shown in Supplementary Figs. 9 – 11. Among the benchmark methods, BenchmarkMixedEffectsAPOE had the best overall rank of 18, obtaining rank 35 on clinical diagnosis prediction, rank 2 on ADAS-Cog13 and rank 23 on Ventricle volume prediction. Removing the APOE status as covariate proved to significantly increase the predictive performance (BenchmarkMixedEffects), although we do not show ranks for this entry as it was found during the evaluation phase. Among participant methods, the submission with the best overall rank was Frog, obtaining rank 1 for prediction of clinical diagnosis, rank 4 for ADAS-Cog13 and rank 10 for Ventricle volume prediction.

For clinical diagnosis, the best submitted forecasts (team Frog) scored better than all benchmark methods, reducing the error of the best benchmark methods by 58% for the multiclass area under the receiver operating characteristic curve (MAUC) and by 38% for balanced classification accuracy (BCA). Here, the best benchmarks obtained a MAUC of 0.846 (BenchmarkMixedEffects) and a BCA of 0.792 (BenchmarkLastVisit). Among participant methods, Frog had the best MAUC score of 0.931, significantly better than all entries other than Threedays according to the bootstrap test (p-value = 0.24, see Supplementary section B.1 for details on significance testing). Supplementary Figure 9 further shows the variability in performance ranking over bootstrap samples and highlights that the top two entries consistently remain at the top of the ranking. In terms of BCA, ARAMIS-Pascal had the best score of 0.850. Moreover, ensemble methods (ConsensusMedian) achieved the second best MAUC score of 0.925 and the best BCA score of 0.857. In contrast, the best randomised prediction (RandomisedBest) achieved a much lower MAUC of 0.800 and a BCA of 0.803, suggesting entries below these scores did not perform significantly better than random guessing according to the bootstrap test (p-value = 0.01). MAUC and BCA performance metrics had a relatively high correlation across all submissions ($r$ = 0.88, Supplementary Fig. 4).

For Ventricle volume, the best submitted forecasts among participants (team EMC1) also scored considerably better than all benchmark methods, reducing the error of the best benchmark methods by almost one third (29%) for mean absolute error (MAE) and around one half (51%) for weighted error score (WES). Here, the best benchmark method (BenchmarkMixedEffects) had an overall Ventricle MAE and WES of 0.56. Among participant submissions, EMC1-Std/-Custom had the best MAE of 0.41 (% ICV), significantly lower than all entries other than lmaUCL-Covariates/-Std/-half-D1, BORREGOTECMTY and SBIA according to the Wilcoxon signed-rank test (see Supplementary section B.2) – this is also confirmed in Supplementary Fig. 11 by the variability in performance ranking over bootstrap samples. Team EMC1 also had the best Ventricle WES of 0.29, while DIKU-ModifiedMri-Custom/-Std had the best Ventricle coverage probability accuracy (CPA) of 0.01. Ensemble methods (ConsensusMean) achieved the best Ventricle MAE of 0.38. In contrast, the best randomised prediction (RandomisedBest) achieved a higher MAE of 0.46, WES of 0.45 and CPA of 0.33. MAE and WES scores showed high correlation ($r$ = 0.99, Supplementary Fig. 4) and were often of equal value for many submissions ($n=24$), as teams set equal weights for all subjects analysed. CPA did not correlate ($r\approx-0.01$, Supplementary Fig. 4) with either MAE or WES.

For ADAS-Cog13, the best submitted forecasts did not score significantly better than the simple benchmarks. Here, the simple BenchmarkMixedEffects model obtained the second-best MAE of 4.19, which was significantly lower than all other submitted entries according to the Wilcoxon signed-rank test. BenchmarkMixedEffects also had the best ADAS-Cog13 WES of 4.19, while VikingAI-Sigmoid had the best ADAS-Cog13 CPA of 0.02. Among participants’ submissions, FortuneTellerFish-Control ranked first in ADAS-Cog13 prediction with a MAE of 4.70, which is 11% higher than the error of the best benchmark. Moreover, all participants’ forecasts scored worse than the best randomised prediction (RandomisedBest), which here achieved a MAE of 4.52 and WES of 4.52. Nevertheless, the ensemble method ConsensusMean obtained the best ADAS scores for MAE (3.75), WES (3.54) and CPA (0.0), which along with BenchmarkMixedEffects were the only entries that performed significantly better than random guessing (p-value = 0.01). The MAE and WES scores for ADAS-Cog13 had relatively high correlation ($r=0.97$, Supplementary Fig. 4) and were often of equal value for many submissions ($n=25$). CPA had a weak but significant correlation with MAE ($r=0.37$, p-value $<$ 0.02) and WES ($r=0.35$, p-value $<$ 0.02).

Table 3 shows the ranking of the forecasts from the cross-sectional D3 prediction set. Box-plots showing the distribution of scores, computed on 50 bootstraps of the test set, are shown in Supplementary Fig 3, while the distribution of ranks is shown in Supplementary Figs. 12 – 14. Due to the lack of longitudinal data, most submissions had lower performance compared to their equivalents from the D2 longitudinal prediction set. Among submitted forecasts, GlassFrog-Average had the best overall rank, as well as rank 2-4 on diagnosis prediction, rank 5 on ADAS-Cog13 prediction and rank 3 on ventricle prediction.

For clinical diagnosis prediction on D3, the best prediction among TADPOLE participants (team IBM-OZ-Res) scored considerably better than all benchmark methods, reducing the error of the best benchmark method by 40% for MAUC and by 25% for BCA, and achieving error rates comparable to the best predictions from the longitudinal prediction set D2. The best benchmark methods obtained a MAUC of 0.839 (BenchmarkMixedEffects) and a BCA of 0.771 (BenchmarkLastVisit). Among participant methods, IBM-OZ-Res had the best MAUC score of 0.905, significantly better than all entries other than GlassFrog-SM/-Average/-LCMEM-HDR, BGU-RF/-RFFIX/-LSTM, VikingAI-Logistic, EMC-EB, Rocket and Tohka-Ciszek-RandomForestLin according to the bootstrap hypothesis test (same methodology as in D2). This is further confirmed in Supplementary Fig. 12 by the variability of ranks under boostrap samples of the dataset, as these teams often remain at the top of the ranking. IBM-OZ-Res also had the best BCA score of 0.830 among participants. Among ensemble methods, ConsensusMean obtained the best Diagnosis MAUC of 0.917. In contrast, the best randomised prediction (RandomisedBest) obtained an MAUC of 0.811 and a BCA of 0.783. MAUC and BCA performance metrics had a relatively high correlation across all submissions ($r=0.9$, Supplementary Fig. 5).

For Ventricle volume prediction on D3, the best prediction (GlassFrog-LCMEM-HDR) scored considerably better than all benchmark methods, reducing the error of the best benchmark methods by 58% for MAE and 41% for WES, and achieving error rates comparable to the best predictions of D2. Here, the best benchmark methods had an overall Ventricle MAE of 1.13 (BenchmarkMixedEffects) and WES of 0.64 (BenchmarkLastVisit). Among participant submissions, GlassFrog-LCMEM-HDR had the best MAE of 0.48, significantly lower than all other submitted entries according to the Wilcoxon signed-rank test – this is also confirmed in Supplementary Fig. 14 by the rank distribution under dataset boostraps. GlassFrog-LCMEM-HDR also had the best Ventricle WES of 0.38, while submissions by team DIKU had the best Ventricle CPA of 0.05. Among ensemble methods, ConsensusMedian obtained a Ventricle MAE of 0.71 (4th best) and WES of 0.65 (7th best). In contrast, the best randomised prediction (RandomisedBest) obtained a Ventricle MAE of 0.92, WES of 0.50 and CPA of 0. As in D2, MAE and WES scores in D3 for Ventricles had very high correlation ($r=0.99$, Supplementary Fig. 5), while CPA showed weak correlation with MAE ($r=0.24$, p-value = 0.17) and WES ($r=0.37$, p-value $<0.032$).

For ADAS-Cog13 on D3, the predictions submitted by participants again did not perform better than the best benchmark methods. BenchmarkMixedEffects had the best MAE of 4.23, which was significantly lower than all entries by other challenge participants. Moreover, the MAE of 4.23 was only marginally worse than the equivalent error (4.19) by the same model on D2. BenchmarkMixedEffects also had the best ADAS-Cog13 WES of 4.23, while ATRI-Biostat-MA had the best ADAS-Cog13 CPA of 0.04. Among participants’ submissions, CyberBrains ranked first in ADAS-Cog13 prediction with a MAE of 4.72, an error 11% higher than the best benchmark. Among ensemble methods, ConsensusMean obtained an ADAS-Cog13 MAE of 4.58, WES of 4.34, better than all participants’ entries. As in D2, the best randomised predictions (RandomisedBest) obtained an ADAS-Cog13 MAE of 4.54 (2nd best) and WES of 4.50 (3rd best). As in D2, MAE and WES scores for ADAS-Cog13 had high correlation ($r$ = 0.97, Supplementary Fig. 5), while CPA showed weak, non-significant correlation with MAE ($r$ = 0.34, p-value $\approx$ 0.052) or WES ($r$ = 0.33, p-value $\approx$ 0.057).

Results on the custom prediction sets are presented in Supplementary Table 6.

To understand what characteristics of algorithms could have yielded higher performance, we show in Figure 1 associations from a general linear model between predictive performance and feature selection methods, different types of features, methods for data imputation, and methods for forecasting of target variables. For each type of feature/method and each target variable (clinical diagnosis, ADAS-Cog13 and Ventricles), we show the distribution of estimated coefficients from a general linear model, derived from the approximated inverse Hessian matrix at the maximum likelihood estimator (see Online Methods section 5.3). From this analysis we removed outliers, defined as submissions with ADAS MAE higher than 10 and Ventricle MAE higher than 1.15 (%ICV). For all plots, distributions to the right of the gray dashed vertical line denote increased performance compared to baseline (i.e. when those characteristics are not used).

For feature selection, Figure 1 shows that methods with manual selection of features tend to be associated with better predictive performance in ADAS-Cog13 and Ventricles. In terms of feature types, CSF and DTI features were generally associated with an increase in predictive performance for clinical diagnosis, while augmented features were associated with performance improvements for ventricle prediction. In terms of data imputation methods, while some differences can be observed, no clear conclusions can be drawn. In terms of prediction models, the only positive association that indicates increased performance is in the neural networks for ventricle prediction. However, given the small number of methods tested ($<$50) and the large number of degrees of freedom, these results should be interpreted with care.

When comparing to previous state-of-the-art results in the literature, the best TADPOLE methods show similar or higher performance in AD diagnostic classification while also tackling a harder problem than most previous studies of predicting future, rather than estimating current, classification. A comparison of 15 studies presented by moradi2015machine reported lower performance (maximum AUC of 0.902 vs 0.931 obtained by the best TADPOLE method) for the simpler two-class classification problem of separating MCI-stable from MCI-converters in ADNI. A more recent method by long2017prediction reported a maximum AUC of 0.932 and accuracy of 0.88 at the same MCI-stable vs -converter classification task. However, a) TADPOLE’s discrimination of CN-converters from CN-stable subjects is harder as disease signal is weaker at such early stages, and b) the predictive performance drops in three-class problems like TADPOLE compared to two-class. Furthermore, the best out of 19 algorithms in the CADDementia Challenge bron2015standardized obtained an MAUC of 0.78.

We are unaware of previous studies forecasting future ventricle volume or ADAS-Cog13, so TADPOLE sets a new benchmark state-of-the-art performance on these important prediction tasks.

The results on the longitudinal D2 prediction set suggest no clear winner on predicting all target variables – no single method performed well on all tasks. While Frog had the best overall submission with the lowest sum of ranks, for each performance metric individually we had different winners: Frog (clinical diagnosis MAUC of 0.931), ARAMIS-Pascal (clinical diagnosis BCA of 0.850), BenchmarkMixedEffects (ADAS-Cog13 MAE and WES of 4.19), VikingAI-Sigmoid (ADAS-Cog13 CPA of 0.02), EMC1-Std/EMC1-Custom (ventricle MAE of 0.41 and WES or 0.29), and DIKU-ModifiedMri-Std/-Custom (ventricle CPA of 0.01). Moreover, on the cross-sectional D3 prediction set, the methods by Glass-Frog had the best performance. Associations of method-type with increased performance in Fig. 1 confirm no clear increase in performance for any types of prediction methods (with the exception of neural networks for ventricle volume prediction). This raises an important future challenge to algorithm designers to develop methods able to perform well on multiple forecasting tasks and also in situations with limited data, such as D3.

Consistently strong results from ensemble methods (ConsensusMean/ConsensusMedian outperformed all others on most tasks) might suggest that different methods over-estimate future measurements for all subjects while others under-estimate them, likely due to the underlying assumptions they make. This is confirmed by plots of the difference between true and estimated measures (Supplementary Figures 6–8), where most methods systematically under- or over-estimate in all subjects. However, even if methods were completely unbiased, averaging over all methods could also help predictions by reducing the variance in the estimated target variables.

ADAS-Cog13 scores were more difficult to forecast than clinical diagnosis or ventricle volume. The only single method able to forecast ADAS-Cog13 better than informed random guessing (RandomisedBest) was the BenchmarkMixedEffects, a simple mixed effects model with no covariates and age as a regressor. The difficulty could be due to variability in administering the tests or practice effects. A useful target performance level comes from the 4 points change generally used to identify responders to a drug treatment grochowalski2016examining . With the exception of the ensemble method, all submitted forecasts failed to produce mean error below 4, highlighting the substantial challenge of estimating change in ADAS-Cog13 over the 1.4 year interval – although over longer time periods, non-trivial forecasts are likely to improve in comparison to RandomisedBest, which is independent of time period. Nevertheless, for the longitudinal D2 prediction set, the MAE in ADAS-Cog13 from ConsensusMean was 3.75, which restores hope in forecasting cognitive score trajectories even over relatively short timescales.

For clinical diagnosis, the best performance on the limited, cross-sectional D3 prediction set was similar to the best performance on the D2 longitudinal prediction set: 0.917 vs 0.931 for MAUC (p-value = 0.14), representing a 3% error increase for D3 compared to D2. Slightly larger and significant differences were observed for ADAS MAE (3.75 vs 4.23, p-value $<$ 0.01) and Ventricle MAE (0.38 vs 0.48, p-value $<$ 0.01). It should be noted that Ventricle predictions for D3 were extremely difficult, given that only 25% of subjects to be forecasted had MRI data in D3. This suggests that, for clinical diagnosis, current forecast algorithms are reasonably robust to lack of longitudinal data and missing inputs, while for ADAS and Ventricle volume prediction, some degree of performance is lost. Future work is also required to determine the optimal balance of input data quality and quantity versus cost of acquisition.

DTI and CSF features are most associated with increases in clinical diagnosis forecast performance. CSF, in particular, is well established as an early marker of AD jack2010hypothetical and likely to help predictions for early-stage subjects, while DTI, measuring microstructure damage, may be informative for middle-stage subjects. On the other hand, for prediction of ventricle volume, augmented features had the highest association with increases in prediction performance. Future work is required to confirm the added value of these features and others in a more systematic way.

TADPOLE Challenge has several limitations that future editions of the challenge may consider addressing. One limitation is the reliability of the three target variables: clinical diagnosis, ADAS-Cog13 and Ventricle volume. First of all, clinical diagnosis has only moderate agreement with gold-standard neuropathological post-mortem diagnosis. In particular, one study beach2012accuracy has shown that a clinical diagnosis of probable AD has sensitivity between 70.9% and 87.3% and specificity between 44.3% and 70.8%. With the advent of post-mortem confirmation in ADNI, future challenges might address this by evaluating the algorithms on subjects with pathological confirmation. Similarly, ADAS-Cog13 is known to suffer from low reliability across consecutive visits grochowalski2016examining , and TADPOLE algorithms fail to forecast it reliably. However, this might be related to the short time-window (1.4 years), and more accurate predictions might be possible over longer time-windows, when there is more significant cognitive decline. Ventricle volume measurements depend on MRI scanner factors such as field strength, manufacturer and pulse sequences han2006reliability , although these effects have been removed to some extent by ADNI through data preprocessing and protocol harmonization. TADPOLE Challenge also assumes all subjects either remain stable or convert to Alzheimer’s disease, whereas in practice some of them might develop other types of neurodegenerative diseases.

For performance evaluation, we elected to use very simple yet reliable metrics as the primary performance scores: the multiclass area under the curve (mAUC) for the clinical categorical variable and the mean absolute error (MAE) for the two numerical variables. While the mAUC accounts for decision confidence, the MAE does not, which means that the confidence intervals submitted by participants do not contribute to the rankings computed in Tables 2 and 3. While the weighted error score (WES) takes confidence intervals into account, we consider it susceptible to “hacking”, e.g. participants might assign high confidence to only one or two data points and thereby skew the score to ignore most of the predictions – in practice, we did not observe this behaviour in any submission. For clinical relevance, we believe that confidence intervals are an extremely important part of such predictions and urge future studies to consider performance metrics that require and take account of participant-calculated confidence measures.

TADPOLE has other limitations related to the algorithms’ comparability and generalisability. First of all, the evaluation and training were both done on data collected by ADNI – in future work, we plan to assess how the models will generalise on different datasets. Another limitation is that we can only compare full methods submissions and not different types of features, and strategies for data imputation and prediction used within the full method. While we tried to evaluate the effect of these characteristics in Figure 1, in practice the numbers were small and hence most effects did not reach statistical significance. Moreover, the challenge format does not provide an exhaustive comparison of all combinations of data processing, predictive model, features, etc., so does not lead to firm conclusions on the best combinations but rather provides hypotheses for future testing. In future work, we plan to test inclusion of features and strategies for data imputation and prediction independently, by changing one such characteristic at a time.

Another limitation is that the number of controls and MCI converters in the D4 test set is low (9 MCI converters and 9 control converters). However, these numbers will increase over time as ADNI acquires more data, and we plan to re-run the evaluation at a later stage with the additional data acquired after April 2019. A subsequent evaluation will also enable us to evaluate the TADPOLE methods on longer time-horizons, over which the effects of putative drugs would be higher.

The challenge uses data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) weiner2017recent . Specifically, the TADPOLE Challenge made four key data sets available to the challenge participants:

• •

  D1: The TADPOLE standard training set draws on longitudinal data from the entire ADNI history. The data set contains measurements for every individual that has provided data to ADNI in at least two separate visits (different dates) across three phases of the study: ADNI1, ADNI GO, and ADNI2.

• •

  D2: The TADPOLE longitudinal prediction set contains as much available data as we could gather from the ADNI rollover individuals for whom challenge participants are asked to provide predictions. D2 includes data from all available time-points for these individuals. It defines the set of individuals for which participants are required to provide forecasts.

• •

  D3: The TADPOLE cross-sectional prediction set contains a single (most recent) time point and a limited set of variables from each rollover individual in D2. Although we expect worse predictions from this data set than D2, D3 represents the information typically available when selecting a cohort for a clinical trial.

• •

  D4: The TADPOLE test set contains visits from ADNI rollover subjects that occurred after 1 Jan 2018 and contain at least one of the three outcome measures: diagnostic status, ADAS-Cog13 score, or ventricle volume.

While participants were free to use any training datasets they wished, we provided the D1-D3 datasets in order to remove the need for participants to pre-process the data themselves, and also to be able to evaluate the performance of different algorithms on the same standardised datasets. Participants that used custom training data sets were asked also to submit results using the standard training data sets to enable direct performance comparison. We also included the D3 cross-sectional prediction set in order to simulate a clinical trial scenario. For information on how we created the D1-D4 datasets, see Supplementary section A. The software code used to generate the standard datasets is openly available on Github: https://github.com/noxtoby/TADPOLE.

Table 4 shows the demographic breakdown of each TADPOLE data set as well as the proportion of biomarker data available in each dataset. Many entries are missing data, especially for certain biomarkers derived from exams performed on only subsets of subjects, such as tau imaging (AV1451). D1 and D2 also included demographic data typically available in ADNI (e.g. education, marital status) as well as standard genetic markers (e.g. Alipoprotein E – APOE epsilon 4 status).

For evaluation of clinical status predictions, we used similar metrics to those that proved effective in the CADDementia challenge bron2015standardized : (i) the multiclass area under the receiver operating characteristic curve (MAUC) and (ii) the overall balanced classification accuracy (BCA). For ADAS-Cog13 and ventricle volume, we used three metrics: (i) mean absolute error (MAE), weighted error score (WES) and coverage probability accuracy (CPA). BCA and MAE focus purely on prediction accuracy ignoring confidence, MAUC and WES account for accuracy and confidence, while CPA assesses the confidence interval only. The formulas for each performance metric are summarised in Table 5. See the TADPOLE white paper marinescu2018tadpole for further rationale for choosing these performance metrics. In order to characterise the distribution of these metric scores, we compute scores based on 50 bootstraps with replacement on the test dataset.

To identify which features and types of algorithms enable good predictions, we annotated each TADPOLE submission with a set of 21 attributes related to (i) feature selection (manual/automatic and large vs. small number of features), (ii) feature types (e.g. “uses Amyloid PET”), (iii) strategy for data imputation (e.g. “patient-wise forward-fill”) and (iv) prediction method (e.g. “neural network”) for clinical diagnosis and ADAS/Ventricles separately. To understand which of these annotations were associated with increased performance, we applied a general linear model kiebel2007general , $Y=X\beta+\epsilon$, where $Y$ is the performance metric (e.g. diagnosis MAUC), $X$ is the nr_submissions x 21 design matrix of binary annotations, and $\beta$ show the contributions of each of the 21 attributes towards achieving the performance measure $Y$.

Team: AlgosForGood    (Members: Tina Toni, Marcin Salaterski, Veronika Lunina, Institution: N/A)
Feature selection: Manual + Automatic: Manual selection of uncorrelated variables from correlation matrix and automatic selection of variables that have highest cumulative hazard rates in survival regression from MCI to AD.
Selected features: Demographics (age, education, gender, race, marital status), cognitive tests (ADAS-Cog13, RAVLT immediate, RAVLT forgetting, CDRSOB, ADAS11, FDG), Ventricles, AV45, ICV, APOE4.
Missing data: Fill-in using last available value from corresponding patient
Confounder correction: none
Method category: Statistical Regression / Proportional hazards model
Prediction method:

• •

  Diagnosis: Aalen additive regression constructing a cumulative hazard rate for progressing to AD.

• •

  ADAS-Cog13: regression using change in ventricles/ICV as predictive variable, stratified by last known diagnosis.

• •

  Ventricles: regression over month, with several pre-processing steps: 1. Enforced monotonicity by accumulating maximum value, 2. For APOE positive patients used only last three visits due to non-linearity 3. Stratified by diagnosis
  

Team: Apocalypse    (Members: Manon Ansart, Stanley Durrleman, Institution: Institut du Cerveau et de la Moelle épinière, ICM, Paris, France)
Feature selection: Manual – important features were identified by looking at the correlations with the diagnosis. Personal knowledge of the disease was also used to complement those results and select relevant features. Different feature sets were compared using cross-validation.
Selected features: Cognitive features (ADAS-Cog13, MMSE, RAVLT immediate, FAQ, CDRSOB), MRI features (WholeBrain, Entorhinal, Fusiform, MidTemp, Ventricles, Entorhinal, Hippocampus), APOE4, education, age, clinical diagnosis
Missing data: Filled in using the mean feature value
Confounder correction: none
Method category: Machine learning / Regression
Prediction method: Linear regression is used to first predict the future of a set of features (MMSE, ADAS-Cog13, CDRSOB, RAVLT, Ventricles) at the prediction dates. Afterwards, an SVM is used to predict the current diagnosis for each prediction date, based on the forecasted features as well as other features which are constant for the subject (APOE4, education, age at last known visit).


Team: ARAMIS-Pascal    (Members: Pascal Lu, Institution: Institut du Cerveau et de la Moelle épinière, ICM, Paris, France)
Feature selection: Manual, based on known biomarkers from the literature.
Selected features: APOE4, cognitive tests (CDRSB, ADAS11, ADAS-Cog13, MMSE, RAVLT immediate, FAQ), volumetric MRI (hippocampus, ventricles, whole brain, entorhinal, fusiform, middle temporal, ICV), whole brain FDG, CSF biomarkers (amyloid-beta, tau, phosphorylated tau), education and age.
Missing data: Imputed using the average biomarker values across the population.
Confounder correction: none
Method category: Statistical Regression / Proportional hazards model
Prediction method: For diagnosis prediction, the Aalen model for survival analysis was used to predict the conversion from MCI to AD, which returns the probability of a subject remaining MCI as a function of time. The method assumes cognitively normal and dementia subjects will not convert and thus will remain constant. The method did not predict ADAS-Cog13 or Ventricles.
Publication link: https://hal.inria.fr/tel-02433613/document


Team: ATRI_Biostat (Members: Samuel Iddi^1,2, Dan Li¹, Wesley K. Thompson³ and Michael C. Donohue¹. Institutions: ¹Alzheimer’s Therapeutic Research Institute, USC, USA; ²Department of Statistics and Actuarial Science, University of Ghana, Ghana; ³Department of Family Medicine and Public Health, University of California, USA)
Feature selection: Automatic - features were ranked by their importance in classifying diagnostic status using a random forest algorithm. All cognitive tests, imaging biomarkers, demographic information and APOE status were considered as potential features
Selected features: ADAS-Cog13, EcogTotal, CDRSOB, FAQ, MOCA, MMSE, RAVLT immediate, Ventricles/ICV, Entorhinal, Hippocampus/ICV and FDG Pet. Age, gender and APOE status were included as covariates. The interaction between diagnosis at first available visit and years since first visit was also considered.
Missing data: Imputed using the MissForest Algorithm, based on a non-parametric random forest methodology stekhoven2011missforest . The algorithm was chosen based on its ability to handle mixed-type outcomes, complex interactions and non-linear relationships between variables.
Confounder correction: APOE status, last known clinical status, age and gender.
Method category: Machine learning and data-driven disease progression models
Prediction method: The method applied different types of mixed-effects models to forecast ADAS-Cog13 and Ventricles, and then used a Random Forest classifier to predict the clinical diagnosis from the forecasted continuous scores.

• •

  JMM    – Joint Mixed-effect Modelling with subject-specific intercept and slope.

• •

  LTJMM    – Latent Time Joint Mixed-effect Modelling with subject-specific intercept, slope and time-shift

• •

  MA    – Model average of the two models above, as well as a third model where random intercepts are shared across outcomes.

Confidence Intervals: The 50% prediction intervals for ADAS-Cog13 and Ventricles were obtained by taking the 25th and 75th percentile of the posterior predicted samples.
Publication link: https://braininformatics.springeropen.com/articles/10.1186/s40708-019-0099-0


Team: BGU (Members: Aviv Nahon, Yarden Levy, Dan Halbersberg, Mariya Cohen, Institution: Ben Gurion University of the Negev, Beersheba, Israel)
Feature selection: Automatic – used the following algorithm: 1. Find the two variables with highest correlation (Spearman for continuous variables and Mutual information for discrete variables). 2. Compute the correlation of each variable with the target variables separately and remove the variable with the lower correlation. 3. If there are still pairs of variables with a correlation of more than 80%, repeat from step 1.
Selected features: Cognitive tests (CDRSOB, MMSE, RAVLT, MOCA, all Ecog), MRI biomarkers (Freesurfer cross-sectional and longitudinal), FDG- PET (hypometabolic convergence index), AV45 PET (Ventricles, Corpus Callosum, Hippocampus), White-matter hypointensities’ volume, CSF biomarkers (amyloid-beta, tau, phosphorylated tau). For each continuous variable, an additional set of 20 augmented features was used, representing changes and trends in variables (e.g. mean, standard deviation, trend mean, trend standard deviation, minimum, mean minus global mean, baseline value, last observed value). This resulted in 233 features, which were used for prediction.
Missing data: Random forest can deal automatically with missing data. LSTM network used indicator that was set to zero for missing data.
Confounder correction: None
Method category: Machine learning
Prediction method:

• •

  BGU-LSTM   : This model consisted of two integrated neural networks: an LSTM network for modelling continuous variables and a feed-forward neural network for the static variables.

• •

  BGU-RF   : A semi-temporal Random Forest was used which contained the augmented features.

• •

  BGU-RFFIX   : Same as BGU-RF, but with small correction for the prediction of diagnosis: whenever the model predicted AD with probability higher than 80%, the probability of CN was changed to zero and vice versa.
  
  

Team: BIGS2    (Members: Huiling Liao, Tengfei Li, Kaixian Yu, Hongtu Zhu, Yue Wang, Binxin Zhao, Institution: University of Texas, Houston, USA)
Feature selection: Automatic – used auto-encoder to extract aggregated features.
Selected features: All continuous features in D1/D2, which represented the input for the autoencoder. Apart from the autoencoder-extracted features, other features used for the classifier were demographic information, APOE status, whole brain biomarkers from MRI (volume) and PET (FDG, PIB and AV45), and MMSE.
Missing data: SoftImpute method mazumder2010spectral was used for imputing missing data. The complete dataset was then used as input to the autoencoder.
Confounder correction: none
Method category: Regression and Machine Learning
Prediction method: Linear models were used to predict ADAS-Cog13 and Ventricle scores independently. For prediction of clinical diagnosis, a random forest was used based on the autoencoder-extracted features and the other selected features.


Team: Billabong    (Members: Neil Oxtoby, Institution: University College London, UK)
Feature selection: Manual, using knowledge from literature
Selected features: MRI biomarkers normalised by ICV (ventricles, hippocampus, whole brain, entorhinal, fusiform, middle temporal), FDG, AV45, CSF biomarkers (amyloid beta, tau, phosphorylated tau) and cognitive tests (ADAS-Cog13, MMSE, MOCA, RAVLT immediate). Separate submissions (Billabong-UniAV45, Billabong-MultiAV45) were made which also included AV45, that was initially excluded due to noise.
Missing data: Only imputed during staging via linear regression against age. The method can deal with missing data during training.
Confounder correction: None
Method category: Data-driven disease progression model
Prediction method: For each selected feature independently, a data-driven longitudinal trajectory was estimated using a differential equation model based on Gaussian Process Regression oxtoby2018data . Subjects were staged using either a multivariate or univariate approach:

• •

  Billabong-Uni: Univariate staging which estimates disease stage for each target variable independently.

• •

  Billabong-Multi: Multivariate staging that combines all selected features, producing an average disease stage.

For the prediction of clinical diagnosis, the historical ADNI diagnoses were mapped to a linear scale using partially-overlapping squared-exponential distribution functions. The linear scale and the three distributions were used to forecast the future diagnoses.
Custom prediction set: Predictions were made also for a custom dataset, which was similar to D3 but missing data was filled in using the last available biomarker data.
Confidence Intervals: The 25th and 75th percentiles of the GPR posterior were each integrated into a trajectory to obtain 50% confidence (credible) intervals for the forecasts of ADAS-Cog13 and Ventricles/ICV.
Publication link: https://doi.org/10.1093/brain/awy050


Team: BORREGOSTECMTY    (Members: José Gerardo Tamez-Peña, Institution: Tecnologico de Monterrey, Monterrey, Mexico)
Feature selection: Automatic, using bootstrapped stage-wise selection.
Selected features: Main cognitive tests (excluding subtypes), MRI biomarkers, APOE status, demographic information (age, gender, education) and diagnosis status. Augmented features were further constructed from the MRI set: the cubic root of all volumes, the square root of all surface areas, the compactness, the coefficient of variation, as well as the mean value and absolute difference between the left and right measurements.
Missing data: Imputed using nearest-neighbourhood strategy based on L1 norm.
Confounder correction: Gender and intracranial volume (ICV) adjustments relative to controls.
Method category: Regression (ensemble of statistical models)
Prediction method: ADAS-Cog13 and Ventricles were predicted using an ensemble of 50 linear regression models, one set for each diagnostic category. The best models were selected using Bootstrap Stage-Wise Model Selection, using statistical fitness pencina2008evaluating tests to evaluate models and features to use within the models. All selected models were then averaged in a final prediction using bagging. For prediction, the last known diagnosis of the subject was used to select the category of models for forecasting.
For the prediction of clinical diagnosis, a two-stage approach was used based on prognosis and time-to-event estimation. The prognosis approach used an ensemble of 50 regression models to estimate the future diagnosis, while the time-to-event method used an ensemble of 25 models to estimate the square root of the time it took for a subject to convert to MCI or AD. These approaches were performed independently for CN-to-MCI, MCI-to-AD and CN-to-AD conversion.
Confidence Intervals: The 50% confidence intervals for ADAS-Cog13 and Ventricle volume were estimated by extracting the interquartile range of the 50 regression estimates.
Repository link: https://github.com/joseTamezPena/TADPOLE


Team: BravoLab    (Members: Aya Ismail, Timothy Wood, Hector Corrada Bravo, Institution: University of Maryland, USA)
Feature selection: Automatic, using a random forest to select features with highest cross-entropy or GINI impurity reduction.
Selected features:

• •

  Ventricle prediction: MRI volumes of ventricular sub-regions (Freesurfer cross-sectional and longitudinal)

• •

  ADAS-Cog13 prediction: RAVLT, Diagnosis, MMSE, CDRSOB

• •

  Diagnosis prediction: ADAS-Cog13, ADAS11, MMSE, CSRSOB

Missing data: Imputation using Hot Deck andridge2010review was done only for data missing at random.
Confounder correction: None
Method category: Machine learning
Prediction method: A long-short term memory network (LSTM) with target replication was trained independently for each category: Diagnosis, ADA13 and Ventricles. All existing data was used for the first forecast, after which the output of the last prediction was used as input for the next prediction, along with other features that remain constant over time. Since subjects had a different number of visits and available biomarker data, the network was adapted to accept inputs of variable length. For predictions, the network used a weighted mean absolute error as a loss function. In addition, for the prediction of the clinical diagnosis, a soft-max function was used to get the final prediction.



Team: CBIL    (Members: Minh Nguyen, Nanbo Sun, Jiashi Feng, Thomas Yeo, Institution: National University of Singapore, Singapore)
Feature selection: Manual, based on model performance on D1 subset.
Selected features: Cognitive tests (CDRSOB, ADAS11, ADAS-Cog13, MMSE, RAVLT immediate, learning, forgetting and percent forgetting, MOCA, FAQ), MRI biomarkers (entorhinal, fusiform, hippocampus, ICV, middle temporal, ventricles, whole brain), whole brain AV45 and FDG, CSF biomarkers (amyloid-beta, tau, phosphorylated tau).
Missing data: Imputation using interpolation.
Confounder correction: None
Method category: Machine learning and data-driven disease progression model
Prediction method: Recurrent neural network adapted for variable duration between time-points. A special loss function was designed, which ensured forecasts at timepoints close together are more correlated than those at timepoints further apart.
Confidence Intervals: hardcoded values
Publication link: https://www.biorxiv.org/content/10.1101/755058v1
Repository link: https://github.com/ThomasYeoLab/CBIG/tree/master/stable_projects/predict_phenotypes/Nguyen2020_RNNAD


Team: Chen-MCW    (Members: Gang Chen, Institution: Medical College of Wisconsin, Milwaukee, USA)
Feature selection: Manual
Selected features: ADAS-Cog13, MMSE, MRI volumes (hippocampus, whole brain, entorhinal, fusiform and middle temporal), APOE status, gender and education.
Missing data: No imputation performed.
Confounder correction: None
Method category: Regression and data-driven disease progression model
Prediction method: Prediction of ADAS-Cog13 and Ventricles was made using linear regression using age, APOE status, gender and education as covariates. Different models were estimated for CN, MCI and AD subjects. For diagnosis prediction, an AD risk stage was calculated based on the Event-based probability (EBP) model chen2016staging . Prediction of clinical diagnosis was then made based on two approaches:

• •

  Chen-MCW-Std: Predict diagnosis based on AD stage as well as APOE4, gender and education using a Cox proportional hazards model.

• •

  Chen-MCW-Stratify: As above, but the model was further stratified based on AD risk stages, into low risk and high-risk.
  
  

Team: CN2L (Members: Ke Qi¹, Shiyang Chen^1,2, Deqiang Qiu^1,2, Institutions: ¹Emory University, ²Georgia Institute of Technology)
Feature selection: Automatic
Selected features: For the neural network, all features in D1/D2 are used; For the random forest, the main cognitive tests, MRI biomarkers (cross-sectional only), FDG, AV45, AV1451, DTI, CSF, APOE, demographics and clinical diagnosis are used.
Missing data: Imputation in a forward filled manner (i.e. using last available value).
Confounder correction: None
Method category: Machine learning
Prediction method:

• •

  CN2L-NeuralNetwork    : 3-layer recurrent neural network, 1024 units/layer. Dropout layers (dropout rate:0.1) were added to output and state connections to prevent overfitting. Adam method was used for training. Validation was performed using a leave-last-time-point-out approach.

• •

  CN2L-RandomForest   : Random forest method was used, where features of small importance for diagnosis prediction were filtered out. For the prediction of clinical diagnosis, an ensemble of 200 trees was trained on a class-balanced bootstrap sample of the training set. For the prediction of ADAS-Cog13 and Ventricles, an ensemble of 100 trees was used. Different predictions are made for different previous visits of a patient, and the final prediction is taken as the average of all predictions.

• •

  CN2L-Average   : The average of the above two methods.

Confidence Intervals: Confidence intervals are estimated based on probabilities output of the model.
Publication link: https://cds.ismrm.org/protected/18MPresentations/abstracts/3668.html Chen et al., ISMRM, 2018 chen2018is

Team: CyberBrains    (Members: Ionut Buciuman, Alex Kelner, Raluca Pop, Denisa Rimocea, Kruk Zsolt, Institution: Vasile Lucaciu College, Baia Mare, Romania)
Feature selection: Manual
Selected features: MRI volumes (Ventricles, middle temporal), ADAS-Cog13, APOE status
Missing data: For subjects with no ventricle measurements, authors computed an average value based on ADAS-Cog13 tests. This was used especially for D3 predictions.
Confounder correction: None
Method category: Regression
Prediction method: Fit a linear model of monthly difference in ventricle volume, as a function of ventricle volume, stratified by clinical diagnosis and ventricle volumes smaller and larger than 140,000 mm3. A similar model is applied for ADAS-Cog13 prediction, but stratified by APOE status and middle temporal volume smaller or greater than 16,000 mm3. Prediction of clinical diagnosis also used a linear model that was stratified based on the ADAS-Cog13, for ADAS-Cog13 ranges between 10 and 45. For ADAS-Cog13 greater than 45 and smaller than 10, pre-defined values were used for the probabilities of each diagnosis.



Team: DIKU    (Members: Mostafa Mehdipour Ghazi^1,2,3,5, Mads Nielsen^1,2,3, Akshay Pai^1,2,3, Marc Modat^4,5, M. Jorge Cardoso^4,5, Sebastien Ourselin^4,5, Lauge S$\o$rensen^1,2,3; Institutions: ¹Biomediq A/S, ²Cerebriu A/S, ³University of Copenhagen, Denmark, ⁴King’s College London, UK, ⁵University College London, UK)
Feature selection: Semi-automatic; linear discriminant analysis (LDA) was applied to select the top most-informative biomarkers, and ventricular volume and a few other MRI measures were subsequently manually added.
Selected features: Cognitive tests (CDR-SB, ADAS-11, ADAS-13, MMSE, FAQ, MOCA, RAVLT-Immediate, RAVLT-Learning, RAVLT-Percent-Forgetting), CSF measures (amyloid-beta, phosphorylated tau), MRI volumetric measures divided by ICV (ventricles, hippocampus, whole brain, entorhinal, fusiform, middle temporal).
Missing data: Method automatically deals with missing data.
Confounder correction: Linear transformation of age as part of the algorithm.
Method category: Data-driven disease progression model.
Prediction method: For predicting ADAS-13 and Ventricles, a data-driven disease progression model was used, which estimated a parametric trajectory for each selected feature over a common disease progression axis reflecting an estimated latent disease progression score (DPS). The chosen parametric function was generalised logistic function (Richard’s curve), and the DPS was a linear transformation of the age of subjects representing the subject-specific time shift and progression speed. The constrained fitting was performed alternating between estimation of subject-specific DPS transformations and global biomarker trajectories using L2-norm loss functions. The authors made three submissions:

• •

  DIKU-GeneralisedLog-Std: constrained, generalised logistic function for the trajectory model;

• •

  DIKU-ModifiedLog-Std: constrained sigmoid function for the trajectory model;

• •

  DIKU-ModifiedMri-Std: as above, but separately fitting MRI biomarkers for Ventricles prediction.

The above models were trained on D1 data only. Authors also made predictions from a custom training set (D1+D2 together), named DIKU-***-Custom. Clinical diagnosis was predicted based on the DPS scores using both a Bayesian classifier with likelihoods modeling using Gaussian mixture models, as well as an ensemble of LDAs. The final prediction was obtained through bagging of the two classifiers’ predictions. The whole method and a robust extension developed post-TADPOLE is described in ghazi2019robust .
Confidence Intervals: They were obtained by using bootstrapping via Monte Carlo resampling and evaluating the model performance assuming a Gaussian distribution.
Publication link: https://arxiv.org/abs/1908.05338



Team: DIVE    (Members: Razvan Marinescu, Institution: University College London, UK, Massachusetts Institute of Technology, USA)
Feature selection: Manual
Selected features: FDG, AV45, CDRSOB, ADAS-Cog13, MRI volumes (ventricles, hippocampus, whole brain, entorhinal, middle temporal), CSF (amyloid-beta, tau, phosphorylated tau)
Missing data: Method automatically deals with missing data
Confounder correction: None
Method category: Data-driven disease progression model
Prediction method:
For predicting the ADAS-Cog13 and Ventricle volume, the “Data-Driven Inference of Vertexwise Evolution” (DIVE) algorithm was used marinescu2019dive , which clusters the input biomarkers based on how similar their progression is over the disease time-course. While the original DIVE method was a spatio-temporal model, for TADPOLE it was applied on extracted features directly. The model estimates a parametric, sigmoidal trajectory of the biomarkers, which are a function of subjects’ disease progression scores (DPS), representing a linear transformation of their age. Subject-specific parameters included the latent time-shift and progression speed, as well as an intercept. For the prediction of clinical diagnosis, the posterior probability of each class was computed given the future DPS scores using non-parametric Kernel Density Estimators (KDE), fitted on the DPS scores for each diagnostic class independently. Code for the model is available online:
Confidence Intervals: The model estimates a variance parameter under a gaussian noise model, which was scaled accordingly to obtain the 50% confidence intervals for ADAS-Cog13 and Ventricle volume.
Publication link: https://www.sciencedirect.com/science/article/pii/S1053811919301491
Repository link: https://github.com/mrazvan22/dive


Team: EMC1    (Members: Vikram Venkatraghavan, Esther Bron, Stefan Klein, Institution: Erasmus MC, The Netherlands)
Feature selection: Automatic – Only the subjects who had converted to AD were used for feature selection. Features with the largest changes over time after correcting for age, gender, education and ICV were selected
Selected features: 250 features from the set of FDG, AV45, DTI, MRI (cross-sectional Freesurfer volumes), Arterial Spin Labelling (ASL) MRI, CSF and cognitive tests.
Missing data: Imputed using nearest-neighbour interpolation. For D2, visits with missing diagnosis were excluded. For the D3 subjects with no known diagnosis, this was estimated using a nearest-neighbour search based on disease severity
Confounder correction: Corrected for age, gender, education and ICV using linear regression based on data from controls.
Method category: Data-driven disease progression model and machine learning
Prediction method: Authors hypothesize that aging and progression of AD are the primary causes for the change in biomarker values with time and that these changes eventually lead to a change in clinical status. To predict biomarker values at future timepoints, the rate of AD progression is estimated in each subject. This is followed by estimating the interactions of aging and AD progression in the progression of different biomarkers. Lastly, authors use the biomarkers estimated at the future timepoint to predict the change in clinical status. These steps are elaborated below:
Rate of Progression of AD: To assess the severity of AD, we estimated the sequence in which the selected features became abnormal in AD using a Discriminative Event-Based Model venkatraghavan2019disease and used it to estimate the disease severity at all the timepoints for each subject. A linear mixed effect model was fit to estimate the rate of change of disease severity for different subjects. This model was used for predicting the disease severity at all the future timepoints.
Interactions of aging and AD progression: For predicting the biomarker values at the future timepoint, we fit linear mixed effect models for each biomarker considering interactions between the estimated disease severity and age, with gender and ICV as additional covariates. This model was used to forecast the future values of all 250 selected features, including ADAS-Cog13 scores and Ventricle volumes.
Predicting the change in clinical status: For the diagnosis prediction, the forecasted values of the biomarkers and the last known clinical diagnosis of the subject were used as inputs for a soft-margin SVM classifier with a radial basis function kernel. Two separate submissions were made:

• •

  EMC1-Std (ID 1): ASL based features were excluded in this model

• •

  EMC1-Custom (ID 2): ASL based features were included in this model

Confidence Intervals: Standard errors of the predicted values of Ventricles and ADAS-Cog-13 were estimated by repeating the prediction procedure, including the estimation of disease severity, for 10 repetitions of bootstrap sampling. These standard errors were used to define the confidence intervals.
Publication link: https://doi.org/10.1016/j.neuroimage.2018.11.024
Repository link: https://github.com/88vikram/TADPOLE_submission_with_debm


Team: EMC-EB    (Members: Esther E. Bron, Vikram Venkatraghavan, Stefan Klein, Institution: Erasmus MC, The Netherlands)
Feature selection: Automatic – For the D2 prediction, features were selected that had the largest change over time in subjects who converted to AD using corrections for age, gender, education and ICV, i.e. the same approach as EMC1. For the D3 prediction, features with less than 50% missing data were selected.
Selected features: 200 (D2 prediction) and 338 (D3 prediction) from the set of clinical diagnosis, cognitive tests, MRI volumes (Freesurfer cross-sectional), FDG PET, DTI measures (FA, MD, RD, AD) and CSF features.
Missing data: Imputation using nearest-neighbour interpolation based on the subject’s earlier timepoints. If not possible, imputation by the mean of training set was used. Visits with no clinical diagnosis were excluded for classifier training.
Confounder correction: None
Method category: Machine learning
Prediction method: All predictions were based on SVMs. For diagnosis and ADAS-Cog13 prediction, respectively a classifier with balanced class weights and a regressor were trained to predict the target measures at the next visit. These predictions do not explicitly take account of time, but assume that the times between two visits are roughly equal. For Ventricle prediction, a regressor was trained to predict the change of ventricle volume per year. Ventricle volumes were normalized using ICV at baseline (ICV at current time point for D3). Using the predicted change, the normalized ventricle volume at each future visit was computed. For all predictions, authors used a radial basis function (RBF) kernel SVM, of which the C-parameter was set to $C=0.5$ and gamma to the reciprocal of the number of features. All features were normalized to zero mean and unit standard deviation.
Confidence Intervals: Bootstrap resampling ($n=100$) of the training set.
Repository link: https://github.com/tadpole-share/tadpole-algorithms/tree/master/tadpole_algorithms/models/ecmeb


Team: FortuneTellerFish (Members: Alexandra Young, Institutions: University College London, UK, King’s College London, UK)
Feature selection: Manual
Selected features: Age at assessment, age at scan, APOE4 status, education, gender, MRI volumes (ventricles, hippocampus, whole brain, entorhinal, fusiform, middle temporal, all major lobes, insula, and basal ganglia). The probability of being amyloid positive, obtained from joint mixture modelling of CSF amyloid-beta and global AV45, was also included as a feature. Two key features, disease subtype and stage, were derived from the Subtype and Stage Inference (SuStaIn) model based on the MRI features young2018uncovering .
Missing data: Imputed by averaging over the k-nearest neighbours with $k=5$
Confounder correction: Brain volumes were corrected for age, intracranial volume and field strength using linear regression. Parameters for the linear regression were estimated based on amyloid-negative controls.
Method category: Data-driven disease progression models + statistical regression
Prediction method: For ADAS-Cog13 and Ventricle prediction, a linear mixed effects model was used which used a different set of fixed/random effects, depending on the submission:

• •

  FortuneTellerFish-Control   : For Ventricles, fixed effects were age at scan and gender. For ADAS-Cog13 and MMSE the fixed effects were age at scan, education, APOE status and amyloid positivity. For all target measures, there was one random effect per subject.

• •

  FortuneTellerFish-SuStaIn   : two additional fixed effects from the SuStaIn model: subtype and stage.

For the prediction of clinical diagnosis, a multiclass error-correcting output codes (ECOC) classifier based on SVMs was trained with the following inputs: age at assessment, age at scan, APOE status, amyloid positivity, gender, education, SuStaIn subtype and stage, ADAS-Cog13, MMSE and ventricle volume. For diagnosis prediction at future timepoints, the forecasted values for ADAS-Cog13, MMSE and Ventricle volume were used as input to the classifier.


Team: Frog    (Members: Keli Liu, Christina Rabe, Paul Manser Institution: Genentech, USA)
Feature selection: Automatic using the Xgboost package chen2016xgboost
Selected features: Cognitive tests (ADAS-Cog13, CDRSB, MMSE, RAVLT), clinical diagnosis, MRI measurements, FDG PET measurements, APOE status and CSF measurements. For each longitudinal measurement (e.g. test scores and MRI), the following transformations were computed and used to augment the original feature set: most recent measurement, time since most recent measurement, the historical highest (lowest) measurement, time since the historical highest and lowest measurement, and the most recent change in measurement.
Missing data: Xgboost package automatically deals with missing data through inference based on reduction of training loss.
Confounder correction: None
Method category: Statistical prediction using regression
Prediction method: Flexible models and features were chosen automatically using gradient boosting (Xgboost package). Different models were trained for the following forecast windows: 0-8 months, 9-15, 16-27, 28-39, 40-60, $>$60 (given windows are for clinical status prediction, slightly different windows used for ADAS-Cog13 and ventricular volume prediction) . Variable importance scores from Xgboost suggest that MRI features play a bigger role in models trained for longer forecast windows.
Confidence Intervals: Standard deviation for prediction error was estimated based on cross validation (on training set). Normality of prediction errors was then assumed to construct prediction intervals based on estimated standard deviation.



Team: GlassFrog-LCMEM-HDR    (Members: Steven Hill¹, James Howlett¹, Robin Huang¹, Steven Kiddle¹, Sach Mukherjee², Anaïs Rouanet¹, Bernd Taschler², Brian Tom¹, Simon White¹, Institutions: ¹MRC Biostatistics Unit, University of Cambridge, UK; ²German Center for Neurodegenerative Diseases (DZNE), Bonn, Germany)
Feature selection:

• •

  MSM: Automatic, by selecting features which passed a likelihood ratio test when compared against a model with no covariates.

• •

  LCMEM: Manual

• •

  HDR: Automatic, selected via sparse Lasso regression.

Selected features:

• •

  MSM: D2 - gender, age, education, ADAS-Cog13, diagnosis, MMSE, CDRSOB, APOE status, first 5 principal components from imaging, amyloid positivity, tau level. D3 - gender, age, education, ADAS-Cog13, diagnosis, MMSE, Ventricles/ICV

• •

  LCMEM: ADAS-Cog13, gender, education, age at baseline

• •

  HDR: all features were provided to the method, excluding some features with many missing values

Missing data:

• •

  MSM: Filling with last known value, or nearest neighbour if feature was never observed.

• •

  LCMEM: Complete case analysis (assumption that missing data are missing at random)

• •

  HDR: Imputation using within-subject interpolation and nearest neighbour matching

Confounder correction: None
Method category: Combination of statistical regression and data-driven disease progression models
Prediction method: The prediction of clinical diagnosis was done using a Multi-State Model (MSM). Multi-state models (MSMs) kalbfleisch1985analysis are continuous-time Markov chain models, here with states corresponding to CN, MCI, AD, and transition rates estimated from the data using covariates selected as described above. The model accounts for noise in the historical diagnostic labels. Predictions for a given forecast month were made using the last observed disease state and associated covariates.

Prediction of ADAS-Cog13 was done using a Latent class mixed effects model (LCMEM). The model used four latent classes, where class membership probability was modelled via a multinomial logistic. For each latent class a specific linear mixed effects model defined a Gaussian latent process, with class-specific fixed effects, random effects for intercept, slope and square slope, and Gaussian noise. Age at baseline, gender and education were also included as covariates. Finally, a Beta cumulative distribution link function was used (and estimated simultaneously) between ADAS-Cog 13 and the latent process, to account for the departure from the Gaussian assumption on the outcome. The number of latent classes was optimised with the Bayesian Information Criterion (BIC).

Prediction of Ventricle volume was done using high-dimensional regression (HDR) and disease state-specific slope models: Subject-specific slopes were obtained by a combination of Lasso regression and shrinkage towards disease-state-specific shrinkage targets. Conversion times, from one disease state to another, were forecasted using the MSM model.
Confidence Intervals: LCMEM: the 50% confidence intervals for ADAS-Cog13 were obtained using a bootstrap approach. HDR: Confidence intervals were set as percentages of the predicted values.



Team: GlassFrog-SM    (members and affiliations as above)
Feature selection: Manual
Selected features: ADAS-Cog13, Ventricles/ICV, age at visit, APOE status, education, diagnosis
Missing data: For training, complete case analysis was performed. For prediction, imputation was performed for missing outcomes using a linear model with age, education, diagnosis and APOE status as covariates.
Confounder correction: None
Method category: Combination of statistical regression and data-driven disease progression models
Prediction method: The prediction of clinical diagnosis was using MSM models as in GlassFrog-LCMEM-HDR. The prediction for ADAS-Cog13 and Ventricles used a Slope Model (SM), which used a quadratic function to model the slope of the outcome variable as a function of the current outcome value and covariates. Covariates used were age at visit, education and APOE status.
Confidence Intervals: SM: Confidence intervals were set as percentages of the predicted values. The percentages used were manually selected and depended on the missingness of covariates for each individual.


Team: GlassFrog-Average    (members and affiliations as above)
Prediction method: The prediction of clinical diagnosis was using MSM models as in GlassFrog-LCMEM-HDR. For ADAS-Cog13 and Ventricles an ensemble approach was used that averaged the predictions from three methods: LCMEM, HDR and SM, as described above. Confidence interval bounds were also averaged.

Team: IBM-OZ-Res (Members: Noel Faux, Suman Sedai, Institution: IBM Research Australia, Melbourne, Australia)
Feature selection: using boosting regression
Selected features: Ventricle volume, AV45, FDG PET, cognitive tests, clinical diagnosis, age
Missing data: Imputed with zero; observations with missing ventricle volume are dropped.
Confounder correction: None
Method category: Machine Learning
Prediction method: A stochastic gradient boosting regression machine (GBM) was used to predict Ventricle Volume, with a Huber loss function. To reduce overfitting, a shrinkage mechanism was adopted, where the response of each tree is reduced by a factor of 0.01. Independent predictions were made for each individual visit, and averaged when a subject had more than one visit. For the prediction of clinical status, a similar GBM model was adopted, but with a multinomial deviance loss function.


Team: ITESMCEM    (Members: Javier de Velasco Oriol¹, Edgar Emmanuel Vallejo Clemente¹, Karol Estrada². Institution: ¹Instituto Tecnológico y de Estudios Superiores de Monterrey, ²Brandeis University)
Feature selection: Manual
Selected features: Demographics, MRI volumes, FDG PET and all cognitive tests
Missing data: Imputation using the mean of previous values of that patient, otherwise mean across all patients.
Confounder correction: None
Method category: Machine learning
Prediction method: ADAS-Cog13 is predicted with a Lasso model with $\alpha=0.1$. Ventricles are predicted using a Bayesian ridge regression. Clinical diagnosis is predicted using two different Random Forest models, one which selected between CN and either MCI or AD and the second which in turn predicted between MCI and AD for those selected as non-CN. The predictions for all target variables made use of a “transition model”, which predicted the next timepoint given the current one, until all 60 monthly predictions were made. The transition model was implemented using a total of 29 Lasso models.
Confidence Intervals: They were calculated by sampling the test samples multiple times, evaluating the performance of the model with those samples and analyzing the CIs supossing a Gaussian distribution and the corresponding t-distribution.


Team: lmaUCL    (Members: Leon Aksman, Institution: University College London, UK)
Feature selection: Manual
Selected features: Diagnosis, gender, education, APOE4 status and MMSE.
Missing data: Imputation using regression over ventricles and demographics (age, gender, education)
Confounder correction: None
Method category: Statistical regression and machine learning
Prediction method: For ADAS-Cog13 and Ventricles, a multi-task learning model was used with similar trajectories across subjects. The regression model was a linear model over age, but dependencies between different subjects were modelled through a special prior structure over the coefficients of the linear model. The prior structure has hyperparameters that control for the amount of coupling across subjects, and are optimised through empirical Bayes. Clinical diagnosis was predicted using the ADAS-Cog13 trajectory estimates plus a simple estimate of the mean and standard deviation of ADAS-Cog13 in each diagnostic group (AD/MCI/CN). Each ADAS-Cog13 prediction was then assigned a probability of belonging to each group. Three different submissions were made:

• •

  lmaUCL-Std: used only last available diagnosis as covariate

• •

  lmaUCL-Covariates: as above, but also used gender, education, APOE status and MMSE as covariates

• •

  lmaUCL-halfD1: trained only on half of the D1 dataset, but allowed for longer training time of 10 hours.

Confidence Intervals: Both the multi-task learning and clinical diagnosis models are probabilistic, providing estimates of predictive mean and standard deviation assuming a normal distribution. These were converted to confidence intervals using the inverse normal CDF.
Publication link: https://doi.org/10.1002/hbm.24682
Repository link: https://github.com/LeonAksman/bayes-mtl-traj

Team: Mayo-BAI-ASU    (Members: Cynthia M. Stonnington¹, Yalin Wang², Jianfeng Wu², Vivek Devadas³, Institution: ¹Mayo Clinic, Scottsdale, AZ, USA, ²School of Computing, Informatics and Decision Systems Engineering, Arizona State University, USA, ³Banner Alzheimer’s Institute, Phoenix, AZ, USA)
Feature selection: Manual, from clinical experience
Selected features: Age, PET(AV45, AV1451, FDG), hippocampal volume/ICV, ventricle volume/ICV, diagnosis, cognitive tests (ADAS11, ADAS-Cog13, MMSE, RAVLT, MOCA, Ecog), amyloid-beta, tau, phosphorylated tau, APOE status. All features except age were z-score normalised.
Missing data: Imputation with zero.
Confounder correction: None
Method category: Statistical regression
Prediction method: ADAS-Cog13 and Ventricles were forecasted using a linear mixed effects model, using all features as fixed effects and one random effect per subject (intercept). Training used all visits, but the forecasts only used the last visit. Clinical diagnosis was predicted with a similar model, by converting to CN/MCI/AD to a categorical variable (1/2/3).


Team: Orange    (Members: Clementine Fourrier, Institution: Institut du Cerveau et de la Moelle épinière, ICM, Paris, France)
Feature selection: Manual, through knowledge from literature
Selected features: demographics (age, education, gender), cognitive tests (ADAS11, CDSRB, MMSE), imaging(AV45 PET, FDG PET, hippocampus size, cortical thickness) and molecular markers (phosphorylated tau to amyloid-beta ratio, total tau, CMRgl, HCI)
Method category: Decision tree of a clinician
Prediction method: This method is based on the decision tree of a clinician. It looks at the latest available visit for a patient, and based on the value of the selected features, it predicts a duration to conversion. The duration to conversion depends on the initial clinical diagnosis and the other available data. Depending on the initial diagnosis, the algorithm assumes the patient will convert within a certain time period, and this time period is modulated by the available data about the patient. In that regard, the algorithm does not need to account for missing values. The ADAS-Cog13 and Ventricle measures at each month are computed assuming a linear evolution between the current time point and the conversion date.


Team: Rocket    (Members: Lars Lau Raket, Institutions: H. Lundbeck A/S, Denmark; Clinical Memory Research Unit, Department of Clinical Sciences Malmö, Lund University, Lund, Sweden )
Feature selection: Manual
Selected features: ADAS-Cog13, baseline diagnosis, and APOE4 carrier status.
Missing data: APOE4: Imputed using median number of alleles per baseline diagnostic group. Missing ADAS-Cog13 scores were imputed using multivariate imputation by chained equations based on age, sex, diagnosis and cognitive tests.
Confounder correction: None
Method category: Statistical regression and data-driven disease progression modelling
Prediction method: Prediction of ADAS-Cog13 is done through a latent-time non-linear mixed-effects model, where the trajectory is parameterised using an exponential function. The time shift is built from a fixed effect shift relative to time since baseline for different diagnostic groups (e.g. AD patients will be shifted to be later in the course of cognitive decline) and a random effect shift for each subject. The model also includes APOE status as a fixed effect that modifies rate of decline. This disease progression modeling methodology along with several extensions is presented in raket2019disease . Prediction of Ventricles/ICV uses a linear mixed-effects model using an integrated B-spline basis (5 knots + intercept) in predicted ADAS-Cog13 disease time. A random intercept is included per subject. Prediction of clinical diagnosis is based on kernel density estimation of the states (CN/MCI/AD) across the disease time from the ADAS-Cog13 model.
Confidence Intervals: Prediction intervals conditioned on the predicted disease time of the subject were derived based on the estimated variance-covariance matrix of the model. Because of the monotone nature of cognitive decline, and to heuristically compensate for the conditioning on disease time, the upper limit of the prediction interval was multiplied by 1.5.
Publication link: https://doi.org/10.1101/2019.12.13.19014860

Team: SBIA    (Members: Aristeidis Sotiras, Guray Erus, Jimit Doshi, Christos Davatzikos, Institution: Center for Biomedical Image Computing and Analytics, University of Pennsylvania)
Feature selection: Manual
Selected features: demographics, cognitive tests, diagnosis, MRI features (Freesurfer cross-sectional). Imaging indices (SPARE-AD and SPARE-MCI) that summarise brain atrophy patterns were estimated through support vector machines with linear kernels. Another index representing brain age (SPARE-BA) was estimated using a regressor model applied to imaging features.
Missing data: Features and time-points with missing data were not included
Confounder correction: Age, gender, APOE4, education, and SPARE scores were used as covariates in the linear mixed effects models. Also, the regression model was applied separately on different diagnosis groups.
Method category: Statistical regression and machine learning
Prediction method: A linear mixed effects model was used to forecast the SPARE indices for future timepoints. For diagnosis predictions, authors used class probability distribution estimations based on the forecasted SPARE-AD score. For the prediction of ADAS-Cog13 and Ventricles, linear mixed effects models were used, with age, gender and SPARE scores as covariates.


Team: SmallHeads – BigBrains (Members: Jacob Vogel, Andrew Doyle, Angela Tam, Alex Diaz-Papkovich, Institution: McGill University, Montreal, Canada)
Feature selection: Automatic
Selected features:

• •

  SmallHeads-NeuralNetwork   : All features in the TADPOLE spreadsheet were considered, as long as they had less than 50% missing data. This resulted in a final set of 376 features. Features were normalised to zero mean and unit variance.

• •

  SmallHeads-LinMixedEffects   : All features were normalised to zero mean and unit variance. A LASSO feature selection algorithm with ADAS-Cog13 and “Y” variable was used to select the best features, which were required to have a weight greater than 0.001. 10-fold cross-validation was used to estimate the best LASSO parameters.

Missing data: Imputed using 5-nearest neighbour method, using Euclidean distance (FancyImpute 0.0.4)
Confounder correction: None
Method category: Machine learning
Prediction method:

• •

  SmallHeads-NeuralNetwork: A Deep fully connected neural network was trained to predict the future diagnosis using the selected features and time until future timepoint as input. Network has 5 fully-connected layers with Leaky ReLU activations. Each layer has 512, 512, 1024, 1024 and 256 neurons, with softmax layer at output. Training used P(0.5) dropout using the Adam optimiser, based on a class-unweighted categorical cross-entropy loss function.

• •

  SmallHeads-LinMixedEffects: Only ADAS-Cog13 was predicted with a linear mixed effects model, using months since baseline as an interaction term. A random (subject — time) effect was also added, allowing variable subject-specific slopes over time.

Repository link: https://github.com/SmallHeads/tadpole


Team: SPMC-Plymouth    (Members: Emmanuel Jammeh, Institution: University of Plymouth, UK)
Feature selection: Automatic – Authors used the WEKA (https://www.cs.waikato.ac.nz/ml/weka/) machine learning tool.
Selected features: Age, gender, education, ApoE4, CDRSB, ADAS11, MMSE, RAVLT, Moca, Ecog, Hippocampus, WholeBrain, Entorhinal, MidTemp, FDG, AV45, PIB, ABETA, TAU, PTAU
Missing data:
Confounder correction: None
Method category: Machine learning
Prediction method: A machine learning classifier based on k-nearest neighbours, Naive Bayes, Random Forest and SVM was used to predict the clinical diagnosis. ADAS-Cog13 and Ventricle volume were not predicted. Two predictions were made, SPMC-Plymouth1 and SPMC-Plymouth2, but the authors could not be contacted to provide details on the differences between the two.

Team: Sunshine    (Members: Igor Koval, Stanley Durrleman, Institution: Institut du Cerveau et de la Moelle épinière, ICM, Paris, France)
Feature selection: Semi-automatic – An initial set of 60 features was selected by a clinical expert. Out of this set, the features that had more than 30% missing data were removed. A final subset of features was chosen based on cross-validation results using trial and error.
Selected features: Age, APOE status, MMSE, ADAS-Cog13, RAVLT immediate and CDRSOB
Missing data: Imputed using mean value
Confounder correction: None
Method category: Statistical regression and machine learning
Prediction method: A linear model was used to predict future values of ADAS-Cog13 and Ventricles, as well as other cognitive tests: MMSE, RAVLT and CDRSOB. For the prediction of clinical diagnosis, forecasted values of the previous five measures were used as input to an SVM. APOE4 status and education were also used as inputs to the SVM. Adding extra features did not seem to increase prediction scores based on cross-validation. Two submissions were made:

• •

  Sunshine-Conservative: CN and AD subjects were forecasted to have the same diagnosis (i.e. no conversion) for all future timepoints, after observing that a small proportion of them convert after 1 year.

• •

  Sunshine-Std: Without the above modification.
  
  

Team: Threedays    (Members: Paul Moore¹, Terry J. Lyons¹, John Gallacher², Institution: ¹Mathematical Institute, University of Oxford, ²Department of Psychiatry, University of Oxford, UK)
Feature selection: Manual
Selected features: Age, months since baseline, gender, race, marital status, diagnosis, cognitive tests (MMSE, CDRSB, ADAS11, ADAS-Cog13, RAVLT immediate, learning, forgetting and percent forgetting, FAQ) and APOE status.
Missing data: Random forest method deals with missing data automatically, by finding optimal splits with existing data only.
Confounder correction: None
Method category: Machine learning
Prediction method: For the prediction of clinical diagnosis, two random forest models are trained, the first for transitions from a healthy diagnosis, and the second for transitions from an MCI diagnosis. For AD individuals, authors assume that the diagnosis will not change. The training data was generated by ordering each participant’s data by time, then associating the feature vector x with diagnosis y for each time horizon available for the participant. ADAS-Cog13 and Ventricles were not predicted. The PLOS paper describes a method similar to the original, but using different predictors and a single random forest.
Publication link: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0211558


Team: Tohka-Ciszek (Members: Jussi Tohka, Robert Ciszek Institution: A.I. Virtanen Institute for Molecular Sciences, University of Eastern Finland, Finland)
Feature selection: Manual
Selected features:

• •

  D2: diagnosis, gender, education, race, marital and APOE status, age, cognitive tests (CDRSB, ADAS11, ADAS-Cog13, MMSE, RAVLT learning, immediate and perc. forgetting, FAQ, MOCA, all Ecog), MRI volumes (ventricles, hippocampus, whole brain, entorhinal, fusiform, middle temporal, ICV)

• •

  D3: diagnosis, age, gender, education, ethnicity, race, marital status, ADAS-Cog13, MMSE, MRI volumes as above

Missing data:

• •

  SMNSR   : A sub-model is trained for each subset of features which occurs without missing values. A specific catch-all subset is used for patients for which well performing whole measurement set cannot be found. For this data set, values are imputed using the median of k-nearest neighbours or replaced with -1.

• •

  RandomForestLin   : Imputation using mean values from the timepoints of the same subject (D2) or diagnostic category (for D3).

Confounder correction: None
Method category: Machine learning
Prediction method:

• •

  Tohka-Ciszek-SMNSR   : For ADAS-Cog13 prediction, a Sparse Multimodal Neighborhood Search Regression was used. This method first uses a linear regression model to estimate ADAS-Cog13 from the selected features belonging to the current subject and neighbour subjects, estimated based on a K-nearest neighbour algorithm. The forecasts from this model were passed to a gradient-boosted tree model, providing the final prediction. Ventricles and clinical diagnosis were not predicted.

• •

  Tohka-Ciszek-RandomForestLin   : To predict ADAS-Cog13 and Ventricles, a weighted average of two models was used: 1) a unimodal linear model 2) a linear model taking the response variable from the final time point and predictor variables from a time point before that. For diagnosis prediction, a random forest was trained using ADAS-Cog13, Ventricle/ICV, age and APOE status.

Confidence Intervals: Predicted score +/- cross-validation MAE.
Repository link: https://github.com/jussitohka/tadpole(RandomForestLin),https://github.com/rciszek/SMNSR (SMNSR)

Team: VikingAI    (Members: Bruno Jedynak¹, Kruti Pandya¹, Murat Bilgel², William Engels¹, Joseph Cole¹, Institutions: ¹Portland State University, USA, ²Laboratory of Behavioral Neuroscience, National Institute on Aging, National Institutes of Health, Baltimore, MD, USA)
Feature selection: Manual
Selected features: Diagnosis, age, ADAS-Cog13, CDRSOB, MMSE, RAVLT immediate, tau,
ventricles/ICV, hippocampus/ICV. Features were normalized prior to model fitting.
Missing data: Method automatically deals with missing data through Bayesian inference
Confounder correction: None
Method category: Data-driven disease progression model
Prediction method: For the prediction of ADAS-Cog13 and Ventricles, a latent-time parametric model was used, estimating a linear subject-specific model over the age of subjects, resulting in a disease progression score (DPS). The trajectories of biomarkers were assumed to be either sigmoidal functions or a sum of logistic basis functions over the DPS space. Some feature-specific parameters are estimated from the features’ histograms, while the rest are optimized. Priors over the parameters to be optimized are set a-priori. Two submissions were made:

• •

  VikingAI-Sigmoid: sigmoidal function as biomarker trajectory

• •

  VikingAI-Logistic: the sum of 15 logistic basis functions as the biomarker trajectory

Confidence Intervals: Bayesian predictive intervals

In addition to the above entries, the organisers also included several benchmark algorithms as well as some extra predictions: (1) two ensemble predictions averaging all the predictions submitted by the participants, and (2) 62 randomised predictions which can tell how likely it is that top submissions obtained their scores due to chance, by building a null distribution of the performance metrics. The
Source code of some benchmarks (BenchmarkLastVisit, BenchmarkMixedEffectsAPOE, BenchmarkSVM) was offered to participants before the conference deadline, as a starting point for making predictions.

Benchmark: BenchmarkLastVisit    (Authors: Daniel Alexander, Razvan Marinescu, Institutions: University College London, UK, Massachusetts Institute of Technology, USA)
Feature selection: None
Selected features: ADAS-Cog13, ventricle volume, diagnosis
Missing data: Not required
Confounder correction: None
Method category: Regression
Prediction method: For ADAS-Cog13 and Ventricles, the last available measure is used, otherwise the average for the current diagnostic group is used. Confidence intervals are set to default widths of 2 for ADAD13 and 0.001 for Ventricles/ICV. For prediction of clinical diagnosis, the last available diagnosis is used with probability 100%, and 0% probability for the other diagnoses.
Confidence Intervals: hard-coded
Repository link: https://github.com/noxtoby/TADPOLE/blob/master/evaluation

Benchmark: BenchmarkMixedEffects    (Author: Razvan Marinescu, Daniel Alexander, Institution: University College London, UK, Massachusetts Institute of Technology, USA)
Feature selection: None
Selected features: ADAS-Cog13, ventricle volume, diagnosis (, APOE status)
Missing data: Automatic, since model is univariate.
Confounder correction: APOE status was used as covariate in the linear mixed effects model
Method category: Regression
Prediction method: Linear Mixed Effects Model with age at visit as the predictor variable. Model was fitted independently for ADAS-Cog13 and Ventricles. Predictions for clinical diagnosis were derived from the corresponding ADAS-Cog13 forecasts, using three Gaussian likelihood models for CN, MCI and AD. The likelihoods for diagnostic classes were finally converted to probabilities by normalisation. Default values were used for confidence intervals. Two predictions were made:

• •

  BenchmarkMixedEffectsAPOE: the slope of the population trajectory was stratified by APOE status

• •

  BenchmarkMixedEffects: as above but without APOE

Confidence Intervals: hard-coded
Repository link: https://github.com/noxtoby/TADPOLE/blob/master/evaluation

Benchmark: BenchmarkSVM    (Author: Esther Bron, Institution: Erasmus MC)
Feature selection: Manual
Selected features: Diagnosis, age, ADAS-Cog13, Ventricles, ICV, APOE
Missing data: Fill-in using average value of biomarker from past visits of the same subject, otherwise population average.
Confounder correction: None
Method category: Machine Learning
Prediction method: For the prediction of clinical diagnosis, a probabilistic SVM was used based on the selected features, while for the prediction of ADAS-Cog13 and Ventricles, a Support Vector Regressor (SVR) was used. All SVM/SVRs used linear kernels. Default values were used for confidence intervals.
Confidence Intervals: hard coded
Repository link: https://github.com/noxtoby/TADPOLE/blob/master/evaluation

Benchmark: RandomisedBest    (Author: Razvan Marinescu, Institutions: University College London, UK, Massachusetts Institute of Technology, USA)
Feature selection: Manual
Selected features: Diagnosis, age, ADAS-Cog13, Ventricles, ICV
Missing data: Fill-in using last available measurement
Confounder correction: None
Method category: Regression
Prediction method: The method aims to construct a null distribution of values, to check how likely a high score could be obtained by chance alone. Starting from the simplest prediction method, i.e. BenchmarkLastVisit which simply takes the last available measure, 62 randomised predictions were created (as many as the total number of predictions in D2) by adding random perturbations to the predictions. For Diagnosis prediction, the probability of controls and MCI subjects to convert within 1 year was computed from ADNI historical data, and then each control and MCI was randomly chosen to convert with those probabilities. For ADAS-Cog13 and Ventricles, random uniform noise was added to the predictions as follows:

• •

  ADAS: new_adas $\sim$ last_available_adas + U(0, 7)

• •

  Ventricles: new_ventricles $\sim$ last_available_ventricles + U(0, 0.01)

The new values new_adas and new_ventricles, as well as the new diagnosis, were assigned to all 60 months, thus assuming no change across the 60 month predictions. All 62 different predictions were evaluated, and the RandomisedBest entry shows the best scores obtained by all 62 submissions in each category separately.
Confidence Intervals: hard-coded
Repository link: https://github.com/noxtoby/TADPOLE/blob/master/evaluation