Disease progression is defined as worsening of diagnosis on the last visit compared to the the baseline diagnosis.
RID | PTID | VISCODE | SITE | COLPROT | ORIGPROT | EXAMDATE | DX_bl | AGE | PTGENDER | PTEDUCAT | PTETHCAT | PTRACCAT | PTMARRY | APOE4 | FDG | PIB | AV45 | ABETA | TAU | PTAU | CDRSB | ADAS11 | ADAS13 | ADASQ4 | MMSE | RAVLT_immediate | RAVLT_learning | RAVLT_forgetting | RAVLT_perc_forgetting | LDELTOTAL | DIGITSCOR | TRABSCOR | FAQ | MOCA | EcogPtMem | EcogPtLang | EcogPtVisspat | EcogPtPlan | EcogPtOrgan | EcogPtDivatt | EcogPtTotal | EcogSPMem | EcogSPLang | EcogSPVisspat | EcogSPPlan | EcogSPOrgan | EcogSPDivatt | EcogSPTotal | FLDSTRENG | FSVERSION | IMAGEUID | Ventricles | Hippocampus | WholeBrain | Entorhinal | Fusiform | MidTemp | ICV | DX | mPACCdigit | mPACCtrailsB | EXAMDATE_bl | CDRSB_bl | ADAS11_bl | ADAS13_bl | ADASQ4_bl | MMSE_bl | RAVLT_immediate_bl | RAVLT_learning_bl | RAVLT_forgetting_bl | RAVLT_perc_forgetting_bl | LDELTOTAL_BL | DIGITSCOR_bl | TRABSCOR_bl | FAQ_bl | mPACCdigit_bl | mPACCtrailsB_bl | FLDSTRENG_bl | FSVERSION_bl | Ventricles_bl | Hippocampus_bl | WholeBrain_bl | Entorhinal_bl | Fusiform_bl | MidTemp_bl | ICV_bl | MOCA_bl | EcogPtMem_bl | EcogPtLang_bl | EcogPtVisspat_bl | EcogPtPlan_bl | EcogPtOrgan_bl | EcogPtDivatt_bl | EcogPtTotal_bl | EcogSPMem_bl | EcogSPLang_bl | EcogSPVisspat_bl | EcogSPPlan_bl | EcogSPOrgan_bl | EcogSPDivatt_bl | EcogSPTotal_bl | ABETA_bl | TAU_bl | PTAU_bl | FDG_bl | PIB_bl | AV45_bl | Years_bl | Month_bl | Month | M | update_stamp | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2 | 011_S_0002 | bl | 11 | ADNI1 | ADNI1 | 2005-09-08 | CN | 74.3 | Male | 16 | Not Hisp/Latino | White | Married | 0.0 | 1.36665 | NaN | NaN | NaN | NaN | NaN | 0.0 | 10.67 | 18.67 | 5.0 | 28.0 | 44.0 | 4.0 | 6.0 | 54.5455 | 10.0 | 34.0 | 112.0 | 0.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 35475.0 | 118233.0 | 8336.0 | 1229740.0 | 4177.0 | 16559.0 | 27936.0 | 1984660.0 | CN | -4.35795 | -4.15975 | 2005-09-08 | 0.0 | 10.67 | 18.67 | 5.0 | 28 | 44.0 | 4.0 | 6.0 | 54.5455 | 10.0 | 34.0 | 112.0 | 0.0 | -4.35795 | -4.15975 | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 118233.0 | 8336.0 | 1229740.0 | 4177.0 | 16559.0 | 27936.0 | 1984660.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.36665 | NaN | NaN | 0.000000 | 0.00000 | 0 | 0 | 2018-11-08 22:51:22.0 |
1 | 3 | 011_S_0003 | bl | 11 | ADNI1 | ADNI1 | 2005-09-12 | AD | 81.3 | Male | 18 | Not Hisp/Latino | White | Married | 1.0 | 1.08355 | NaN | NaN | 741.5 | 239.7 | 22.83 | 4.5 | 22.00 | 31.00 | 8.0 | 20.0 | 22.0 | 1.0 | 4.0 | 100.0000 | 2.0 | 25.0 | 148.0 | 10.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 32237.0 | 84599.0 | 5319.0 | 1129830.0 | 1791.0 | 15506.0 | 18422.0 | 1920690.0 | Dementia | -16.58450 | -16.16580 | 2005-09-12 | 4.5 | 22.00 | 31.00 | 8.0 | 20 | 22.0 | 1.0 | 4.0 | 100.0000 | 2.0 | 25.0 | 148.0 | 10.0 | -16.58450 | -16.16580 | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 84599.0 | 5319.0 | 1129830.0 | 1791.0 | 15506.0 | 18422.0 | 1920690.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 741.5 | 239.7 | 22.83 | 1.08355 | NaN | NaN | 0.000000 | 0.00000 | 0 | 0 | 2018-11-08 22:51:22.0 |
2 | 3 | 011_S_0003 | m06 | 11 | ADNI1 | ADNI1 | 2006-03-13 | AD | 81.3 | Male | 18 | Not Hisp/Latino | White | Married | 1.0 | 1.05803 | NaN | NaN | NaN | NaN | NaN | 6.0 | 19.00 | 30.00 | 10.0 | 24.0 | 19.0 | 2.0 | 6.0 | 100.0000 | NaN | 19.0 | 135.0 | 12.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 31863.0 | 88580.0 | 5446.0 | 1100060.0 | 2427.0 | 14400.0 | 16972.0 | 1906430.0 | Dementia | -15.02030 | -13.38660 | 2005-09-12 | 4.5 | 22.00 | 31.00 | 8.0 | 20 | 22.0 | 1.0 | 4.0 | 100.0000 | 2.0 | 25.0 | 148.0 | 10.0 | -16.58450 | -16.16580 | NaN | Cross-Sectional FreeSurfer (FreeSurfer Version... | 84599.0 | 5319.0 | 1129830.0 | 1791.0 | 15506.0 | 18422.0 | 1920690.0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 741.5 | 239.7 | 22.83 | 1.08355 | NaN | NaN | 0.498289 | 5.96721 | 6 | 6 | 2018-11-08 22:51:22.0 |
We are keeping only the last visit measurement since it contains all the measurements of the patient at the last visit plus baseline information.
Before we had 5059 rows.
After we had 819 rows.
When we stratify in groups based on the diagnosis of the first visit (left graph), we observe that:
When we stratify in groups based on the diagnosis of the last visit (right graph), we observe that:
As a second step we remove all the information of the last visit, except the time till the last visit and the diagnosis at the last visit.
We are splitting in a training set (80%) and test set (20%).
Left graph: Disease progression from normal Baseline to MCI/AD progression on last visit
The following confusion matrix: Modeling the disease progression from normal Baseline progressing to MCI/AD at last visit only based on time till last visit.
Simple logistic regression modeling with the CLASS WEIGHTS NON BALANCED
Logistic Regresssion predicting desease progression from Normal Baseline to MCI or AD at last visit.
Training accuracy: 0.80
Test accuracy: 0.63
We observe that the model is not predicting anything at all! The most likely case is for people to stay stabile and to show no disease progression at the last visit:
If we don’t balance for the unequal classes we end up in our very basic logistic regression model always “predicting” a person will stay stabile. Here we are still only using the ‘time till the last visit’ as our single variable.
The following confusion matrix: We get a better model when we balance the unequal size groups (by the parameter class_weight='balanced'
in our LogisticRegression()
object. Here we see we are now able to predict the people that are showing disease progression, yet of the cost of more false positives and with low discrimitive power.
Simple logistic regression modeling with the CLASS WEIGHTS BALANCED
Logistic Regresssion predicting desease progression from Normal Baseline to MCI or AD at last visit
Training accuracy: 0.70
Test accuracy: 0.70
The balancing of the unequal size groups by the parameter class_weight='balanced'
did increase our test accuracy from 0.63 to 0.70.
The following confusion matrix: Modelling non-balanced, MCI Baseline, progressing to AD at last visit only based on time till last visit.
Simple logistic regression modeling with the CLASS WEIGHTS NON BALANCED
Logistic Regresssion predicting desease progression from MCI Baseline to AD at last visit
Training accuracy: 0.58
Test accuracy: 0.49
As expected from the first barchart in 2.1.: For MCI to AD disease progression we saw no real difference in distribution of people with disease progression and people without disease when we looked at the variable of when the last visit occurs. The time till last visit will be far less important or non-predictive in our predictive model to model change from MCI baseline to AD at last visit. Here we observe a test accuracy of 0.49, so indeed a very bad performance.
The following confusion matrix: Modelling balanced, MCI Baseline, progressing to AD only based on time till last visit, yet now again with balanced class.
Simple logistic regression modeling with the CLASS WEIGHTS BALANCED
Logistic Regresssion predicting desease progression from MCI Baseline to AD at last visit
Training accuracy: 0.61
Test accuracy: 0.54
We can plot the logistic regression model by looking at the probabilit of disease progression versus the time of our last visit. Our non-balanced model was not good, yet in our balanced model we observed a big improvement of our normal baseline to MCI/AD progression model. Here we can observe this also by an increased slope for the balanced model for predicting normal baseline to MCI/AD progression model (right graph, blue line).
Simple logistic regression modeling with the CLASS WEIGHTS BALANCED
Logistic Regresssion predicting desease progression from Normal Baseline to MCI or AD at last visit
Training accuracy: 0.75
Test accuracy: 0.63
Most important features are the ADAS (Alzheimers Disease Assessment Scale) results,
patient education and FAQ_bl (Functinoal Activities Questionnaire score at baseline).
Simple logistic regression modeling with the CLASS WEIGHTS BALANCED
Logistic Regresssion predicting desease progression from Normal Baseline to MCI or AD at last visit
Training accuracy: 0.76 , Test accuracy: 0.63
Most important features:
1. M : Months after baseline
2. CDRSB_bl : Clinical Dementia Rating–Sum of Boxes at baseline.
3. FAQ_bl : Functinoal Activities Questionnaire score at baseline
Cross-validation setting:
How do we asses the performance of our models, what is our scoring function?
The performance of machine-learning models, was quantified by 4-fold cross-validation as mentioned before. Each of 4 folds produced a machine learning model trained on 75 percent of the dataset and was validated on the remaining 25 percent of the dataset. This resulted in a final independent validation prediction for every single subject of the cohort based on a independent training set of 75 percent of the dataset not including the specific subject itself.
Two performance measurements were used specifically that allowed us to evaluate the predictions in a threshold independent manner. This is important since this is how we can avoid the problem of having to set a threshold or to balance our initial groups to prevent our model from always predicting the same outcome for all (the outcome of the largest group of our training set).
roc_auc_score()
.average_precision_score()
. Average Precision-Recall summarizes the precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight [1]. The APR is approximately AUROC multiplied by the initial precision of the system [2]. For each model AUROC and APR was reported as well as the absolute difference with baseline risk prediction (American ACC/AHA ASCDV or European SCORE risk estimation).[1] Zhu M. Recall, precision and average precision. Working Paper 2004-09, Department of Statistics & Actuarial Science University of Waterloo 2004; :1–11.
[2] Su W, Yuan Y, Zhu M. A Relationship between the Average Precision and the Area Under the ROC Curve. Proceedings of the 2015 International Conference on The Theory of Information Retrieval 2015; :349–52. doi: 10.1145/2808194.2809481
Illustrated example of Parameter Optimization:
Model: LogisticRegression(random_state = 1, solver = 'liblinear', class_weight = 'balanced')
Tested Estimator_parameters à 'C': [0.1, 100]
`
Estimator_parameters = {'C':[0.1,1,10,100] }
Estimator=LogisticRegression(random_state=1, solver='liblinear', class_weight='balanced')
Here below we see a print of all the 4 Cross validations, the accuracy scores with also the best set of paremeters in this run. We see that with every CV run we add a value of the outer validation accuaracy to the last list that is printed.
All 4 PARAMETER OPTIMISATIONS AND CROSS VALIDATIONS OF THE "LogisticRegression()" ALGORITHM
This is optimisation and CV number 1 ----------------------------------------------------------------------------
Mean scores: [0.729 0.6981 0.6993 0.6894]
Differences best - other scores: [0. 0.0309 0.0297 0.0396]
The best paremeters set was Set 1: {'C': 0.1}
Validation AUROC score: [0.7]
This is optimisation and CV number 2 ----------------------------------------------------------------------------
Mean scores: [0.6746 0.6453 0.6724 0.6418]
Differences best - other scores: [0. 0.0293 0.0022 0.0328]
The best paremeters set was Set 1: {'C': 0.1}
Validation AUROC score: [0.7 0.8]
This is optimisation and CV number 3 ----------------------------------------------------------------------------
Mean scores: [0.7271 0.7117 0.7052 0.7052]
Differences best - other scores: [0. 0.0154 0.0218 0.0219]
The best paremeters set was Set 1: {'C': 0.1}
Validation AUROC score: [0.7 0.8 0.56]
This is optimisation and CV number 4 ----------------------------------------------------------------------------
Mean scores: [0.6941 0.6677 0.6755 0.6781]
Differences best - other scores: [0. 0.0264 0.0186 0.016 ]
The best paremeters set was Set 1: {'C': 0.1}
Validation AUROC score: [0.7 0.8 0.56 0.8 ]
Elapsed time: 0.83 seconds.
We can summarize the performance of every parameter set by averaging all 4 validation scores. Looking at which set gave us the highest average score indicates the best parameter setting.
Parameter tested: {'C': [0.1, 1, 10, 100]}
Average of 4 splits mean_test_score for all 4 parameter sets:
[0.705 0.6825 0.69 0.68 ]
Differences best Average - Average for all 4 parameter sets:
[0. 0.0225 0.015 0.025 ]
Best Average mean_test_score = 0.7 for the 1th parameter: {'C': 0.1}
Index of the Best Parameter Set for all 4 splits: [1, 1, 1, 1]
Occurences of all 4 parameters as Best Parameter Set in a split:
(Counter(index: times_the_best))) Counter({1: 4})
After the for runs of cross validation and parameter optimization we see that our second parameter $C=0.1$ is performing the best. It has the best mean test score from all the tested values for $C:[0.1,1,10,100]$ that were tested.
Estimator : All 4 AUROC Scores of validation on the outer test set with
the estimator with the optimised parameters trained on the inner test set:
[0.70292208 0.8041958 0.56118881 0.7972028 ]
Avg (+- STD )
-----------------
AUROC: 0.716 (+- 0.098)
Accuracy: 0.677 (+- 0.097)
Logistic_mu | Logistic_std | |
---|---|---|
Average Precision | 0.43 | 0.10 |
AUROC | 0.72 | 0.10 |
Precision | 0.39 | 0.11 |
Recall | 0.62 | 0.17 |
F1_Score | 0.48 | 0.13 |
Sensitivity | 0.62 | 0.17 |
Specificity | 0.69 | 0.09 |
Accuracy | 0.68 | 0.10 |
def CV_and_Parameter_Optimization(X_CV, y_CV, splits, Estimator, Estimator_parameters):
splits=4
start = time.time()
# All 10 parameter optimisations and cross validations of a certain Classifier
strat_kfold_external = StratifiedKFold(n_splits=splits, shuffle=True, random_state=1)
splitted_indexes_external = strat_kfold_external.split(X_CV, y_CV)
params= [len(Estimator_parameters[param]) for param in Estimator_parameters]
total_params= np.prod(params)
best_index=[]
AP_Score =np.array([])
ROC_Score =np.array([])
Prec_Score =np.array([])
Recall_Score =np.array([])
F1_Score =np.array([])
Sens_Score =np.array([])
Spec_Score =np.array([])
Accuracy_Score =np.array([])
best_predictions_all=np.array([])
test_all=np.array([])
i=1
Sum=[0]*total_params
for train_index_external, test_index_external in splitted_indexes_external:
print("Parameter optimisation and CV number {} ... \t AUROC scores: {}".format(i, ROC_Score .round(3)))
i=i+1
# take i’th fold as test set and the other 3 folds together as training set
X_train, X_test = X_CV.iloc[train_index_external], X_CV.iloc[test_index_external]
y_train, y_test = y_CV.iloc[train_index_external], y_CV.iloc[test_index_external]
# take the training set and split it randomly (stratified)
# in an inner training set of 2/3 size and an inner validation set of 1/3 size
# for each of the ML methods
# for each combination of parameter values
# train the ML method with specified parameters on inner training set and validate on inner validation set
# take the combination of parameter values that yields highest performance (e.g. area under PR curve),
# also write down this parameter value combination train the ML method with best parameter values on
# the (outer) training set and test on the (outer) test set
mod=clone(Estimator)
optimisation_param_AP = GridSearchCV(estimator=mod, param_grid=Estimator_parameters, cv=3, verbose=0, scoring='roc_auc')
optimisation_param_AP.fit(X_train, y_train)
best_index.append(optimisation_param_AP.best_index_+1)
# Building Estimator with best parameters
best= optimisation_param_AP.best_estimator_
best.fit(X_train, y_train)
best_predictions = best.predict_proba(X_test)[:,1]
best_predictions=best_predictions
Average_precision_score=average_precision_score(y_test, best_predictions)
best_predictions_binary = best.predict(X_test)
t=confusion_matrix(y_test, best_predictions_binary)
sensitivity=(t[1][1]/(t[1][0]+t[1][1]))
specificity=(t[0][0]/(t[0][1]+t[0][0]))
AP_Score =np.append(AP_Score , Average_precision_score)
ROC_Score =np.append(ROC_Score , roc_auc_score(y_test, best_predictions))
Prec_Score =np.append(Prec_Score , precision_score(y_test, best_predictions_binary))
Recall_Score =np.append(Recall_Score , recall_score(y_test, best_predictions_binary))
F1_Score =np.append(F1_Score , f1_score(y_test, best_predictions_binary))
Sens_Score =np.append(Sens_Score , sensitivity)
Spec_Score =np.append(Spec_Score , specificity)
Accuracy_Score =np.append(Accuracy_Score , accuracy_score(y_test, best_predictions_binary))
best_predictions_all=np.append(best_predictions_all,best_predictions)
test_all=np.append(test_all,y_test)
Sum=Sum+(optimisation_param_AP.cv_results_['mean_test_score']).round(2)
print('Done! Validation AUROC score: \t\t\t{}'.format(ROC_Score .round(3)))
end = time.time()
total_time=round(end - start,1)
print('Elapsed time: ', total_time,'seconds.' )
print("\nParameter tested: ", Estimator_parameters)
Avg=Sum/splits
#print('Average of {} splits AUROC for all {} parameter sets:\n{}'.format(splits, total_params, Avg.round(4)))
print('Best Average AUROC = {} for the {}th parameter set: {}'.format(round(Avg.max(),4), Avg.argmax()+1, optimisation_param_AP.cv_results_['params'][Avg.argmax()]))
return AP_Score, ROC_Score, Prec_Score, Recall_Score, F1_Score, Sens_Score, Spec_Score, Accuracy_Score, best_predictions_all, test_all, total_time
And its output:
Estimator_parameters = {'C':[0.1, 0.5, 1, 10, 20] }
Estimator=LogisticRegression(random_state=1, solver='liblinear', class_weight='balanced')
LOGI_AP, LOGI_ROC, LOGI_Prec, LOGI_Recall, LOGI_F1, LOGI_Sens, LOGI_Spec, \
LOGI_Accuracy, LOGI_best_predictions_all, LOGI_test_all , LOGI_time \
= CV_and_Parameter_Optimization(X_CV, y_CV, splits, Estimator, Estimator_parameters)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.703]
Parameter optimisation and CV number 3 ... AUROC scores: [0.703 0.804]
Parameter optimisation and CV number 4 ... AUROC scores: [0.703 0.804 0.561]
Done! Validation AUROC score: [0.703 0.804 0.561 0.797]
Elapsed time: 0.8 seconds.
Parameter tested: {'C': [0.1, 0.5, 1, 10, 20]}
Best Average AUROC = 0.705 for the 1th parameter set: {'C': 0.1}
Logistic_mean | Logistic_std | |
---|---|---|
Average Precision | 0.43 | 0.10 |
AUROC | 0.72 | 0.10 |
Precision | 0.39 | 0.11 |
Recall | 0.62 | 0.17 |
F1_Score | 0.48 | 0.13 |
Sensitivity | 0.62 | 0.17 |
Specificity | 0.69 | 0.09 |
Accuracy | 0.68 | 0.10 |
Time | 0.80 | 0.00 |
3.3.1 KNN: class sklearn.neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto',
leaf_size=30, p=2, metric='minkowski', metric_params=None, n_jobs=1, **kwargs)
`
Tested parameters:
KNN_parameters = {'p':[1,2],
'n_neighbors':[1, 3, 5, 9]}
KNN = KNeighborsClassifier(weights='distance')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.492]
Parameter optimisation and CV number 3 ... AUROC scores: [0.492 0.469]
Parameter optimisation and CV number 4 ... AUROC scores: [0.492 0.469 0.509]
Done! Validation AUROC score: [0.492 0.469 0.509 0.42 ]
Elapsed time: 0.6 seconds.
Parameter tested: {'p': [1, 2], 'n_neighbors': [1, 3, 5, 9]}
Best Average AUROC = 0.5725 for the 8th parameter set: {'n_neighbors': 9, 'p': 2}
3.3.2 Logistic Regression: class sklearn.linear_model.LogisticRegression(penalty=’l2’, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver=’liblinear’, max_iter=100, multi_class=’ovr’, verbose=0, warm_start=False, n_jobs=1)
Tested parameters:
LOGI_parameters = {'C':[0.001,0.01,0.1,1,10,100,1000] }
LOGI =LogisticRegression(random_state=1,max_iter=1000,tol=5e-4, solver='liblinear', class_weight='balanced')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.68]
Parameter optimisation and CV number 3 ... AUROC scores: [0.68 0.78]
Parameter optimisation and CV number 4 ... AUROC scores: [0.68 0.78 0.589]
Done! Validation AUROC score: [0.68 0.78 0.589 0.762]
Elapsed time: 0.5 seconds.
Parameter tested: {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]}
Best Average AUROC = 0.7175 for the 4th parameter set: {'C': 1}
3.3.3 SVM:class sklearn.svm.SVC(C=1.0, kernel=’rbf’, degree=3, gamma=’auto’, coef0=0.0, shrinking=True, probability=False, tol=0.001, cache_size=200, class_weight=None, verbose=False, max_iter=-1, decision_function_shape=’ovr’, random_state=None)
Tested parameters:
SVC_parameters = {'C':[0.00001, 0.0001, 0.001,0.1,1,10,100],
'gamma':[0.0001, 0.001,0.1,1,10,100]}
svc=SVC(probability=True,random_state=1,class_weight='balanced', cache_size=20000,kernel='rbf')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.661]
Parameter optimisation and CV number 3 ... AUROC scores: [0.661 0.841]
Parameter optimisation and CV number 4 ... AUROC scores: [0.661 0.841 0.605]
Done! Validation AUROC score: [0.661 0.841 0.605 0.802]
Elapsed time: 5.1 seconds.
Parameter tested: {'C': [1e-05, 0.0001, 0.001, 0.1, 1, 10, 100], 'gamma': [0.0001, 0.001, 0.1, 1, 10, 100]}
Best Average AUROC = 0.7425 for the 38th parameter set: {'C': 100, 'gamma': 0.001}
d['3 SVC_mu'] = [SVC_AP.mean(), SVC_ROC.mean(), SVC_Prec.mean(), SVC_Recall.mean(), SVC_F1.mean(), SVC_Sens.mean(), SVC_Spec.mean(), SVC_Accuracy.mean(), SVC_time]
d['3 SVC_std'] = [SVC_AP.std(), SVC_ROC.std(), SVC_Prec.std(), SVC_Recall.std(), SVC_F1.std(), SVC_Sens.std(), SVC_Spec.std(), SVC_Accuracy.std(), 0]
df = pd.DataFrame(data=d, index=Names)
#df[['1 KNN_mu', '2 LOGI_mu', '3 SVC_mu']].round(2)
3.3.4 Decision Tree: class sklearn.tree.DecisionTreeClassifier(criterion=’gini’, splitter=’best’, max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, class_weight=None, presort=False)
Tested parameters:
TREE_parameters ={ 'max_depth':range(2,15,2) }
TREE = DecisionTreeClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.58]
Parameter optimisation and CV number 3 ... AUROC scores: [0.58 0.601]
Parameter optimisation and CV number 4 ... AUROC scores: [0.58 0.601 0.524]
Done! Validation AUROC score: [0.58 0.601 0.524 0.51 ]
Elapsed time: 0.5 seconds.
Parameter tested: {'max_depth': range(2, 15, 2)}
Best Average AUROC = 0.5675 for the 1th parameter set: {'max_depth': 2}
3.3.5 Random Forest: class sklearn.ensemble.RandomForestClassifier(n_estimators=10->100, criterion='gini',
max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
max_features='auto', max_leaf_nodes=None, min_impurity_split=1e-07, bootstrap=True,
oob_score=False, n_jobs=1, random_state=None, verbose=0, warm_start=False, class_weight=None)
Tested parameters:
RF_parameters = {'n_estimators':[50, 100, 500],
'max_features':('sqrt', 'log2'),
'min_samples_leaf':[1, 5]}
RF = RandomForestClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.557]
Parameter optimisation and CV number 3 ... AUROC scores: [0.557 0.645]
Parameter optimisation and CV number 4 ... AUROC scores: [0.557 0.645 0.561]
Done! Validation AUROC score: [0.557 0.645 0.561 0.603]
Elapsed time: 25.4 seconds.
Parameter tested: {'n_estimators': [50, 100, 500], 'max_features': ('sqrt', 'log2'), 'min_samples_leaf': [1, 5]}
Best Average AUROC = 0.625 for the 7th parameter set: {'max_features': 'log2', 'min_samples_leaf': 1, 'n_estimators': 50}
3.3.6 Gradient boosting: class sklearn.ensemble.GradientBoostingClassifier(loss='deviance', learning_rate=0.1, n_estimators=100, subsample=1.0, criterion='friedman_mse', min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
max_depth=3, min_impurity_split=1e-07, init=None, random_state=None, max_features=None, verbose=0, max_leaf_nodes=None, warm_start=False, presort='auto')
Tested parameters:
BOOST_parameters = {'n_estimators':[50,100,500],
'max_leaf_nodes':[2,4,8,16],
'learning_rate':[1, 0.5, 0.1, 0.05, 0.01]}
BOOST=GradientBoostingClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.62]
Parameter optimisation and CV number 3 ... AUROC scores: [0.62 0.691]
Parameter optimisation and CV number 4 ... AUROC scores: [0.62 0.691 0.565]
Done! Validation AUROC score: [0.62 0.691 0.565 0.702]
Elapsed time: 74.3 seconds.
Parameter tested: {'n_estimators': [50, 100, 500], 'max_leaf_nodes': [2, 4, 8, 16], 'learning_rate': [1, 0.5, 0.1, 0.05, 0.01]}
Best Average AUROC = 0.665 for the 26th parameter set: {'learning_rate': 0.1, 'max_leaf_nodes': 2, 'n_estimators': 100}
3.3.7. XGBoost: class xgboost.XGBClassifier(max_depth=3, learning_rate=0.1, n_estimators=100, silent=True, objective='binary:logistic', booster='gbtree', n_jobs=1, nthread=None, gamma=0, min_child_weight=1, max_delta_step=0, subsample=1, colsample_bytree=1, colsample_bylevel=1, reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5, random_state=0, seed=None, missing=None, **kwargs)
Tested parameters:
XGB2_parameters ={ 'max_depth':range(3,10,2),
'min_child_weight':range(1,3,2),
'gamma':[0, 0.1,0.2],
'subsample':[i/10.0 for i in range(6,9,2)],
'colsample_bytree':[i/10.0 for i in range(6,9,2)],
'reg_alpha':[0, 0.005, 0.01, 0.05]}
XGB2 = XGBClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.537]
Parameter optimisation and CV number 3 ... AUROC scores: [0.537 0.766]
Parameter optimisation and CV number 4 ... AUROC scores: [0.537 0.766 0.584]
Done! Validation AUROC score: [0.537 0.766 0.584 0.584]
Elapsed time: 84.1 seconds.
Parameter tested: {'max_depth': range(3, 10, 2), 'min_child_weight': range(1, 3, 2), 'gamma': [0, 0.1, 0.2], 'subsample': [0.6, 0.8], 'colsample_bytree': [0.6, 0.8], 'reg_alpha': [0, 0.005, 0.01, 0.05]}
Best Average AUROC = 0.6425 for the 183th parameter set: {'colsample_bytree': 0.8, 'gamma': 0.2, 'max_depth': 7, 'min_child_weight': 1, 'reg_alpha': 0.05, 'subsample': 0.6}
3.3.8 MLP: class sklearn.neural_network.MLPClassifier(hidden_layer_sizes=(100, ), activation='relu', solver='adam', alpha=0.0001, batch_size='auto', learning_rate='constant', learning_rate_init=0.001, power_t=0.5, max_iter=200, shuffle=True, random_state=None, tol=0.0001, verbose=False, warm_start=False, momentum=0.9, nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08)
Tested parameters:
MLP_parameters = {'activation':('relu', 'logistic'),
'hidden_layer_sizes':[(3, ),(20, ),(20,20 ),(50, ),(50,50 ),(100, )],
'alpha':[0.0001,0.001,0.09,0.1,0.5]}
MLP = MLPClassifier(random_state=1, max_iter=500,solver='adam')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.674]
Parameter optimisation and CV number 3 ... AUROC scores: [0.674 0.561]
Parameter optimisation and CV number 4 ... AUROC scores: [0.674 0.561 0.582]
Done! Validation AUROC score: [0.674 0.561 0.582 0.778]
Elapsed time: 142.2 seconds.
Parameter tested: {'activation': ('relu', 'logistic'), 'hidden_layer_sizes': [(3,), (20,), (20, 20), (50,), (50, 50), (100,)], 'alpha': [0.0001, 0.001, 0.09, 0.1, 0.5]}
Best Average AUROC = 0.73 for the 60th parameter set: {'activation': 'logistic', 'alpha': 0.5, 'hidden_layer_sizes': (100,)}
3.3.9 LGBM: class lightgbm.Dataset(data, label=None, reference=None, weight=None, group=None, init_score=None, silent=False, feature_name='auto', categorical_feature='auto', params=None, free_raw_data=True)
Test parameters:
LGBM_params = {'max_depth' : [2,3,4,5,6,7,8,9,10, 30],
'n_estimators': [5, 25, 50]}
lgbm=LGBMClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.653]
Parameter optimisation and CV number 3 ... AUROC scores: [0.653 0.73 ]
Parameter optimisation and CV number 4 ... AUROC scores: [0.653 0.73 0.545]
Done! Validation AUROC score: [0.653 0.73 0.545 0.632]
Elapsed time: 14.7 seconds.
Parameter tested: {'max_depth': [2, 3, 4, 5, 6, 7, 8, 9, 10, 30], 'n_estimators': [5, 25, 50]}
Best Average AUROC = 0.6475 for the 5th parameter set: {'max_depth': 3, 'n_estimators': 25}
1 KNN_mu | 2 LOGI_mu | 3 SVC_mu | 4 TREE_mu | 5 RF_mu | 6 BOOST_mu | 7 XGB2_mu | 8 MLP_mu | 9 LGBM_mu | |
---|---|---|---|---|---|---|---|---|---|
Average Precision | 0.24 | 0.43 | 0.47 | 0.27 | 0.35 | 0.35 | 0.36 | 0.39 | 0.38 |
AUROC | 0.47 | 0.70 | 0.73 | 0.55 | 0.59 | 0.64 | 0.62 | 0.65 | 0.64 |
Precision | 0.03 | 0.36 | 0.44 | 0.29 | 0.08 | 0.11 | 0.40 | 0.36 | 0.14 |
Recall | 0.02 | 0.64 | 0.61 | 0.26 | 0.02 | 0.05 | 0.09 | 0.27 | 0.07 |
F1_Score | 0.02 | 0.46 | 0.50 | 0.26 | 0.03 | 0.07 | 0.13 | 0.29 | 0.10 |
Sensitivity | 0.02 | 0.64 | 0.61 | 0.26 | 0.02 | 0.05 | 0.09 | 0.27 | 0.07 |
Specificity | 0.94 | 0.66 | 0.75 | 0.77 | 0.97 | 0.93 | 0.91 | 0.90 | 0.94 |
Accuracy | 0.72 | 0.65 | 0.72 | 0.65 | 0.75 | 0.73 | 0.72 | 0.76 | 0.74 |
Time | 0.60 | 0.50 | 5.10 | 0.50 | 25.40 | 74.30 | 84.10 | 142.20 | 14.70 |
Plot of the AUROC curve for all the classifiers for predicting disease progression from Normal baseline to MCI/AD at last visit
Plot of the Precision-Recall curve for all the classifiers for predicting disease progression from Normal baseline to MCI/AD at last visit
Barchart for all the classifiers for predicting disease progression from Normal baseline to MCI/AD at last visit
3.4.1 KNN
KNN_parameters = {'p':[1,2], 'n_neighbors':[1, 3, 5, 9]}
KNN = KNeighborsClassifier(weights='distance')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.679]
Parameter optimisation and CV number 3 ... AUROC scores: [0.679 0.745]
Parameter optimisation and CV number 4 ... AUROC scores: [0.679 0.745 0.859]
Done! Validation AUROC score: [0.679 0.745 0.859 0.737]
Elapsed time: 0.8 seconds.
Parameter tested: {'p': [1, 2], 'n_neighbors': [1, 3, 5, 9]}
Best Average AUROC = 0.7275 for the 8th parameter set: {'n_neighbors': 9, 'p': 2}
3.4.2 Logistic Regression
LOGI_parameters = {'C':[0.001,0.01,0.1,1,10,100,1000] }
LOGI =LogisticRegression(random_state=1,max_iter=1000,tol=5e-4, solver='liblinear', class_weight='balanced')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.785]
Parameter optimisation and CV number 3 ... AUROC scores: [0.785 0.772]
Parameter optimisation and CV number 4 ... AUROC scores: [0.785 0.772 0.82 ]
Done! Validation AUROC score: [0.785 0.772 0.82 0.772]
Elapsed time: 0.5 seconds.
Parameter tested: {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]}
Best Average AUROC = 0.775 for the 4th parameter set: {'C': 1}
3.4.3 SVM
SVC_parameters = {'C':[0.1,1,10],
'gamma':[0.01,0.1,1] }
svc=SVC(probability=True,random_state=1,class_weight='balanced', cache_size=20000,kernel='rbf')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.766]
Parameter optimisation and CV number 3 ... AUROC scores: [0.766 0.759]
Parameter optimisation and CV number 4 ... AUROC scores: [0.766 0.759 0.83 ]
Done! Validation AUROC score: [0.766 0.759 0.83 0.758]
Elapsed time: 2.3 seconds.
Parameter tested: {'C': [0.1, 1, 10], 'gamma': [0.01, 0.1, 1]}
Best Average AUROC = 0.7725 for the 5th parameter set: {'C': 1, 'gamma': 0.1}
d['3 SVC_mu'] = [SVC_AP.mean(), SVC_ROC.mean(), SVC_Prec.mean(), SVC_Recall.mean(), SVC_F1.mean(), SVC_Sens.mean(), SVC_Spec.mean(), SVC_Accuracy.mean(), SVC_time]
d['3 SVC_std'] = [SVC_AP.std(), SVC_ROC.std(), SVC_Prec.std(), SVC_Recall.std(), SVC_F1.std(), SVC_Sens.std(), SVC_Spec.std(), SVC_Accuracy.std(), 0]
df = pd.DataFrame(data=d, index=Names)
#df[['1 KNN_mu', '2 LOGI_mu', '3 SVC_mu']].round(2)
3.4.4 Decision Tree
TREE_parameters ={ 'max_depth':range(2,15,2) }
TREE = DecisionTreeClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.689]
Parameter optimisation and CV number 3 ... AUROC scores: [0.689 0.719]
Parameter optimisation and CV number 4 ... AUROC scores: [0.689 0.719 0.639]
Done! Validation AUROC score: [0.689 0.719 0.639 0.663]
Elapsed time: 0.5 seconds.
Parameter tested: {'max_depth': range(2, 15, 2)}
Best Average AUROC = 0.645 for the 1th parameter set: {'max_depth': 2}
3.4.5 Random Forest
RF_parameters = {'n_estimators':[50, 100, 500],
'max_features':('sqrt', 'log2'),
'min_samples_leaf':[1, 5]}
RF = RandomForestClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.812]
Parameter optimisation and CV number 3 ... AUROC scores: [0.812 0.778]
Parameter optimisation and CV number 4 ... AUROC scores: [0.812 0.778 0.832]
Done! Validation AUROC score: [0.812 0.778 0.832 0.769]
Elapsed time: 30.5 seconds.
Parameter tested: {'n_estimators': [50, 100, 500], 'max_features': ('sqrt', 'log2'), 'min_samples_leaf': [1, 5]}
Best Average AUROC = 0.78 for the 3th parameter set: {'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 500}
3.4.6 Gradient boosting
BOOST_parameters = {'n_estimators':[50,100,500],
'max_leaf_nodes':[2,4,8,16],
'learning_rate':[1, 0.5, 0.1, 0.05, 0.01]}
BOOST=GradientBoostingClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.807]
Parameter optimisation and CV number 3 ... AUROC scores: [0.807 0.754]
Parameter optimisation and CV number 4 ... AUROC scores: [0.807 0.754 0.855]
Done! Validation AUROC score: [0.807 0.754 0.855 0.782]
Elapsed time: 96.0 seconds.
Parameter tested: {'n_estimators': [50, 100, 500], 'max_leaf_nodes': [2, 4, 8, 16], 'learning_rate': [1, 0.5, 0.1, 0.05, 0.01]}
Best Average AUROC = 0.7825 for the 13th parameter set: {'learning_rate': 0.5, 'max_leaf_nodes': 2, 'n_estimators': 50}
3.4.7. XGBoost
XGB2_parameters ={ 'max_depth':range(3,10,2),
'min_child_weight':range(1,3,2),
'gamma':[0, 0.1,0.2],
'subsample':[i/10.0 for i in range(6,9,2)],
'colsample_bytree':[i/10.0 for i in range(6,9,2)],
'reg_alpha':[0, 0.005, 0.01, 0.05]}
XGB2 = XGBClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.815]
Parameter optimisation and CV number 3 ... AUROC scores: [0.815 0.758]
Parameter optimisation and CV number 4 ... AUROC scores: [0.815 0.758 0.852]
Done! Validation AUROC score: [0.815 0.758 0.852 0.802]
Elapsed time: 126.9 seconds.
Parameter tested: {'max_depth': range(3, 10, 2), 'min_child_weight': range(1, 3, 2), 'gamma': [0, 0.1, 0.2], 'subsample': [0.6, 0.8], 'colsample_bytree': [0.6, 0.8], 'reg_alpha': [0, 0.005, 0.01, 0.05]}
Best Average AUROC = 0.7925 for the 135th parameter set: {'colsample_bytree': 0.8, 'gamma': 0.1, 'max_depth': 3, 'min_child_weight': 1, 'reg_alpha': 0.05, 'subsample': 0.6}
3.4.8 MLP
MLP_parameters = {'activation':('relu', 'logistic'),
'hidden_layer_sizes':[(3, ),(20, ),(20,20 ),(50, ),(50,50 ),(100, )],
'alpha':[0.0001,0.001,0.09,0.1,0.5]}
MLP = MLPClassifier(random_state=1, max_iter=500,solver='adam')
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.774]
Parameter optimisation and CV number 3 ... AUROC scores: [0.774 0.752]
Parameter optimisation and CV number 4 ... AUROC scores: [0.774 0.752 0.84 ]
Done! Validation AUROC score: [0.774 0.752 0.84 0.776]
Elapsed time: 136.9 seconds.
Parameter tested: {'activation': ('relu', 'logistic'), 'hidden_layer_sizes': [(3,), (20,), (20, 20), (50,), (50, 50), (100,)], 'alpha': [0.0001, 0.001, 0.09, 0.1, 0.5]}
Best Average AUROC = 0.77 for the 12th parameter set: {'activation': 'relu', 'alpha': 0.001, 'hidden_layer_sizes': (100,)}
3.4.9 LGBM
LGBM_params = { 'max_depth' : [2,3,4,5,6,7,8,9,10, 30],
'n_estimators': [5, 25, 50]}
lgbm=LGBMClassifier(random_state=1)
Parameter optimisation and CV number 1 ... AUROC scores: []
Parameter optimisation and CV number 2 ... AUROC scores: [0.821]
Parameter optimisation and CV number 3 ... AUROC scores: [0.821 0.771]
Parameter optimisation and CV number 4 ... AUROC scores: [0.821 0.771 0.851]
Done! Validation AUROC score: [0.821 0.771 0.851 0.797]
Elapsed time: 19.8 seconds.
Parameter tested: {'max_depth': [2, 3, 4, 5, 6, 7, 8, 9, 10, 30], 'n_estimators': [5, 25, 50]}
Best Average AUROC = 0.785 for the 3th parameter set: {'max_depth': 2, 'n_estimators': 50}
1 KNN_mu | 2 LOGI_mu | 3 SVC_mu | 4 TREE_mu | 5 RF_mu | 6 BOOST_mu | 7 XGB2_mu | 8 MLP_mu | 9 LGBM_mu | |
---|---|---|---|---|---|---|---|---|---|
Average Precision | 0.69 | 0.77 | 0.75 | 0.60 | 0.78 | 0.78 | 0.80 | 0.77 | 0.81 |
AUROC | 0.75 | 0.79 | 0.78 | 0.68 | 0.80 | 0.80 | 0.81 | 0.79 | 0.81 |
Precision | 0.65 | 0.68 | 0.67 | 0.62 | 0.70 | 0.71 | 0.72 | 0.70 | 0.71 |
Recall | 0.75 | 0.72 | 0.72 | 0.59 | 0.65 | 0.69 | 0.68 | 0.69 | 0.71 |
F1_Score | 0.69 | 0.70 | 0.69 | 0.59 | 0.67 | 0.70 | 0.70 | 0.69 | 0.71 |
Sensitivity | 0.75 | 0.72 | 0.72 | 0.59 | 0.65 | 0.69 | 0.68 | 0.69 | 0.71 |
Specificity | 0.64 | 0.70 | 0.70 | 0.69 | 0.75 | 0.75 | 0.76 | 0.75 | 0.75 |
Accuracy | 0.69 | 0.71 | 0.71 | 0.64 | 0.70 | 0.72 | 0.73 | 0.72 | 0.73 |
Time | 0.80 | 0.50 | 2.30 | 0.50 | 30.50 | 96.00 | 126.90 | 136.90 | 19.80 |
Plot of the AUROC curve for all the classifiers for predicting disease progression from MCI baseline to AD at last visit
Plot of the Precision-Recall curve for all the classifiers for predicting disease progression from MCI baseline to AD at last visit
Barchart for all the classifiers for predicting disease progression from MCI baseline to AD at last visit
Two performance measurements were used specifically that allowed us to evaluate the predictive power of our models in a threshold-independent manner.
Conclusion for the models predicing progression from Normal at Baseline to Mild Cognitive Impairement or Alzheimer at the last visit:
1 KNN_mu | 2 LOGI_mu | 3 SVC_mu | 4 TREE_mu | 5 RF_mu | 6 BOOST_mu | 7 XGB2_mu | 8 MLP_mu | 9 LGBM_mu | |
---|---|---|---|---|---|---|---|---|---|
Average Precision | 0.24 | 0.43 | 0.47 | 0.27 | 0.35 | 0.35 | 0.36 | 0.39 | 0.38 |
AUROC | 0.47 | 0.70 | 0.73 | 0.55 | 0.59 | 0.64 | 0.62 | 0.65 | 0.64 |
Precision | 0.03 | 0.36 | 0.44 | 0.29 | 0.08 | 0.11 | 0.40 | 0.36 | 0.14 |
Recall | 0.02 | 0.64 | 0.61 | 0.26 | 0.02 | 0.05 | 0.09 | 0.27 | 0.07 |
F1_Score | 0.02 | 0.46 | 0.50 | 0.26 | 0.03 | 0.07 | 0.13 | 0.29 | 0.10 |
Sensitivity | 0.02 | 0.64 | 0.61 | 0.26 | 0.02 | 0.05 | 0.09 | 0.27 | 0.07 |
Specificity | 0.94 | 0.66 | 0.75 | 0.77 | 0.97 | 0.93 | 0.91 | 0.90 | 0.94 |
Accuracy | 0.72 | 0.65 | 0.72 | 0.65 | 0.75 | 0.73 | 0.72 | 0.76 | 0.74 |
Time | 0.60 | 0.50 | 5.10 | 0.50 | 25.40 | 74.30 | 84.10 | 142.20 | 14.70 |
Conclusion for the models predicting progression from Mild Cognitive Impairement at baseline to Alzheimer at last visit: LGBM and XGBoost have the highest AUROC and Average Precision Recall score, followed by random forest and gradient boosting and logistic regression.
The AUROC in our models for patients evolving from normal to MCI/AD are lower compared to the AUROC for the models predicting which patient will evolve from MCI to AD. This might indicate it is harder to predict who will develop MCI or Alzheimer from normal baseline measurements. Yet once a patient suffers from Mild Cognitive Impairment already, our model can more easily predict if this patient will progress to Alzheimers Disease or not at the last visit. This observation might be explained by the multiple heterogeneous causes that may result in mild cognitive impaired, for example just the process of normal aging in general results in some degree of the loss of memory when people age. Further conclusion are hard to make concrete since we only have 400 individual patients to train our models on.
1 KNN_mu | 2 LOGI_mu | 3 SVC_mu | 4 TREE_mu | 5 RF_mu | 6 BOOST_mu | 7 XGB2_mu | 8 MLP_mu | 9 LGBM_mu | |
---|---|---|---|---|---|---|---|---|---|
Average Precision | 0.69 | 0.77 | 0.75 | 0.60 | 0.78 | 0.78 | 0.80 | 0.77 | 0.81 |
AUROC | 0.75 | 0.79 | 0.78 | 0.68 | 0.80 | 0.80 | 0.81 | 0.79 | 0.81 |
Precision | 0.65 | 0.68 | 0.67 | 0.62 | 0.70 | 0.71 | 0.72 | 0.70 | 0.71 |
Recall | 0.75 | 0.72 | 0.72 | 0.59 | 0.65 | 0.69 | 0.68 | 0.69 | 0.71 |
F1_Score | 0.69 | 0.70 | 0.69 | 0.59 | 0.67 | 0.70 | 0.70 | 0.69 | 0.71 |
Sensitivity | 0.75 | 0.72 | 0.72 | 0.59 | 0.65 | 0.69 | 0.68 | 0.69 | 0.71 |
Specificity | 0.64 | 0.70 | 0.70 | 0.69 | 0.75 | 0.75 | 0.76 | 0.75 | 0.75 |
Accuracy | 0.69 | 0.71 | 0.71 | 0.64 | 0.70 | 0.72 | 0.73 | 0.72 | 0.73 |
Time | 0.80 | 0.50 | 2.30 | 0.50 | 30.50 | 96.00 | 126.90 | 136.90 | 19.80 |