Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 https://doi.org/10.1186/s12911-022-02054-7 RESEARCH Open Access Adopting transfer learning for neuroimaging: a comparative analysis with a custom 3D convolution neural network model Amira Soliman1* , Jose R. Chang1,2, Kobra Etminani1, Stefan Byttner1, Anette Davidsson3, Begoña Martínez‑Sanchis4, Valle Camacho5, Matteo Bauckneht6, Roxana Stegeran7, Marcus Ressner8, Marc Agudelo‑Cifuentes4, Andrea Chincarini9, Matthias Brendel10, Axel Rominger11, Rose Bruffaerts26, Rik Vandenberghe12,13, Milica G. Kramberger14, Maja Trost14,15, Nicolas Nicastro16, Giovanni B. Frisoni17, Afina W. Lemstra18, Bart N. M. van Berckel19, Andrea Pilotto20, Alessandro Padovani21, Silvia Morbelli22, Dag Aarsland23,27, Flavio Nobili23, Valentina Garibotto24, the Alzheimer’s Disease Neuroimaging Initiative and Miguel Ochoa‑Figueroa3,7,25 From The 17th International Conference on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB 2021) Virtual. 15-17 November 2021. https://davidechicco.github.io/cibb2021/ Abstract Background: In recent years, neuroimaging with deep learning (DL) algorithms have made remarkable advances in the diagnosis of neurodegenerative disorders. However, applying DL in different medical domains is usually chal‑ lenged by lack of labeled data. To address this challenge, transfer learning (TL) has been applied to use state-of-the- art convolution neural networks pre-trained on natural images. Yet, there are differences in characteristics between medical and natural images, also image classification and targeted medical diagnosis tasks. The purpose of this study is to investigate the performance of specialized and TL in the classification of neurodegenerative disorders using 3D volumes of 18F-FDG-PET brain scans. Results: Results show that TL models are suboptimal for classification of neurodegenerative disorders, especially when the objective is to separate more than two disorders. Additionally, specialized CNN model provides better inter‑ pretations of predicted diagnosis. Conclusions: TL can indeed lead to superior performance on binary classification in timely and data efficient man‑ ner, yet for detecting more than a single disorder, TL models do not perform well. Additionally, custom 3D model *Correspondence: amira.soliman@hh.se 1 Center for Applied Intelligent Systems Research (CAISR), Halmstad University, Halmstad, Sweden Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​ mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 2 of 15 performs comparably to TL models for binary classification, and interestingly perform better for diagnosis of multiple disorders. The results confirm the superiority of the custom 3D-CNN in providing better explainable model compared to TL adopted ones. Keywords: Convolution Neural Networks, Transfer Learning, Brain Neurodegenerative Disorders, Medical Image Classification Background different categories [12, 13]. The application of these Neurodegenerative disorders have a huge negative impact pre-trained models to neuroimaging studies is an active on the healthcare systems globally. Alzheimer’s Disease research field for their expectations on improving classi- (AD) is highly prevalent in the elder population, is con- fication performance. Specifically, TL models have shown sidered to be the most common disorder with approxi- to be timely efficient in classifying AD dementia patients mately 60% of all dementia [1]. Dementia with Lewy and achieving a good performance [7, 14, 15]. bodies (DLB) is second most common neurodegenerative Despite the popularity of TL models in medical imag- disorder, increasing prevalence estimates were reported ing, there has been little work studying its precise effects with increasing age. DLB accounted for from 0.3 to 24.4% in the medical imaging settings. Particularly, medi- of all cases of dementia in the prevalence studies [2]. Mild cal tasks often have significantly fewer classes than the cognitive impairment (MCI) is a transition stage between standard ImageNet classification. Additionally, TL normal aging and dementia and 10-15% of patients diag- models are pre-trained with 2D images and thus spa- nosed with MCI progress every year to dementia, most tial information is lost during the transformation of the commonly AD [3]. 3D medical images into the 2D space. Furthermore, the Different neuroimaging techniques such as magnetic nature of features used as representation generated for resonance imaging (MRI), positron emission tomography 3D neuroimaging volumes using pre-trained TL models (PET) and single-photon emission computed tomogra- need further investigation especially with the high reso- phy (SPECT) are proficient to document the functional lution non-RGB nature of input data and the low contrast and anatomical abnormalities informative to diagnose between brain regions and background of scans after the type of the neurodegeneration [4]. F-18 fluorodeoxy- data normalization. glucose positron emission tomography (18F-FDG-PET) Given these open issues with lack of scientific studies scans which measures cerebral glucose metabolism, has highlighting the differences among adopting TL mod- been reported as an accurate biomarker for the discrimi- els and training specialized models for medical imaging, nation of the above-mentioned neurodegenerative disor- in this paper the aim is to compare the performance of ders [5]. deep and transfer learning applied to neuroimaging vol- Different deep convolutional neural network (CNN) umes for classification purposes. The goal is to study the techniques have proven to be effective in supporting the performance of these techniques with respect to multi- diagnosis of most common types of dementia such as ple aspects. The first dimension in our empirical analysis AD, MCI, and DLB using 18F-FDG-PET brain images. represents the type of classification task, such as binary These techniques show ability to extract features and classification when distinguishing AD from Cognitively identify disease-related patterns in imaging input data Normal (CN) cases, 3-way classification by adding DLB without prior-knowledge about the pathophysiological cases, and finally 4-way classification by considering MCI mechanisms of the underlying diseases [6–9]. However, cases. Additionally, we compare performance of these one of the main challenges with analyzing medical imag- models in terms of training overhead, obtained accuracy, ing is that data is limited and expensive to collect. There- explainability, and sensitivity towards other similar types fore, transfer learning (TL) becomes a key component of of dementia. many successful models used for medical diagnosis [7, In this paper, we use InceptionV3 [16], VGG16 9]. The core of TL is to use the knowledge of pre-trained [17], and ResNet50 [18] pre-trained TL models using models on a source dataset, and fine-tune it for a target ImageNet. Furthermore, we develop a 3D CNN that task on a different but related dataset [10, 11]. adopts VGG16 in its architecture, yet with less number There are many CNNs well-trained on ImageNet with of convolution blocks [19]. We analyze the classifica- differing accuracy. These networks have been trained tion performance of TL models for detecting multiple to recognize objects from a huge natural-image data- Neurodegenerative disorders. Additionally, we discuss set which consists of 14 million images of roughly 1,000 the challenges in developing a custom 3D model using Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 3 of 15 Results 3D Models TL Models 100 We start by discussing the dataset and the choice of 90 85 86 hyperparameters for the developed 3D models. Then, Validation accuracy 80 78.3 78.3 71.5 the results of performance comparison among different 70 60 59.7 models are presented. Lastly, we detail our experiments 50 on network attention for explaining the decisions taken 40 30 by different models. 20 All of the experiments were conducted using Ten- 10 sorflow and Keras libraries on a computer with Linux 0 Binary 3 Way 4 Way Ubuntu 18.09 operating system, that has one Nvidia Type of classification Quadro GV100 GPU card with 32GB of memory and 36 Fig. 1 The average validation accuracy obtained for different CPU core Xenon with 128 GB of memory. classification tasks using transfer learning (TL) models and custom 3D models. The error bars show minimum and maximum values obtained during training of different models. Detailed comparison Neuroimaging dataset can be found at Table 4 Our primary dataset was collected retrospectively from two different sources as detailed in Table 1. The anonymized scans from patients with probable DLB were collected from the European DLB (EDLB) Consortium [20] limited data, specifically choosing the CNN archi- having the local institutional ethics committee approv- tecture and addressing overfitting that leads to poor als including the transfer of anonymized imaging brain model performance. Figure 1 highlights one of our 18F-FDG-PET scans. Recruited patients were referred main perspectives regarding the comparison between to and assessed at outpatient clinics including memory, adopting TL models and training a custom 3D model movement disorders, geriatric medicine, psychiatric, and from scratch. As shown, both TL models and custom neurology clinics as previously described in [21]. The diag- models achieve almost similar performance in binary nosis of probable DLB was originally made according to classification (AD versus CN), however, differences diagnostic criteria for probable DLB as defined by [22]. in performances appear when the objective changes The EDLB also provided some normal cases that we to diagnose more disorders. The contributions of this added to the CN cases. In total EDLB provided 201 study can be listed as the following: scans from 2005 to 2018. The rest of the images, i.e. 556 scans, were collected from the Alzheimer’s Disease 1 We evaluate the performance of TL models trained Neuroimaging Initiative (ADNI) [23] across ADNI-1, with natural images such as ImageNet, as well as cus- ADNI-2, ADNI-3 and ADNI-GO (Grand Opportunities) tom models for diagnosis of brain disorders using 3D studies from December 2005 to March 2020 [24]. We also medical imaging data for different classification tasks. used eight Frontotemporal lobar degeneration (FTLD) 2 We investigate whether using pre-trained TL models lead to different learned representations, by visual- izing the generated feature representation by dif- Table 1 The demographics of the used dataset collected from ferent models. More importantly, we analyze net- ADNI and EDLB sites, showing the average age in men and work attention to demonstrate the significant areas women per each clinical diagnosis (class), and percentage of of interest indicated by the adopted and developed samples used for training, validation and testing models. Furthermore, we evaluate robustness and Cases (%) Average age sensitivity of different models towards other similar dementia types. Class Cases women Men Women Source: ADNI The rest of the paper is organized as follows: in Sec- AD 200 72 (36.0%) 76.7 ± 8.2 74.0 ± 7.8 tion Results we present obtained performance from TL MCI 200 76 (38.0%) 75.6 ± 7.5 73.2 ± 8.2 and 3D custom model in different experiments, then CN 156 62 (39.7%) 77.5 ± 5.4 78.3 ± 5 we provide discussion and related work, lastly we con- Source: EDLB clude our paper. Section Methods presents the design DLB 157 59 (37.5%) 73.3 ± 7.2 74.8 ± 6.4 of comparative analysis performed between the differ- CN 44 22 (50.0%) 70.1 ± 10.3 67.5 ± 9.2 ent models. Total 757 291 (38.4%) 75.5 ± 7.6 74.4 ± 7.7 Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 4 of 15 18F-FDG-PET scans that were downloaded from the performing the classification tasks using TL and custom Frontotemporal Lobar Degeneration Neuroimaging Ini- 3D models. tiative (FTLDNI) database. Table 1 summarises the demographics of the data col- lected for this study from EDLB and ADNI. The dataset 3D CNN model specification consisted of 757 cases including 200 AD (from ADNI), VGG16 was designed for 2D images with small and fixed 200 MCI (from ADNI), 157 DLB (from EDLB), and 200 filters across all the convolution layers (i.e. filters of size CN (156 cases from ADNI and 44 cases from EDLB). We 3 × 3). In order to add depth (i.e. 3D), one choice could split the data into two sets with 89% and 11% for train- be keeping a homogeneous filter of size 3 × 3 × 3 across ing and testing using stratified random sampling to keep all convolution layers. However, we wanted to investigate enough cases from all four disorders while considering the performance using different depth values, e.g., hav- the two sources (in specific CN which contains cases ing a filter 3 × 3 × 6. Also, as we are handling 3D data, both from ADNI and EDLB). we wanted to study the effect of treating each slice from the input 18F-FDG-PET scan separately in the first con- Classification algorithms volution layer. Therefore, we developed four models, each We used InceptionV3, VGG16,and ResNet50 models to with a different structure in terms of kernel shape across be evaluated as transfer learning approach which being the convolution layers. We performed the experiments trained with ImageNet. Furthermore, we trained a 3D with end-to-end training using mini-batches of size 6 VGG model from scratch using our 18F-FDG-PET scans. and Adadelta optimizer with 0.01 learning rate for 50 Table 2 lists the details of used models with respect to epochs. Additionally, to prevent the model from overfit- number of trainable parameters. In the following sub- ting we used early stopping condition by monitoring the sections, we describe the different pipelines adopted for validation loss in order to end the model training when the model performance stops improving (i.e., less than 0.0001 change in validation loss for 10 epochs). Table 2 Specification of used models in terms of number of We evaluated the different models through 10 rounds trainable parameters, size of generated features and reported of K-Fold Cross Validation (KFCV) on the training set, accuracy on ImageNet for TL models also computed the 95% confidence intervals, the results Model Parameters Features Accuracy on are shown in Table 3. As results indicate, there is no huge ImageNet difference among obtained accuracy using different ker- [25] nels. So, we choose to keep a homogeneous 3x3x3 filter InceptionV3 23,851,784 2,048 94.49% as it achieves the highest training and validation accu- VGG16 138,357,544 512 91.9% racy for the different folds during the cross validation ResNet50 25,636,712 18 x 25 x 2048 92.9% evaluation. 3D Model 62,997,012 1,024 - Table 3 Performance of 3D model alternatives designed with different values for depth in convolution kernels. L(1): Conv(3x3x1) followed by L(a): Conv(3x3x3) L(1): Conv(3x3x1) followed by L(a): Conv(3x3x6) L(r):Conv(3x3x3) L(r): Conv(3x3x6) CV T acc. V acc. T acc. V acc. T acc. V acc. T acc. V acc. 2-F 0.76 ±0.04 0.58±0.12 0.77 ±0.03 0.59 ±0.14 0.69 ±0.04 0.42 ±0.12 0.7 ±0.03 0.54 ±0.14 3-F 0.83 ±0.04 0.7 ±0.02 0.83 ±0.05 0.71 ±0.03 0.73 ±0.02 0.68 ±0.02 0.73 ±0.02 0.66 ±0.02 4-F 0.79 ±0.02 0.7 ±0.02 0.8 ±0.02 0.7 ±0.02 0.76 ±0.04 0.66 ±0.02 0.75 ±0.04 0.66 ±0.03 5-F 0.8 ±0.04 0.71 ±0.02 0.82 ±0.03 0.72 ±0.02 0.78 ±0.04 0.7 ±0.02 0.75 ±0.03 0.68 ±0.02 6-F 0.82 ±0.03 0.72 ±0.02 0.83 ±0.04 0.72 ±0.02 0.77 ±0.03 0.68 ±0.02 0.76 ±0.03 0.68 ±0.02 7-F 0.81 ±0.03 0.72 ±0.02 0.87 ±0.03 0.72 ±0.02 0.77 ±0.04 0.7 ±0.02 0.77 ±0.03 0.69 ±0.02 8-F 0.84 ±0.03 0.71 ±0.02 0.86 ±0.03 0.72 ±0.02 0.81 ±0.04 0.7 ±0.02 0.76 ±0.03 0.70 ±0.02 9-F 0.86 ±0.03 0.72 ±0.02 0.87 ±0.03 0.72 ±0.02 0.82 ±0.04 0.69 ±0.02 0.82 ±0.04 0.71 ±0.02 10-F 0.86 ±0.03 0.72 ±0.02 0.87 ±0.03 0.72 ±0.02 0.83 ±0.03 0.70 ±0.02 0.82 ±0.03 0.71 ±0.01 L(1) represents the first convolution layer, L(r) represents the remaining convolution layers, L(a) represents all of the convolution layers in the developed 3D model. The values show the obtained accuracy followed by 95% confidence interval, while T and V represent training and validation accuracy, respectively. Bold represents the best value achieved Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 5 of 15 Further optimizations When training a large network, there is a point during training when the model will stop generalizing and start overfitting of the training data. Overfitting results in increasing the generalization error, making the model less useful at making predictions on new data. The challenge is to train the network long enough to make it capable of learning the mapping between inputs and outputs, but not so long to avoid overfitting training data. Particularly, the number of training epochs as a hyperparameter and train the model with an early stopping condition com- Fig. 3 Validation accuracy obtained using 3D custom model with bined with different learning rates and dropout rations. different values of learning rate (lr) and dropout. According to We performed two different experiments to show the obtained validation accuracy, there is no huge difference while effect of different learning strategies (i.e., fixed epoch changing lr and dropout training vs early stopping), learning rates (e.g., testing dif- ferent values such as 0.1, 0.01, and 0.001), and dropout (e.g., with values of 0.2, 0.3, 0.4 and 0.5). In the second experiment, we trained 3D-CNN model dropout to 0.5 as the highest validation accuracy was for 4-way classification tasks for 10 rounds of K-Fold obtained using these values. Cross Validation (CV) on the training set. Also, we com- puted the 95% confidence intervals. Figure 2 illustrates Performance evaluation on multi‑classification tasks the obtained results, as shown training strategy with early Most DL models applied in neurodegenerative diseases stopping allows the model to avoid overfitting of training mainly focus on binary classification or classify mul- data and provide better generalization for validation data. tiple stages of AD from no dementia to moderate AD. We use value of 0.0001 as a minimum change in vali- However, the utility of such models is limited to the AD dation loss, and monitor the change for 10 epochs. On patient population solely, which makes them unable to the other hand, without early stopping in which we run discriminate non-AD patterns from AD, also it becomes model training for 100 epochs, the gap between train- hard to validate their robustness in the presence of non- ing and validation accuracy is huge. Thus, for the further AD disorders. The proper diagnosis of dementia patients experiments, we train our 3D custom model using the requires going beyond binary classification and at least early stopping condition. recognizing the differences among Cognitively Normal For the third experiment, we trained the model with (CN), MCI and other dementia types, especially the most early stopping training strategy with different values of common ones such as AD and DLB. Therefore, in the learning rate (lr) and dropout rations as shown in Fig. 3. following experiment we evaluate the accuracy of used According to obtained results, we set lr to 0.01 and models using different classification tasks. Fig. 2 Confidence Intervals (95% CI) for training and validation of the 3D model during KFCV with different stopping criteria. Early stopping condition helps to avoid overfitting training data Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 6 of 15 In this experiment we evaluated the performance of dif- predicted labels generated by different models. Basi- ferent models for different classification tasks. We started cally, mutual information index is a non-negative quan- with binary classification, where the objective is to rec- tity and is upper bounded by the values of entropy of ognize AD from CN cases. Then, for 3-way classification identified classes that can measure the information that we added the DLB cases, and lastly 4-way classification predicted and ground-truth labels share. The highest is performed to distinguish among the four classes exist- value of NMI can be reached when the predicted labels ing in our dataset. In order to perform this experiment, are exactly the same as the data labels. we randomly divided the training data into 80% used for As shown in Table 4, TL models achieve good perfor- training and 20% used for validation. mance for binary classification. ResNet50 has the high- For 3D model, we trained our model from scratch using est validation accuracy. Also, 3D model performed very mini-batches of size 6 and Adadelta optimizer with 0.01 well, it is the second best in terms of validation accu- learning rate for 50 epochs and Dropout layers with 0.5 racy. It is also worth noticing that fine-tuning ResNet50 rate. Additionally, to prevent the model from overfitting for our data took more time than training a special- we used early stopping condition as discussed in Experi- ized 3D model from scratch with a simpler network ment 2. structure. For TL models, we trained a separate classifier for each For classification tasks that consider multiple disor- classification task using the features of cases belonging ders at the same time, our 3D model performs better to training set, then computed the validation accuracy than TL models, due to its ability to extract relevant with features representing validation set. We repeated features from input data that make separation bound- this process for 10 rounds and obtained results are listed ary among different classes more evident as explained in Table 4. We reported the average time in seconds and in the results of the next experiment. This also can be we also report the standard deviation of the time taken shown with achieving highest validation accuracy for to train classifier for these 10 training rounds. We report 3-way and 4-way classification. Additionally, the val- the average value for training and validation accuracy, ues of NMI obtained by our model are the closest to also we computed 95% confidence intervals. the values computed using ground-truth labels with Additionally, we used normalized mutual information a prominent difference from values reached by TL (NMI) index by comparing ground-truth labels with the models. Table 4 Performance of different models (TL and 3D) across the multiple classification tasks (binary, 3-way, 4-way). Binary classification (AD vs CN) Model Time (SD) Training acc. (± CI) Validation acc. (±CI) NMI InceptionV3 2.6 (0.5) 100 76 ± 0.016 0.94 VGG16 20.7 (0.3) 90 ± 0.006 70 ± 0.044 0.94 ResNet50 635 (0.2) 100 89 ± 0.012 0.97 3D model 108 (0.4) 97 ± 0.012 86 ± 0.042 0.96 3-Way classification (AD vs CN vs DLB) Model Time (SD) Training acc. (± CI) Validation acc. (±CI) NMI InceptionV3 28.2 (0.7) 99.8 ± 0.0006 78 ± 0.019 0.61 VGG16 2.5 (0.2) 89 ± 0.005 74 ± 0.025 0.6 ResNet50 877.3 (0.7) 100 83 ± 0.044 0.79 3D model 161 (0.3) 96 ± 0.01 87 ±0.01 0.9 4-Way classification (AD vs CN vs DLB vs MCI) Model Time (SD) Training acc. (±CI) Validation acc. (± CI) NMI InceptionV3 3.6 (0.6) 97 ± 0.006 59 ± 0.01 0.57 VGG16 3.2 (0.1) 69 ± 0.006 54 ± 0.019 0.56 ResNet50 1193 (0.8) 100 66 ± 0.008 0.62 3D model 296 (0.5) 85 ± 0.026 73 ± 0.015 0.82 Acc stands for accuracy, SD indicated standard deviation, CI is used for confidence interval, and NMI represents Normalized Mutual Information index. Bold represents the best value achieved Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 7 of 15 Analysis of 4‑way classification Visualizing extracted feature The difference in performance between specialized 3D We used the unsupervised UMAP to visualize 1) the orig- model and TL models appears in classifying multiple inal normalized data and 2) the extracted features by dif- brain disorders. Therefore, in this section we analyze fur- ferent models (before the classification layer) as shown in ther the features and classification decisions obtained by Fig. 4. The unsupervised UMAP is used to qualitatively different models when the objective is to perform 4-way evaluate the generated representation by each DL model, classification. specifically, we run UMAP to generate 2D representation 15 CN AD 14 MCI DLB UMAP-D2 13 12 11 10 2 3 4 5 6 7 8 9 UMAP-D1 (a) Original data 7 8 6 6 5 UMAP-D2 UMAP-D2 4 4 3 2 2 0 1 0 −2 −1 2 4 6 8 10 12 14 16 2 4 6 8 10 UMAP-D1 UMAP-D1 (b) InceptionV3 (c) VGG16 7 9 6 8 7 5 UMAP-D2 UMAP-D2 6 4 5 3 4 2 3 2 1 0 2 4 6 8 10 0 2 4 6 8 10 12 UMAP-D1 UMAP-D1 (d) ResNet50 (e) 3D model Fig. 4 UMAP visualizations for the original data (top), after being passed through the pre-trained models, InceptionV3 b, VGG16 c , and ResNet50 d, finally with 3D custom model e. The UMAP embeddings were created using the unsupervised version of UMAP. All subplots maintain the same color scheme for the target classes, i.e., yellow, purple, blue, and green identify CN, AD, MCI, and DLB, respectively using the ground-truth labels Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 8 of 15 of extracted features without specifying any number of with 86% AUC, then both InceptionV3 and VGG16 classes. have 85% AUC. It is also interesting to see that all mod- For TL models, some of DLB cases were separable in els achieve comparable performance in identifying DLB the representation space, however, rest of cases belong- cases in the test set. However, TL models have the lowest ing to CN, AD, and MCI are overlapping that makes it performance in identifying MCI cases. difficult to reach a decision boundary to separate them while training a classifier. On the other hand, features Explainability of model classification extracted by 3D model made DLB cases very well sepa- Occlusion experiment is used to visualize network rated and it is explaining the good performance of the towards a specific class. We applied occlusion experi- model. The other interesting pattern in Fig. 4 (e) is the ment for ResNet50, InceptionV3 and compare it with distribution of cases from CN to MCI and then to AD, results obtained by occlusion on a trained 3D model as which is as happening in reality: people with CN brains explained in Sect. . The results show the cross entropy either will develop DLB or they will develop MCI and response of the network given such occluded data as a then AD (if they get dementia, of course). function of the position of the occlusion box. The experi- Figure 5 shows receiver operating characteristic (ROC) ments were done for all four classes in the training data- curves obtained for test set from the different models. set, i.e. when calculating the maps for the DLB class, only As shown, specialized 3D models achieves the highest DLB subjects were included. The assumption is that when macro-average with 92% AUC, followed by ResNet50 ignoring a relevant region for the correct classification, Fig. 5 ROC-Curves for 4-way classification for specialized 3D and TL models. 3D model achieves the highest macro-average AUC and performes the best for MCI cases compared to TL models. 3D model has the highest macro-average AUC for all classes, and highest per class types except for DLB Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 9 of 15 the cross entropy response will be high.The cross entropy The results are illustrated in Fig. 6. As shown, the maps are then projected using a mosaic of the slices 5 to highlighted regions in each disorder indicate which 54 (to create a 7 × 7 grid) on the axial direction and over brain regions were of more attention from the models in layered with the average brain. The occlusion heatmaps their predictions. Looking at the results, it is clear that visualize metabolism patterns within each class, specifi- the models responded differently to the occluded areas. cally we show the average cross entropy using all samples However, InceptionV3 pays a lot of attention to the back- belonging to each class separately. ground and it is difficult to define discriminative regions Fig. 6 Results of the occlusion experiments for adopted models. The results are projected by creating a mosaic of slices in the axial direction. The cross entropy maps have been over layered with the average brain Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 10 of 15 that can be linked to each class. On the other hand, Table 5 Predicted labels for FTLD cases using different models ResNet50 shows attention to regions belonging to brain ID 3D Model ResNet50 InceptionV3 VGG16 area and these regions differ among classes, however the spatial structure to these regions is lost across the simi- FTLD 1 AD AD CN AD lar consecutive brain slices, e.g., regions highlighted for FTLD2 AD AD AD CN different classes from third till fourth row. The 3D spatial FTLD3 AD AD CN CN structure of brain is better maintained with the 3D model FTLD 4 CN AD CN CN showing discriminative regions that are defined for each FTLD 5 AD DLB CN MCI class type. Thus, we provide clinical explanations for FTLD 6 CN AD CN DLB highlighted regions with custom 3D model. FTLD7 CN MCI CN MCI For the 3D model, AD (Fig. 6.a) the posterior cingu- FTLD 8 CN AD CN AD late cortex is the most discriminating region among oth- ers, while in MCI (Fig. 6.b) pons thalamus and parietal (post-central gyrus/somatosensory cortex) are important out of the eight cases as CN, and the last case was labeled in addition to the posterior cingulate cortex. Further- as AD. VGG16 model distributes the FTLD cases among more, the occipital, left striatum, right frontal cortex, and all of the different class types exist in the training data, right parietal (post-central gyrus/somatosensory cortex) showing only two as AD. ResNet50 predicted one case as are the highlighted regions in CN (Fig. 6.c). And finally in MCI and another one as DLB, while the remaining cases DLB cases (Fig. 6.d) the posterior cingulate cortex is also got the AD label. taking an important role in differentiating DLB besides Figure 7 shows the UMAP visualization of the train- the occipital cortex. ing data as well as FTLD cases. Interestingly, all models The posterior cingulate cortex is important for all the generate representations of FTLD cases in which they given neurodegenerative disorders, i.e., AD, MCI, and are close to each other and not scattered in the repre- DLB, and not in CN. 3D model shows the pattern in this sentation space. Additionally, specialized 3D model and brain region makes the most difference in a cognitively ResNet50 generate representation for FTLD cases that is normal brain compared to dementia-involved ones. The close to the training data that could be expected as brain other interesting pattern is depicted in MCI and CN scans cases share many details in general. Classification maps and probably is the underlying reason for misdiag- outcome of InceptionV3 might seem to be most reason- nosing MCI with CN. The parietal (post-central gyrus/ able as majority of cases receive the same label, 7 out of somatosensory cortex) is highlighted in both maps and 8 were recognized as CN cases. However, according to generally both are sharing many common brain regions previous study that discuss the similarity between AD of interest. and FTLD [26], it should be excepted that more cases to be labeled as AD reflecting similar metabolism patterns Analysis of model robustness linked to AD, learnt by the models when being exposed Our objective of this experiment is to analyze the sen- to training data, which is the case with 3D model that sitivity of models towards similar types of dementia to recognized 4 of the FTLD cases as AD and ResNet50 see how the models recognize these cases and create which labeled 6 out of 8 cases as AD. their equivalent representations. Though, the objective of trained models, as any supervised classification task, Discussion implies that the input space is projected into a finite set of Deep CNNs require large amount of data for training in defined categories (in this case AD, DLB, CN, and MCI). order to achieve good classification performance. How- Yet, in this analysis, we investigate the generated repre- ever, the medical images are hard to obtain, as the col- sentation for FTLD cases and evaluate the labels assigned lecting and labeling of medical data confronted with both to them to see if they are grouped together or scattered data privacy concerns and the requirement for time- across the representation space with labels belonging to consuming expert explanations. Clearly, there are signifi- multiple classes. cantly more datasets of natural images. Thus, TL models Table 5 lists the assigned labels for each one of FTLD have been fine-tuned and applied in medical domain for cases using 3D and TL models. As shown, specialized 3D various classification tasks using their previous learning model and InceptionV3 assigned only two labels to these from ImageNet dataset. cases. For 3D model, half of the cases were recognized as Medical images typically represent much higher reso- AD, while the rest as CN. InceptionV3 recognized seven lution with non-RGB channels volumes. Additionally, Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 11 of 15 CN AD MCI CN 8 5 DLB AD 7 FTLD/AD 4 MCI FTLD/CN UMAP-D2 UMAP-D2 6 DLB 3 FTLD/AD 5 2 FTLD/DLB 4 FTLD/MCI 1 3 2 0 0 2 4 6 8 10 12 0 2 4 6 8 10 UMAP-D1 UMAP-D1 (a) 3D model (b) ResNet50 14 CN 8 CN 12 AD AD 6 10 MCI MCI UMAP-D2 UMAP-D2 8 DLB 4 DLB 6 FTLD/AD FTLD/AD 2 4 FTLD/CN FTLD/DLB 0 FTLD/MCI 2 −2 FTLD/CN 0 −2 −5 0 5 10 0 2 4 6 8 10 UMAP-D1 UMAP-D1 (c) InceptionV3 (d) VGG16 Fig. 7 UMAP visualization showing training data and FTLD cases used as external test-set generated by specialized 3D and TL models. In cases of 3D model and ResNet50 the FTLD cases are identified with red circles as they have very close representation to the training data classification models for medical applications need to stopping conditions help our shallow 3D model, with detect patterns that depend on small and local variations only 4 convolutional layers, to avoid overfitting. From a in the input data. Due to this huge diversity between computational-overhead point of view, feature extrac- natural and medical image modalities, it remains ques- tion from TL models provides timely efficient solution tionable how much ImageNet feature reuse is helpful for for binary classification tasks. However, as we show fine- medical images? In this study we focus our analysis on tuning huge pre-trained model (i.e., ResNet50) on target neuroimaging, with detecting diagnosis of most common medical data can require huge computational resources types of dementia (AD, MCI, and DLB) using 18F-FDG- and take more time than training a smaller custom 3D PET scans of the brain. model from scratch. Designing and training 3D model from scratch can We have demonstrated with three state-of-the-art Ima- be a challenging task, particularly with limited avail- geNet deep architectures and a well-trained 3D model able data. Too simple models might not be able to learn that a customized 3D model can achieve comparable enough representation of input data, leading to poor and even better performance for neuroimaging. In our performance. On the other hand, training a very com- study, we have successfully demonstrated the effective- plex network with limited data is hard which leads to ness of the different models in distinguishing neurode- overfitting. Having a network model with proper size generative brain disorders with binary, 3-way, and 4-way and other effective methods preventing overfitting, such classification tasks. The results show that TL models as proper dropout, learning rate, and early stopping, can obtain superior performance in differentiating AD from get the best results. We have demonstrated with different CN cases, however, performance decreases when add- experimentation the effect of these hyperparameters on ing more disorders to the classification task. Particularly, training 3D model from scratch. Results show that small performance of 3D custom model becomes better when learning rate leads to a long training process that could predicting the diagnosis of two or more brain disorders. get stuck. Furthermore, dropout regularization and early We have shown the improvements obtained by 3D model Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 12 of 15 for 4-way classification over TL models using AUC, also chest x-ray dataset, TL models are worse for recogniz- by visualizing representation space of extracted features. ing Atelectasis, Cardiomegaly, and Consolidation cases. We have also provided further analysis that goes beyond These results are in line with the findings of our work. classification accuracy and demonstrated with occlusion Compared to these previous works, our work takes experiments the areas of interest indicated by 3D model, a step forward and studies the effectiveness of adopt- ResNet50, and InceptionV3. The results show that both ing TL for different classification tasks. Furthermore, we 3D model and ResNet50 provide heatmaps with specific investigated the generated representation of TL mod- regions identified per each class. However, TL models els and compared it with the ones obtained from a cus- were not able to maintain the spatial information among tom 3D model. Additionally, we showed using occlusion brain regions across the consecutive slices. It is worth experiment that the decisions of TL models might not be mentioning that our results are limited to adopting TL as informative nor related to medical data properties (i.e., feature extractors, as we froze the pre-trained weights for the case with InceptionV3). convolution layers of TL models and we only fine-tuned fully connected layers to have more specialized classifiers. Conclusions To understand the benefits and limits of TL and training specialized models for diagnosis of different brain disor- Related work ders, we not only look at standard performance metrics, Pre-training has received much attention in medical but also include analysis of key properties particularly image analysis. For example, Nobili et al. [15] introduced important to training and fine-tuning models, extracted a study to compare performance of deep/TL models and features representing medical data, and network atten- support vector machine (SVM) model for the early diag- tions. The design of TL models is likely to be suboptimal nosis and prognosis of AD using MRI scans by designing for the classification of neurodegenerative diseases. Spe- different binary classification tasks. They trained a very cifically, we found that supervised TL can indeed lead to simple 3D CNN model that has a single convolution layer superior performance on diagnosis of AD versus CN in followed by ReLU activation and max-pooling layer. For timely and data efficient manner, yet for detecting more SVM model, they adopted a feature selection method than a single disorder, TL models do not significantly help to reduce the dimensionality of input. Using this sim- performance. Furthermore, custom 3D trained models ple structure of custom 3D CNN, the results show that perform comparably to TL models for binary classifica- ImageNet pre-trained models outperform SVM and 3D tion, and interestingly perform better for diagnosis of model trained from scratch. Similar results were achieved multiple disorders. Additionally, the results confirm the with a comparative study on a chest X-ray dataset to clas- superiority of the custom 3D-CNN in providing better sify pneumonia [27]. explainable model compared to TL adopted ones. Ding el al [7] used InceptionV3 model for predicting development of Alzheimer’s disease from 18F-FDG-PET Methods scans. The algorithm achieved area under the ROC curve Data preprocessing of 0.98. However, interpreting the model decisions using The 18F-FDG-PET scans were spatially normalized to Saliency maps was not successful as the patterns pre- match the International Consortium of Brain Mapping sented were not specific enough to be mapped to human (ICBM space template for European brains) template interpretable imaging biomarkers. The occlusion experi- [29]. Subsequently, the probability maps of grey matter, ment performed in this study showed the same issue with white matter, cerebrospinal fluid, bone and soft tissue/ air interpreting the decisions of InceptionV3 model. were extracted. The skull stripping was done by retain- Few studies shed the light on the limitations of TL in ing the voxels with high probability of being grey mat- medical imaging. For example, Raghu et al. [28] show ter, white matter or cerebrospinal fluid while discarding that ImageNet pre-training does not improve medi- those likely being bone and soft tissue/air. The normal- cal image classification tasks by evaluating the perfor- ized and skull stripped scans were then visually inspected mance of ResNet50 and InceptionV3 models using Retina to assess their normalization quality and ensure that the images for binary classification task and chest x-ray data- spatial normalization converged to an acceptable solu- set for diagnosing of five different pathologies. Their tion. Both the spatial normalization and skull stripping experiments suggest that the domain mismatch between processes were done using Matlab R2016a and SPM12. natural and medical images inhibits transfer learning. All the brains positioned approximately in the center of The results show that TL models have minimal effect on the volume. performance of detecting Diabetic Retinopathy. Also, for Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 13 of 15 After the spatial normalization step for the input scans, Training Pre-trained Models the first 10 slices as well as the last 9 slices of each scan Testing were excluded as they contain very small objects. Thus, we have the input data as a 3D volume of 95 × 79 × 60 Input Data 3D Model InceptionV3 VGG16 ResNet50 for each case from 757 cases in the dataset. Since scans Experiment 2 Experiment 3 are from various sites, it is required to perform inten- & Experiment 1 Learning rate Kernel Depth Learning strategy sity normalization in order to bring voxel intensities to Dropout a common scale across the whole dataset. Therefore, we Experiment 4 adopted a feature-wise standardization technique pro- Binary classification vided by Keras library. & 3-Way classification 4-Way classification The main idea is to treat each 18F-FDG-PET scan sepa- Experiment 5 rately, then normalize the voxel values using mean and 4-Way classification standard deviation. Particularly, we treated each scan as a Generated representation Classification accuracy (AUC) & sequence of 2D images along the axial plane. We applied Network attention (occlusion) Model Robustness/Sensitivity feature-wise normalization such that each 3D voxel was Fig. 9 The experiments performed for comparative analysis in this normalized by subtracting feature-specific mean then study. We split our data into train and test sets. We have dedicated dividing by the feature-specific standard deviation per experiments for tuning custom 3D model as well as experiments for each scan. We performed further scaling to have all comparative analysis of TL and 3D models intensities values in the range of [0,1]. To transform the input format from 3D to 2D for TL models, we organ- ized the 60 slices of each normalized scan into a 2D grid shown in Fig. 9, the first set of experiments is dedicated resulting in having 2D image with 570 × 790 pixels for to the process of building the CNN from scratch, specifi- each 18F-FDG-PET scan as shown in Fig. 8. Lastly, input cally choosing the hyperparameters to reach an efficient 3D of TL models should have three channels representing model. Three different experiments were designed to deter- RGB color of input images. Our data is considered as gray mine kernel depth to be used for the 3D convolution filters, scale, so we replicate the values across the RGB channels. also we tested different learning strategies with and without early stopping conditions to avoid overfitting. Lastly, com- Comparative analysis pared different values for learning rate and dropout. We briefly describe the experiments developed for this Additionally, we designed the fourth experiment to study and illustrate the objectives of each experiment. As compare efficiency of 3D and TL models in discriminat- ing different neurodegenerative disorders with multiple classification tasks. We evaluate the models by perform- ing binary, 3-way, and 4-way classification. Furthermore, …. …. our fifth experiment provides detailed comparison of per- Slice: 10 forming 4-way classification. Besides evaluating accuracy of different models, we investigate the learned representa- tions by visualizing the generated feature representation …. …. 3D input (95 x 79 x 60) extracted by different models using a dimensionality reduc- Slice: 22 tion technique. We used Uniform Manifold Approximation and Projection (UMAP) to produce 2D features of gener- …. …. ated representations of input data using adopted models. We chose UMAP as it preserves the global data structure as Slice: 34 well as the relative closeness of data points [30]. During the experiments 1 to 4, we adopted 80% and …. …. 20% strategy to split data into training and validation raw scan after normalization datasets. Regarding experiment 5, as we want to report 2D input (570 x 790) accuracy using AUC under ROC curves, we needed to use Fig. 8 Processing 18F-FDG-PET raw scans for generating 3D as well the holdout testing dataset for this purpose, thus we split as 2D inputs for custom 3D model and TL models. For 3D input we build 3D matrix having 60 slices along the axial plane. To transform the data into 90% to be used for training and validation input for 2D-TL models, we arrange these 60 slices into a 2D grid and the remaining 10% is used as an independent testset. Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 14 of 15 We used occlusion to analyze network attention AP2, SM, DA, FN, VG, and MOF provided data of DLB cases, clinical guidelines, and clinical evaluation. All authors read and approve the final manuscript. towards significant areas of interest indicated by the mod- Part of the data used in preparation of this article were obtained from the els. Occlusion Sensitivity helps to determine whether the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and the Frontotemporal output of the model is based on the correct identifica- Lobar Degeneration Neuroimaging Initiative (FTLDNI). ADNI researchers col‑ lect, validate and utilize data, including MRI and PET images [32]. As such, the tion of objects with the high sensitivity associated to spe- investigators from ADNI and FTLDNI contributed data collection regarding AD, cific local structures in input images [31]. The occlusion MCI and FLTD cases. Investigators from ADNI and FTLDNI provided data but experiment is performed by repeatedly occluding spe- did not participate in the analysis or writing of this report. All authors read and approved by the final manuscript. cific regions in the input image and observe the change in the output probability of the model. When important Funding regions for correct classifications are occluded, the prob- This study was part of a collaborative project between Center for Applied Intelligent System Research (CAISR) at Halmstad University, Sweden, and ability drops, hence we can observe a significant change Department of Clinical Physiology, Department of Radiology and the Center in activations of the corresponding feature maps. We per- for Medical Imaging Visualization (CMIV) at Linköping University Hospital, formed occlusion sensitively for 3D and TL models using Sweden, and the European DLB consortium, which was funded by Analytic Imaging Diagnostics Arena (AIDA) initiative, jointly supported by VINNOVA 2D window of size 6 × 5 to be removed from each input (Grant 2017-02447), Formas and the Swedish Energy Agency. VG was sup‑ slice with a stride of 2. We used 2D window for occlusion ported by the Swiss National Science Foundation (projects 320030_169876, to hide the same number of pixels for TL and 3D models. 320030_185028) and the Velux Foundation (project 1123). RB is a senior post‑ doctoral fellow of the Flanders Research Foundation (FWO 12I2121N). Besides Our last comparative analysis task is dedicated to evalu- financial support, AIDA organized multiple meetings to share knowledge ate the robustness of the models using brain scans of a new among participating institutions. dementia type that was not included in training set. For Availability of data and materials such sensitivity analysis we used eight Frontotemporal lobar art of data was collected from Alzheimer’s Disease Neuroimaging Initiative degeneration (FTLD) 18F-FDG-PET scans as another exter- (ADNI) at https://​adni.​loni.​usc.​edu/. nal test set for different models. 18F-FDG-PET brain scan of an FTLD patient is expected to have low FDG uptake in Declarations the frontal and temporal lobes [26]. A patient with a chronic Ethics approval and consent to participate AD can eventually have involvement of the frontal lobes This research study was conducted retrospectively using data obtained and eventually see like a FTLD. Thus, we preformed the last from European DLB (EDLB) Consortium. Local institutional ethics commit‑ experiment to evaluate the predicted labels of these eight tee approvals for the retrospective analyses were available for all centers in Europe, including the transfer of fully anonymized imaging data. Regarding cases using different models, also to visualize the generated the data from Linköping’s University Hospital, informed consent was waived representation of these cases and analyze sensitivity of the for this retrospective assessment and additionally, all patients were informed models towards similar common types of dementia. by letter that their medical data can be rendered anonymous and used for sci‑ entific purposes. All patients from the rest of the centers gave informed writ‑ ten consent for the imaging procedure and radiopharmaceutical application. The study has approval by the Swedish Ethical Review Authority (Etikprövn‑ Abbreviations ingsmyndigheten) with approval number: 2019-00526. Part of data collection 18-F-FDG-PET: 18 F Fluorodeoxyglucose (18F-FDG) Positron emission tomog‑ and sharing for this project was funded by the Alzheimer’s Disease Neuroim‑ raphy; 2D: Two dimensional; 3D: Three dimensional; AD: Alzheimer’s disease; aging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and ADNI: Alzheimer’s disease neuroimaging initiative; AUC​: Area Under the ROC DOD ADNI (Department of Defense award number W81XWH-12-2-0012) curve; CI: Confidence Interval; CN: Cognitively normal brain; CNN: Convolution neural network; DL: Deep learning; DLB: Dementia with Lewy bodies; EDLB: Consent to publish European DLB consortium; FTLD: Frontotemporal lobar degeneration; KFCV: Not applicable. K-Fold cross validation; MCI: Mild cognitive impairment; NMI: Normalized mutual information; ROC: Receiver operating characteristic curve; TL: Transfer Competing interests learning; UMAP: Uniform manifold approximation and projection. The authors declare no conflicts of interest. Acknowledgements Author details The authors would like to thank Johannes van Esch from Halmstad University 1 Center for Applied Intelligent Systems Research (CAISR), Halmstad University, for his continuous support on running the developed scripts using GPU Halmstad, Sweden. 2 National Cheng Kung University in Tainan, Taipei City, support. Taiwan. 3 Department of Clinical Physiology, Institution of Medicine and Health Sciences, Linköping, Sweden. 4 Department of Nuclear Medicine, Medical About this supplement Imaging Area, La Fe University Hospital, Valencia, Spain. 5 Servicio de Medicina This article has been published as part of BMC Medical Informatics and Deci‑ Nuclear, Hospital de la Santa Creu i Sant Pau, Universitat Autónoma de Barce‑ sion Making Volume 22 Supplement 6, 2022 Selected articles from the 17th lona, Barcelona, Spain. 6 Nuclear Medicine Unit, IRCCS Ospedale Policlinico San International Conference on Computational Intelligence Methods for Bioinfor‑ Martino, Genoa, Italy. 7 Department of Diagnostic Radiology, Linköping Univer‑ matics and Biostatistics (CIBB 2021). The full contents of the supplement are sity Hospital, Linköping, Sweden. 8 Department of Medical Physics, Linköping available online at https://​bmcme​dinfo​rmdec​ismak.​biome​dcent​ral.​com/​artic​ University Hospital, Linköping, Sweden. 9 National Institute of Nuclear Physics les/​suppl​ements/​volume-​22-​suppl​ement-6. (INFN), Genoa section, Genoa, Italy. 10 Department of Nuclear Medicine, University Hospital, LMU Munich, Munich, Germany. 11 Department of Nuclear Author contributions Medicine, Inselspital, University Hospital Bern, Bern, Switzerland. 12 Labora‑ AS, KE, and SB wrote the manuscript; AS and JRC developed the models and tory for Cognitive Neurology, Department of Neurosciences, KU, Leuven, created the figures; The rest of authors as order in authorship: AD, BMS, VC, Belgium. 13 Neurology Department, University Hospitals Leuven, Leuven, MB1, RS, MR, MAC, AC, MB2, AR, RB, RV, MGK, MT, NN, GBF, AWL, BNMB, AP1, Soliman et al. BMC Medical Informatics and Decision Making (2022) 22:318 Page 15 of 15 Belgium. 14 Department of Neurology, University Medical Centre, Ljubljana, 14. Maqsood M, Nazir F, Khan U, Aadil F, Jamal H, Mehmood I, Song O-Y. Slovenia. 15 Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia. Transfer learning assisted classification and detection of alzheimer’s 16 Department of Clinical Neurosciences, Geneva University Hospitals, Geneva, disease stages using 3d mri scans. Sensors. 2019;19(11):2645. Switzerland. 17 LANVIE (Laboratoire de Neuroimagerie du Vieillissement), 15. Nobili F, Arbizu J, Bouwman F, Drzezga A, Agosta F, Nestor P, Walker Z, Department of Psychiatry, University Hospitals, Geneva, Switzerland. 18 VU Boccardi M. EANM-EAN Task Force for the Prescription of FDG-PET for Medical Center Alzheimer Center, Amsterdam, The Netherlands. 19 Department Dementing Neurodegenerative Disorders. European Association of of Radiology and Nuclear Medicine, Amsterdam Neuroscience , Amsterdam Nuclear Medicine and European Academy of Neurology recommenda‑ UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands. 20 Neurology tions for the use of brain 18 F-fluorodeoxyglucose positron emission Unit, Department of Clinical and Experimental Sciences, University of Brescia, tomography in neurodegenerative cognitive impairment and dementia: Brescia, Italy. 21 Nuclear Medicine Unit, IRCCS Ospedale Policlinico San Martino, Delphi consensus. Eur J Neurol. 2018;25(10):1201–17. Genoa, Italy. 22 Centre for Age‑Related Medicine (SESAM), Stavanger University 16. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Van‑ Hospital, Stavanger, Norway. 23 Department of Neuroscience (DINOGMI), houcke V, Rabinovich A. Going deeper with convolutions, In: proceedings University of Genoa, Genoa, Italy. 24 Division of Nuclear Medicine and Molecu‑ of the IEEE conference on computer vision and pattern recognition, lar Imaging, University Hospitals and NIMTLab, Geneva University, Geneva, 2015;pp. 1–9. Switzerland. 25 Center for Medical Image Science and Visualization (CMIV), 17. Simonyan K, Zisserman A. Very deep convolutional networks for large- Linköping University, Linköping, Sweden. 26 Department of Biomedical Sci‑ scale image recognition, arXiv preprint arXiv:​1409.​1556. ences, University of Antwerp, Antwerp, Belgium. 27 Department of Old Age 18. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition, Psychiatry, Institute of Psychiatry, Psychology and Neuroscience, King’s College In: proceedings of the IEEE conference on computer vision and pattern London, London, England. recognition, 2016;pp. 770–778. 19. Etminani K, Soliman A, Davidsson A, Chang J, Martínez-Sanchis B, Byttner Received: 10 November 2022 Accepted: 15 November 2022 S, Camacho V, Bauckneht M, Stegeran R, Ressner M, et al. A 3d deep learning model to predict the diagnosis of dementia with lewy bodies, alzheimer’s disease and mild cognitive impairment using brain 18f-fdg pet. Eur J Nucl Med Mol Imaging. 2022;49(2):563–84. 20. European dlb (edlb) consortium with its core laboratory at genoa, italy, References https://​www.​ge.​infn.​it/​wordp​ress/?​page_​id=​77, [Online; accessed 1. Ferri CP, Prince M, Brayne C, Brodaty H, Fratiglioni L, Ganguli M, Hall K, 2022-05-09]. Hasegawa K, Hendrie H, Huang Y, et al. Global prevalence of dementia: a 21. Kramberger MG, Auestad B, Garcia-Ptacek S, Abdelnour C, Olmo JG, Delphi consensus study. The lancet. 2005;366(9503):2112–7. Walker Z, Lemstra AW, Londos E, Blanc F, Bonanni L, et al. Long-term 2. Hogan DB, Fiest KM, Roberts JI, Maxwell CJ, Dykeman J, Pringsheim T, cognitive decline in dementia with lewy bodies in a large multicenter, Steeves T, Smith EE, Pearson D, Jetté N. The prevalence and incidence international cohort. J Alzheimers Dis. 2017;57(3):787–95. of dementia with lewy bodies: a systematic review. Can J Neurol Sci. 22. McKeith IG, Dickson DW, Lowe J, Emre M, O’brien J, Feldman H, Cum‑ 2016;43(S1):S83–95. mings J, DudaJ, Lippa C, Perry E, et al. Diagnosis and management of 3. Farias ST, Mungas D, Reed BR, Harvey D, DeCarli C. Progression of mild dementia with lewy bodies: third report of the dlb consortium, Neurol‑ cognitive impairment to dementia in clinic-vs community-based cohorts. ogy. 2005;65 (12) 1863–1872. Arch Neurol. 2009;66(9):1151–7. 23. Alzheimer’s disease neuroimaging initiative, http://​adni.​loni.​usc.​edu/, 4. Ahmed MR, Zhang Y, Feng Z, Lo B, Inan OT, Liao H. Neuroimaging and [Online; accessed 2022-05-09]. machine learning for dementia diagnosis: recent advancements and 24. Mueller SG, Weiner MW, Thal LJ, Petersen RC, Jack C, Jagust W, Trojanowski future prospects. IEEE Rev Biomed Eng. 2018;12:19–33. JQ, Toga AW, Beckett L. The alzheimer’s disease neuroimaging initiative. 5. Nobili F, Arbizu J, Bouwman F, Drzezga A, Agosta F, Nestor P, Walker Z, Neuroimaging Clin N Am. 2005;15(4):869. Boccardi M, E.-E. T. F. for the Prescription of FDG-PET for Dementing Neu‑ 25. Papers with code: image classification on imagenet, https://​paper​swith​ rodegenerative Disorders, Festari C, et al. European association of nuclear code.​com/​sota/​image-​class​ifica​tion-​on-​image​net, [Online; accessed medicine and european academy of neurology recommendations for the 2022-05-09]. use of brain 18f-fluorodeoxyglucose positron emission tomography in 26. Brown RK, Bohnen NI, Wong KK, Minoshima S, Frey KA. Brain pet in neurodegenerative cognitive impairment and dementia: Delphi consen‑ suspected dementia: patterns of altered fdg metabolism. Radiographics. sus. European J Neurol. 2018; 25 (10) 1201–1217. 2014;34(3):684–701. 6. Al-Shoukry S, Rassem TH, Makbol NM. Alzheimer’s diseases detection by 27. Yadav SS, Jadhav SM. Deep convolutional neural network based medical using deep learning algorithms: a mini-review. IEEE Access. 2020;8:77131–41. image classification for disease diagnosis. J Big Data. 2019;6(1):1–18. 7. Ding Y, Sohn JH, Kawczynski MG,Trivedi H, Harnish R, Jenkins NW, Lituiev 28. Raghu M, Zhang C, Kleinberg J, Bengio S. Transfusion: Understanding D, Copeland TP, Aboian MS, Mari Aparici C, et al. A deep learning model transfer learning for medical imaging, arXiv preprint arXiv:​1902.​07208. to predict a diagnosis of alzheimer disease by using 18f-fdg pet of the 29. Mazziotta JC, Toga AW, Evans A, Fox P, Lancaster J, et al. A probabilistic brain, Radiology. 2019;290 (2) 456–464. atlas of the human brain: theory and rationale for its development. Neu‑ 8. Singh S, Srivastava A, Mi L, Caselli RJ, Chen K, Goradia D, Reiman EM, roimage. 1995;2(2):89–101. Wang Y. Deep-learning-based classification of fdg-pet data for alzheimer’s 30. McInnes L, Healy J, Melville J. Umap: Uniform manifold approximation disease categories, in: 13th international conference on medical informa‑ and projection for dimension reduction, arXiv preprint arXiv:​1802.​03426. tion processing and analysis, Vol. 10572, International Society for Optics 31. Zeiler MD, Fergus R. Visualizing and understanding convolutional net‑ and Photonics, 2017; p. 105720J. works. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, editors. Computer vision 9. Choi H, Kim YK, Yoon EJ, Lee J-Y, Lee DS. Cognitive signature of brain fdg - ECCV 2014. Cham: Springer International Publishing; 2014. p. 818–33. pet based on deep learning: domain transfer from alzheimer’s disease to 32. A complete listing of adni investigators, http://​adni.​loni.​usc.​edu/​wp-​ parkinson’s disease. Eur J Nucl Med Mol Imaging. 2020;47(2):403–12. conte​nt/​uploa​ds/​how_​to_​apply/​ADNI_​Ackno​wledg​ement_​List.​pdf, 10. Weiss K, Khoshgoftaar TM, Wang D. A survey of transfer learning. J of Big [Online; accessed 2022-05-09]. data. 2016;3(1):1–40. 11. Alzubaidi L, Fadhel MA, Al-Shamma O, Zhang J, Santamaría J, Duan Y, Oleiwi SR. Towards a better understanding of transfer learning for medical Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub‑ imaging: a case study. Appl Sci. 2020;10(13):4523. lished maps and institutional affiliations. 12. Zhang Y, Davison BD. Impact of imagenet model selection on domain adaptation, In: proceedings of the IEEE/CVF winter conference on appli‑ cations of computer vision workshops, 2020;pp. 173–182. 13. Kornblith S, Shlens J, Le QV. Do better imagenet models transfer better?, In: proceedings of the IEEE/CVF conference on computer vision and pat‑ tern recognition, 2019;pp. 2661–2671.