NeuroImage: Clinical 30 (2021) 102659 Contents lists available at ScienceDirect NeuroImage: Clinical journal homepage: www.elsevier.com/locate/ynicl Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references Alexandra de Sitter a, 1, Jessica Burggraaff b, 1, *, Fabian Bartel a, Miklos Palotai c, Yaou Liu a, Jorge Simoes a, Serena Ruggieri d, e, Katharina Schregel c, f, Stefan Ropele g, Maria A. Rocca h, i, Claudio Gasperini e, Antonio Gallo j, Menno M. Schoonheim k, Michael Amann l, m, Marios Yiannakas n, Deborah Pareto o, Mike P. Wattjes a, p, Jaume Sastre-Garriga q, Ludwig Kappos m, Massimo Filippi h, i, r, s, Christian Enzinger t, Jette Frederiksen u, Bernard Uitdehaag b, Charles R.G. Guttmann c, Frederik Barkhof a, v, 2, Hugo Vrenken a, 2, 3 a Radiology and Nuclear Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, Amsterdam, NL, Netherlands b Department of Neurology, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, Amsterdam, NL, Netherlands c Center for Neurological Imaging, Department of radiology, Brigham and Women’s Hospital, Harvard Medical School Boston, MA, USA d Department of Human Neurosciences, “Sapienza” University of Rome, Rome, IT, Italy e Department of Neurosciences, San Camillo Forlanini Hospital, Rome, IT, Italy f Institute of Neuroradiology, University Medical Center Goettingen, Goettingen, DE, Germany g Department of Neurology, Medical University of Graz, Graz, AT, Austria h Neuroimaging Research Unit, Institute of Experimental Neurology, Division of Neuroscience, United States i Neurology Unit, San Raffaele Scientific Institute, UniSR, Milan, IT, Italy j Division of Neurology and 3T MRI Research Center, Department of Advanced Medical and Surgical Sciences, University of Campania “Luigi Vanvitelli”, Naples, IT, Italy k Department of Anatomy and Neurosciences, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, NL, Netherlands l Medical Image Analysis Center (MIAC), United States m Neurologic Clinic and Policlinic and Neuroradiology, Department of Biomedical Engineering, University Hospital Basel, Basel, CH, Switzerland n Department of Neuroinflammation, Institute of Neurology, UCL, London, UK o Section of Neuroradiology and MRI Unit, Department of Radiology, University Hospital Valld’Hebron, Autonomous University of Barcelona, Barcelona, ES, Spain p Deptartment of Diagnostic and Interventional Neuroradiology, Hannover Medical School, Hannover, DE, Germany q Department of Neurology, University Hospital iValld’Hebron, Autonomous University of Barcelona, Barcelona, ES, Spain r Neurophysiology Unit, San Raffaele Scientific Institute, Italy s Vita-Salute San Raffaele University, Milan, IT, Italy t Division of Neuroradiology, Vascular and Interventional Radiology, Department of Radiology, Medical University of Graz, Graz, AT, Austria u Department of Neurology, Glostrup University Hospital, Copenhagen, DK, Denmark v Institutes of Neurology & Healthcare Engineering, UCL, London, UK A R T I C L E I N F O A B S T R A C T Keywords: Background: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple Multiple Sclerosis sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are MRI needed for objective evaluation and training of new methods. Deep grey matter Objectives: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate Atrophy Segmentation the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated Reference set segmentation approach (FASTSURF). * Corresponding author at: Amsterdam UMC, location VUmc, Department of Neurology, PO Box 7057, 1007 MB Amsterdam, The Netherlands. E-mail address: j.burggraaff@amsterdamumc.nl (J. Burggraaff). 1 Both authors contributed equally. 2 Both authors contributed equally. 3 for the MAGNIMS Study Group. https://doi.org/10.1016/j.nicl.2021.102659 Received 4 August 2020; Received in revised form 19 March 2021; Accepted 31 March 2021 Available online 6 April 2021 2213-1582/© 2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 Methods: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. Results: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. Conclusions: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload. 1. Introduction protocol for manually tracing the caudate, putamen and thalamus. Secondly, the reliability of the protocol was investigated with multiple Patients with multiple sclerosis (MS) exhibit damage of the grey expert readers, on multi-center MS images. Thirdly, we evaluated the matter (GM), including focal lesions and atrophy. (Du Toit et al., 2008; accuracy of FASTSURF to reconstruct full segmentations of the dGM in Bagnato et al., 2006; Geurts et al., 2005) GM atrophy can be quantified which sparse delineations served as input. from structural brain magnetic resonance images (MRI) and has become an important and clinically relevant imaging outcome measure of MS. In 2. Materials and methods particular, atrophy of deep GM (dGM) structures such as the caudate nucleus, putamen and thalamus has become of interest in MS, as it has 2.1. Dataset and MRI acquisition been shown to correlate with important clinical outcome such as cognition. (Schoonheim et al., 2015; Bishop et al., 2017; Bermel et al., Brain MRI scans of 12 healthy controls (HCs) (8 females) and 23 MS 2003; Houtchens et al., 2007; Pagani et al., 2005) Atrophy measures of patients (12 females) from nine centers were retrospectively included, the dGM may serve as potential imaging biomarkers in MS. However, which were all acquired as part of two previously described MAGNIMS the applicability for everyday clinical use is limited, in part because studies (www.magnims.eu). (Rocca et al., 2014; Ropele et al., 2014) The there is a so far unmet need for reliable automated segmentation sample used for this study was selected to ensure that: many different methods. (Wattjes et al., 2015; Sastre-Garriga et al., 2020) MR scanners were included, most of the patients had progressive MS Current state-of-the-art and frequently used automated segmentation disease course types, and that the distributions of sex and age were methods suffer from substantial limitations with respect to both repro­ closely matched to the overall dataset. The HCs were matched with the ducibility and accuracy, which is partly due to the presence of MS MS patients on scanner type, sex and age. Demographics of the subjects pathological changes. (Popescu et al., 2014, 2016; Gelineau-Morel et al., are shown in Table 1. Table 2 shows the number of subjects per center 2012; Meijerman et al., 2018; Amiri et al., 2018; de Sitter et al., 2020) (MR scanner). All local institutional review boards approved the original Specifically, there are various confounds that can affect the measure­ study and written informed consent had been obtained from all partic­ ment of dGM atrophy: image registration and segmentation can be ipants. MR imaging was performed on 3.0 Tesla whole-body MR sys­ negatively affected by the presence of white matter lesions, (Gelineau- tems, and near-isotropic, ~1mm (Geurts et al., 2005) voxel size, 3D T1- Morel et al., 2012; de Sitter et al., 2020) generalized or local atrophy, or weighted datasets were included. Details on image acquisition param­ subtle tissue contrast changes (Amiri et al., 2018; Westlye et al., 2009). eters used in each center are listed in Table 2. To achieve accurate automated dGM segmentation in the presence of MS abnormalities, it is important that new methods are validated against 2.2. Manual segmentation protocol expert reference outlines of dGM in representative MS samples. There­ fore, we developed a standardized protocol for manually delineating the The segmentation protocol (see Supplementary File S1 for the full caudate, putamen and thalamus on 3D T1-weighted MRI and evaluated protocol) was specifically developed for manually tracing dGM its quality in terms of reliability within and amongst multiple expert raters, using a multi-center MS imaging dataset. Table 1 To validate the automated methods for measuring dGM atrophy, a Demographics of healthy controls and MS patients. more complete analysis in a larger multi-center set of image volumes is required. Since manual outlining is difficult, labor-intensive and time- Set Type Na Age in Disease DD in EDSSc yearsb types yearsb consuming, (Grimaud et al., 1996; Paty et al., 1986; Fischl et al., 2002) we endeavored to reduce the workload by reconstructing full Total HC 12 38.4 ± (8) 7.8 semi-automated segmentations from sparse delineations as input. Spe­ Patient 23 42.9 ± 11 RR, 5 SP, 11.6 ± 2.5 cifically, we investigated the performance of a recently developed semi- (12) 9.9 7 PP 6.9 (2.5) automated technique called ‘FAst Segmentation Through SURface Training HC 5 (5) 34.7 ± Fairing’ (FASTSURF), (Bartel et al., 2019) which was demonstrated as a 8.0 proof-of-concept for the hippocampus in Alzheimer patients by Bartel Patient 12 44.4 ± 7 RR, 2 SP, 3 12.1 ± 2.0 (6) 11.9 PP 8.3 (2.5) et al. (2019). Since this technique exhibited excellent accuracy for Test HC 7 (3) 41.1 ± hippocampus, we hypothesized that FASTSURF can also be used to 7.1 generate accurate reference segmentations of various other brain Patient 11 41.3 ± 4 RR, 3 SP, 4 11.1 ± 3.5 structures, with substantially lower workload than full manual tracings. (6) 7.4 PP 5.59 (2.5) This may provide an important impetus towards improved segmentation a Number of subjects (Number of females). of dGM. In future work, when this protocol is applied, such segmenta­ b Mean ± standard deviation. tions can be used to train or optimize automated methods such that these c Median (range). will segment the structures of interest well in MS cases. Abbreviations: HC = healthy control, DD = disease duration, EDSS = expanded To summarize, in this study we aimed first, to develop a standardized disability status scale, RR = relapsing-remitting, SP = secondary-progressive, PP = primary-progressive. 2 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 structures on 3D T1-weighted MRI scans of MS patients, by neurologists on mesh processing procedures using a surface fairing technique that has and neuroradiologists with broad experience in the field of MS and MRI, been described in detail previously. (Bartel et al., 2019) Briefly, to supervised by neuroradiologists (F.B. with>30 years of experience and reduce the delineation time for manual observers, only a few contours M.P.W. with>20 years of experience). Together, we reviewed the have to be outlined, at regular slice intervals. First, these sparse contours literature and studied images of histopathological specimens, MRI, are interpolated so that each contour has the same number of points. A (stereotactic) anatomy and computational 3D reconstructions; most of closed mesh is then constructed by placing intermediate contours be­ which are also listed in the protocol as recommended study material for tween the known contours. Vertex positions for the intermediate con­ the readers, since they help to understand the 3D anatomical position/ tours are obtained by solving the following bi-Laplacian system of location and shape of the structure of interest in the human brain. equations for the unknown x, y and z-coordinates of the vertices of the Anatomical definitions were specified for each structure, supported in intermediate contours: the protocol with example images from our own dataset. Alongside the ∑ ∑ ∑ anatomical landmarks, strict guidelines on how to recognize the outer­ Ln,m 2 xm = Ln,m 2 ym = Ln,m 2 zm = 0 m m m most edges of the structures on orthogonal planes were described. Certain decisions on whether to include the geniculate bodies as part of in which the Laplacian filter Ln,m represents the connectivity graph the thalamus and how to distinguish the caudate and the putamen from with n and m being the mesh vertices. Solving these equations leads to a the nucleus accumbens were based on a mixture of literature studies, smooth surface mesh passing through the delineated points with mini­ expert opinion and practical reasoning. mum curvature. Practically, the segmentation procedure consisted of two phases. Originally FASTSURF was designed for the hippocampus. In the First, demarcating the edges of the dGM structures on orthogonal slices, present study, we quantitatively investigated this application for seg­ and second, tracing and fill the inside of the path defined by the reader in mentation of the thalamus, caudate nucleus and putamen. First, similar the axial plane, respecting the boundaries previously defined and the to the approach of Bartel et al. (2019), sparse contours were extracted anatomical definitions that were specified for each structure. from fully manually segmented structures. The segmented structures were converted to meshes using the marching cubes algorithm and sparse contours were extracted at regular intervals, which served as 2.3. Manual tracing input for FASTSURF (From now on: ‘FASTSURF with sparse contours’). Second, in a small subset of six images, one of the raters manually Manual outlining was performed within the online framework of the traced 10 predefined contours for each structure de novo, i.e., without SPINE virtual laboratory (https://spinevirtuallab.org/), developed by creating outlines of the structures on the intermediate slices. These 10 de the Center for Neurological Imaging (CNI) at Brigham and Women’s novo contours were used as input for FASTSURF (From now on: ‘FAST­ Hospital. This web-based program allows visualization of MR images in SURF with de novo contours’). This allowed us to evaluate whether the axial, coronal, and sagittal orientations to facilitate 3D anatomical protocol can be combined with FASTSURF to reconstruct full segmen­ interpretation. The voxel-wise labeling process was completely manual. tations of the dGM using only 10 de novo delineations as input. It involved no thresholding, seed-growing, shape fitting or other auto­ Extra information on sparse contour simulation and training of mated interference. Following the segmentation protocol described FASTSURF can be found in the supplementary data (File S2 and above and presented in Supplementary File S1, three expert readers Table S1-S4). manually delineated the caudate nucleus, putamen and thalamus as a whole on axial slices, in a slice-by-slice manner, for all 35 images. The 2.5. Quantitative performance analysis readers were a trained neurologist (J.B.), neuroscientist (J.S.) and neurologist (S.R.), blinded to the subject characteristics. To assess the In Fig. 1 an overview of the study design is shown. The two main intra-rater variability, a random subset of dGM structures for 3 subjects quantitative performance metrics are; (i) intra- and inter-rater agree­ (1 HC and 2 MS patient) were delineated a second time by all 3 raters in ment of manual outlines of 3 raters on 35 images (ii) accuracy of a separate session more than three months later. To assess the validity of FASTSURF in terms of volumetric and spatial agreement. In the next FASTSURF, another subset of six subjects (2 HC and 4 MS patients) were subparagraphs more details are described on the experiment and the delineated in a separate session by one reader (J.B.), which included statistical analyses. only 10 predefined axial slices per structure. 2.6. Intra- and inter-rater agreement of manual tracings 2.4. Reconstructions from sparse delineations: FASTSURF method The manual outlines of the three raters were evaluated on both intra- To allow construction of reference segmentations with reduced rater and inter-rater reliability. workload, the possibility of reconstructing full segmentations from sparse delineations was investigated. For this purpose, the semi- automated segmentation method FASTSURF was used, which is based Table 2 An overview of the acquisition parameters for each center. Institute Na Scanner manufacturer, scanner type TR TE TI FA Acquisition (Voxel size (mm3) (ms) (ms) (ms) (◦) A 13 GE, Signa HDxt 7.8 3 450 12 256x256x188 (0.976x0.976x1) B 2 Siemens, Trio 2300 2.98 900 9 232x256x176 (1x1x1) C 2 Siemens, Trio 1570 2.70 900 9 160x256x256 (1x1x1) D 1 Philips, Achieva 6.9 2.78 831 9 160x240x240 (1x1x1) E 5 Siemens, Trio 1900 2,1 900 9 224x256x176 (1x1x1) F 2 Siemens, Trio 2200 2,94 900 10 256x192x192 (1x1x1) G 5 Philips, Achieva 8,3 3,72 1000 8 256x256x192 (1x1x1) H 2 GE, Signa HDxt 5,5 1,76 450 10 256x256x188 (1x1x1) I 1 Philips, Achieva 8,3 3,72 1000 8 256x256x192 (1x1x1) a Number of subjects per institute. Abbreviations: TR = repetition time, TE = echo time, TI = inversion time, FA = flip angle. 3 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 Fig. 1. Overview of study design. A flowchart of the study design divided in two boxes; 1) manual segmentation protocol and 2) quantitative performance metrics. The manual segmentation protocol was used to create three datasets; quantitative performance metrics were used to assess spatial (JI, DSC and CIgen) and volumetric agreement (ICC). Abbreviations: CIgen = generalized conformity index; ICC = intraclass correlations; DSC = Dice Similarity Coefficient, JI = Jaccard index; N = number of subjects. Intra-rater agreement was assessed spatially with the Jaccard Index contours’ and manual references. With DSC = 2TP/(2TP + FP + FN), (JI); JI = Vi∩j between the first and second manual tracing of the with TP, FP and FN, respectively True Positive, False Positive and False Vi∪j Negative. structures. Vi∩j is volume of intersection of outline i and j and Vi∪j is volume of union of outline i and j. ‘FASTSURF with de novo contours’ was validated on six images Inter-rater spatial agreement was assessed spatially with the gener­ containing only 10 contours of each structure as input by one rater alized conformity index (CIgen), (Kouwenhoven et al., 2009) which (J.B.). The segmentations that were obtained through ‘FASTSURF is essentially a generalization of the Jaccard index for multiple raters; with de novo contours’ were compared to the manual reference on a full definition and explanation is provided in the supplementary spatially agreement (JI and DSC), and compared with the agreement File S3. Volumetrically, a two-way mixed effects model for intraclass between ‘FASTSURF with sparse contours’ and manual reference. correlation coefficients (ICC) using an absolute agreement definition To evaluate the agreement between the fused manual labels and was measured between the three raters (Shrout and Joseph, 1979). ‘FASTSURF with sparse contours’ and ‘FASTSURF with de novo contours’, Bland-Altman plots were created in which the difference 2.7. Fastsurf of two paired measurements [(A-B)] was plotted against the average of the two measurements [(A + B)/2], (Giavarina, 2015; Altman, The performance of FASTSURF for dGM structures was evaluated in 1983) with separate colors for MS and controls to visually inspect four ways; a) optimization of FASTSURF parameters; b) optimized whether there are disease specific effects. We ran a paired sample t- FASTSURF with sparse delineations from full segmentations as input; c) test (two-sided) to examine whether the mean of the difference optimized FASTSURF with 10 de novo contours as input; and d) agree­ equals 0. ment between the expert manual labels and ‘FASTSURF with sparse contours’ and ‘FASTSURF with de novo contours’ using Bland-Altman plots. 2.8. Interpretation of statistical results For optimization of FASTSURF parameters, the dataset was divided JI, DSC and CIgen range between 0 and 1, where perfect overlap into a training set (N = 17) and a test set (N = 18). In both groups, the yields a JI, DSC or CIgen value of 1, and no overlap yields a JI, DSC or different centers and numbers of patients and controls were equally CIgen value of 0. A JI or CIgen > 0.7 and a DSC > 0.8 is regarded as distributed (see Table 2). The training set was used to find the excellent. (Bartko, 1991) optimal settings for the parameters of FASTSURF and the test set was ICC also ranges between 0 and 1. We used Altman’s criteria to used to study the performance of optimized FASTSURF compared to interpret the ICCs: <0.40 was considered as poor reliability, 0.40 to 0.74 the manual outlines. was considered fair to good, and ≥ 0.75 was considered excellent. The optimal settings obtained from the training set were applied in (Cicchetti, 1994) the test sets of each rater’s segmentations separately. Optimal set­ tings can be found in Supplementary Table 5; for contours the 3. Results optimal setting was 10. The spatial agreement between the resulting three datasets of ‘FASTSURF with sparse contours’ were evaluated Fig. 2 shows example images of dGM delineations for each rater with CIgen. The results were compared to the inter-rater agreement separately and their overlap. Fig. 3 shows the tracings of one rater and of the expert manual outlines. the reconstructed FASTSURF segmentations for the caudate, putamen and thalamus. Additionally, the segmentations of all 3 raters were pooled as one dataset, which served to compare ‘FASTSURF with sparse contours’ to 3.1. Intra- and inter-rater agreement of manual tracings the manual references on both volumetric as spatial agreement. Volu­ metrically, the agreement was quantified with the ICC for absolute The intra-rater agreement on spatial overlap was excellent with a agreement; (Koch, 1982) spatial agreement was assessed through the JI mean (across raters) JI of 0.83 ± 0.11, 0.86 ± 0.05 and 0.86 ± 0.10 for and Dice Similarity Coefficient (DSC) between ‘FASTSURF with sparse the caudate nucleus, putamen and thalamus, respectively. 4 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 Fig. 2. Overview of manual delineations of the 3 raters and their overlap. From left to right: Axial 3D T1-weighted MRI slice with segmentations, 2D view of manual reference of rater 1 to 3 and 2D view of overlap of raters with green, blue and red one rater, purple and pink two and orange three raters. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) Abbreviations: CN = caudate nucleus; TH = thalamus; PU = putamen. Fig. 3. Overview of manual delineation of one rater and FASTSURF segmentations. (a) the tracings of one rater; (b) the extracted contours from full seg­ mentations which were used for ‘FASTSURF with sparse contours’; (c) the reconstructed FASTSURF segmentations for the caudate, putamen and thalamus; (d) an intersection of ‘FASTSURF with sparse contours’ and the manual references in 3D; and (e) the overlap of c and d. The inter-rater agreement on spatial overlap was excellent; with both number of intermediate contours added by FASTSURF between two left and right hemisphere pooled together, the average CIgen for the outlined contours and the number of points used for each contour. caudate nucleus, putamen and thalamus were 0.74 ± 0.05, 0.74 ± 0.06 and 0.75 ± 0.06 respectively. The volumetric agreement between the 3.2.2. Agreement of ‘FASTSURF with sparse contours’ raters was also excellent with an ICC of 0.76 for the caudate, 0.79 for the In the test set (N = 18), the performance of optimized FASTSURF was putamen and 0.79 for the thalamus (left and right hemisphere pooled quantitatively evaluated. In Table S6 the CIgen values for ‘FASTSURF together). Table 3 provides average CIgen and ICC values for all struc­ with sparse contours’ are provided for all structures bilaterally, as well tures, both for each hemisphere separately and averaged. as averaged across hemispheres. Inter-rater agreement on spatial over­ lap for ‘FASTSURF with sparse contours’ was almost identical to the agreement between expert manual references. 3.2. Fastsurf The volumetric and spatial agreement of ‘FASTSURF with sparse contours’ with manual reference segmentations was excellent (Table 4), 3.2.1. Parameter optimization for FASTSURF with total bilateral volume ICCs for absolute agreement of 0.979 for the The results of the parameter optimization for the FASTSURF soft­ caudate nucleus, 0.999 for the putamen and 0.999 for the thalamus and ware, carried out on the test set (N = 17) are shown in the supple­ mean JI of 0.92 ± 0.02, 0.95 ± 0.01, 0.96 ± 0.02, respectively. mentary Tables S1-S5. The parameters that were optimized were: the orientation of outlining planes, the number of the outlined contours, the Table 3 Table 4 Inter-rater agreement between the three raters; the generalized conformity ICC, Jaccard Index and Dice Similarity Coefficient between manual references index (CIgen) and intra-class correlations (ICC) between raters, separated for and ‘FASTSURF with sparse contours’ and manuala references. structure and hemisphere. Structure Hemispheres ICC Jaccard Dice Similarity Coefficient Structure Hemipshere CIgena ICC Indexa a Caudate Both 0.738 ± 0.049 0.762 Caudate Both 0.979 0.918 ± 0.023 0.924 ± 0.026 Left 0.733 ± 0.054 0.771 Left 0.984 0.920 ± 0.021 0.925 ± 0.028 Right 0.753 ± 0.042 0.766 Right 0.973 0.914 ± 0.024 0.923 ± 0.025 Putamen Both 0.736 ± 0.059 0.794 Putamen Both 0.999 0.952 ± 0.013 0.960 ± 0.019 Left 0.728 ± 0.061 0.769 Left 0.999 0.951 ± 0.012 0.958 ± 0.020 Right 0.753 ± 0.049 0.833 Right 0.999 0.954 ± 0.013 0.961 ± 0.017 Thalamus Both 0.746 ± 0.058 0.785 Thalamus Both 0.999 0.962 ± 0.021 0.967 ± 0.030 Left 0.762 ± 0.039 0.815 Left 0.999 0.960 ± 0.023 0.965 ± 0.030 Right 0.741 ± 0.072 0.762 Right 0.999 0.964 ± 0.019 0.970 ± 0.030 a Abbreviations: CIgen = generalized conformity index, ICC = intra-class Mean ± standard deviation. correlations. Abbreviations: ICC = the intraclass correlation coefficient (two-way mixed a Mean ± standard deviation. model with absolute agreement). 5 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 3.2.3. Agreement of ‘FASTSURF with de novo contours’ of the structures. Therefore it could be that the widest part of the The average volumes of the reconstructed dGM using ‘FASTSURF structure was not taken into account. Nevertheless, future studies should with de novo contours’ are displayed in Table 5, alongside the average help to improve FASTSURF to ensure greater accuracy. volumes of the manual reference tracings and segmentations from Visually, the Bland-Altman plots did not reveal clear disease specific ‘FASTSURF with sparse contours’ for the same six subjects. Furthermore, effects on the agreement in MS patient versus controls. In future studies, the JI between FASTSURF results and the manual references are shown. more images with de novo partial contours obtained by multiple raters The average JI between ‘FASTSURF with de novo contours’ and the should be generated to further validate the accuracy of FASTSURF in this manual segmentations were in the same range as the overlap between manner. Lastly, while FASTSURF was originally developed for the hip­ ‘FASTSURF with sparse contours’ and the manual references. pocampus in Alzheimer’s patients, (Bartel et al., 2019) our results conclusively demonstrated that this method is also accurate for cross- 3.2.4. Bland-Altman plots and analysis sectional segmentation of the dGM in MS patients. Future studies Fig. 4, and Table 6 show the results of the Bland-Altman scatter plots should investigate whether this technique is suitable for longitudinal and analysis of the combined (left + right) dGM volume measurements: observations as well. FASTSURF minus the combined expert manual labels; with separate la­ Although the dGM manual segmentation protocol showed good bels for MS patients and controls. For all structures, FASTSURF obtained reproducibility within and among raters, certain guidelines might be smaller volumes (mL) than the manual output [mean difference (SD): debated. Considering the thalamus, it was decided that the geniculate caudate: − 0.20 (0.26); putamen: − 0.06 (0.12); thalamus: − 0.14 (0.10), bodies should be included, as they form part of extensions of the all p-values < 0.001]. Visual inspection of the data revealed the same structure itself. Hence, the thalamus comprises mixed WM-GM voxel effects in the MS patients and controls. Because of the small number of intensities, which makes it rather difficult to separate different thalamic subjects (N = 6) in ‘FASTSURF with the novo contours’ we were unable subparts from the background, especially in the presence of atrophy. to perform similar analysis (For scatter plot see supplementary figure 1). (Houtchens et al., 2007; Fischl et al., 2002; Derakhshan et al., 2010) Therefore, in order to minimize error and reduce variability, we decided 4. Discussion to delineate all dGM structures as a whole. Although for the thalamus it is clear that specific nuclei are more sensitive to the MS disease process, In this study we presented a novel protocol with stringent guidelines which was a limitation of this study. Furthermore, the nucleus accum­ for manually tracing the caudate nucleus, putamen and thalamus on 3D bens is difficult to distinguish from adjacent structures due to close T1-weighted MR images, which exhibited excellent reliability in a multi- proximity to the caudate nucleus and putamen. Therefore, we used well- center dataset of MS patients. Moreover, we provided evidence that defined anatomical landmarks to identify the anterior and posterior FASTSURF can be used to generate equally accurate dGM reference limits of the nucleus accumbens in the coronal plane, and the bottom of segmentations as high quality manual tracings of experienced raters. the lateral ventricles as the inferior border of the caudate. (Lucas-Neto The high levels of agreement between the experts’ manual outlines of et al., 2013) the dGM structures (JI ~ 0.75, ICC ~ 0.78) indicate that our segmen­ Interestingly, our data revealed slightly worse estimations of the tation protocol can be used to create dGM reference datasets with suf­ caudate nucleus compared to the putamen and thalamus, both manually ficient levels of accuracy, even in multi-center settings. (Cicchetti, 1994; as well as with FASTSURF. This finding probably results from the Bocchetta et al., 2015) Also, our data revealed that the described method different shapes of the structures. The tail of the caudate is substantially can be used on conventional as well as more advanced 3D T1 images. In more elongated and curved compared to the other structures, and addition, this study demonstrated that FASTSURF can be used to therefore difficult to trace unambiguously. Most importantly, our generate accurate dGM measurements with de novo partial contours as consistent approach allowed the readers to reproduce references with input. This will ultimately reduce the workload and timely effort to great accuracy. Incorporating features from advanced imaging tech­ create sufficient reference datasets for training and validation purposes niques such as diffusion tensor imaging (DTI) or quantitative suscepti­ of algorithms for measuring dGM atrophy in MS. bility mapping (QSM) would probably lead to more refined estimations The output obtained through ‘FASTSURF with sparse contours’ as of these boundaries. (Glaister et al., 2017; Daniel, 2017) However, the well as the segmentations from ‘FASTSURF with de novo contours’ guidelines presented here were strictly based on 3D T1-weighted MRI, showed high levels of agreement with the manual references, both considering that this is the standard imaging contrast in clinical practice volumetrically and spatially, indicating that semi-automation will not for these purposes. While this work focused on standard 3D T1-weighted compromise the quality of the data. However, the Bland-Altman plots imaging sequences that are readily available in a clinical setting, there revealed that overall the volumes of FASTSURF were slightly lower than have also been developments on other MR imaging techniques, such as the manual annotations. This probably resulted from the location of the MPRAGE with additional suppression of WM or GM, or susceptibility- 10 predefined contours, which were distributed equally over the width based contrasts. (Tanner et al., 2012; Kecskemeti et al., 2016) While Table 5 Average volumes (mL) of manual reference, FASTSURF with sparse contours and FASTSURF with de novo contours. And the average Jaccard Index (JI) and Dice Similarity Coefficient (DSC) between manual reference and FASTSURF with sparse contours and with de novo contours. a. N=6 Manual FASTSURF with sparse contours FASTSURF with de novo contours Structure Hemispheres Volume Volume Jaccard Index Dice Similarity Coefficient Volume Jaccard Index Dice Similarity Coefficient Caudate Both 4.64 ± 0.68 4.39 ± 0.51 0.823 ± 0.042 0.900 ± 0.011 4.38 ± 0.64 0.798 ± 0.042 0.923 ± 0.030 Left 4.66 ± 0.73 4.45 ± 0.55 0.822 ± 0.045 0.901 ± 0.011 4.43 ± 0.75 0.797 ± 0.043 0.923 ± 0.035 Right 4.62 ± 0.69 4.34 ± 0.52 0.823 ± 0.042 0.900 ± 0.011 4.33 ± 0.58 0.800 ± 0.041 0.922 ± 0.027 Putamen Both 5.44 ± 1.09 5.38 ± 1.01 0.884 ± 0.035 0.943 ± 0.008 5.41 ± 1.07 0.880 ± 0.039 0.947 ± 0.022 Left 5.54 ± 1.23 5.42 ± 1.08 0.883 ± 0.042 0.944 ± 0.009 5.36 ± 0.81 0.883 ± 0.040 0.944 ± 0.030 Right 5.35 ± 1.05 5.33 ± 1.02 0.886 ± 0.030 0.943 ± 0.007 5.46 ± 1.17 0.877 ± 0.038 0.950 ± 0.022 Thalamus Both 6.83 ± 0.83 6.70 ± 0.79 0.893 ± 0.032 0.938 ± 0.015 6.74 ± 0.79 0.887 ± 0.035 0.953 ± 0.35 Left 6.83 ± 0.84 6.75 ± 0.80 0.885 ± 0.037 0.927 ± 0.012 6.78 ± 0.77 0.877 ± 0.032 0.948 ± 0.037 Right 6.83 ± 0.90 6.64 ± 0.85 0.892 ± 0.033 0.947 ± 0.012 6.69 ± 088 0.901 ± 0.036 0.958 ± 0.035 a mean ± standard deviation. Abbreviations: mL = milliliter, N = number of subjects. 6 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 Fig. 4. Bland Altman scatter plots of the deep grey matter volume measurements of the MS patients and controls for ‘FASTSURF with sparse contours’. The difference of two paired measurements [(FASTSURF–manual)/average] was plotted against the average of the two measurements [(FASTSURF + manual)/2]. Writing - review & editing. Miklos Palotai: Investigation, Visualization. Table 6 Yaou Liu: Investigation, Writing - review & editing. Jorge Simoes: Pairwise Bland-Altman comparisons between ‘FASTSURF with sparse contours’ Investigation, Writing - review & editing. Serena Ruggieri: Investiga­ and combined expert manual labels. tion, Writing - review & editing. Katharina Schregel: Investigation, Structure Subjects µ diff SD SE µ p-Value Writing - review & editing. Stefan Ropele: Conceptualization, Investi­ Caudate Total − 0.20 0.26 0.03 <0.001 gation, Writing - review & editing. Maria A. Rocca: Conceptualization, HC − 0.22 0.25 0.04 <0.001 Investigation, Writing - review & editing. Claudio Gasperini: . Antonio Patients 0.19 0.26 0.03 − <0.001 Gallo: Conceptualization, Investigation, Writing - review & editing. Putamen Total − 0.06 0.12 0.01 <0.001 HC − 0.07 0.13 0.02 0.003 Menno M. Schoonheim: Conceptualization, Investigation, Writing - Patients − 0.06 0.11 0.01 <0.001 review & editing. Michael Amann: Conceptualization, Investigation, Thalamus Total − 0.14 0.10 0.01 <0.001 Writing - review & editing. Marios Yiannakas: Conceptualization, HC − 0.19 0.09 0.01 <0.001 Investigation, Writing - review & editing. Deborah Pareto: Conceptu­ Patients 0.12 0.10 0.01 alization, Investigation, Writing - review & editing. Mike P. Wattjes: − <0.001 Abbreviations: HC = healthy controls; µ diff = mean difference; SD = standard Conceptualization, Investigation, Writing - review & editing. Jaume deviation; SE µ= standard error of µ; p-value in bold represent significant values. Sastre-Garriga: Conceptualization, Investigation, Writing - review & editing. Ludwig Kappos: Conceptualization, Investigation, Writing - those techniques may not be ready yet for widespread clinical applica­ review & editing. Massimo Filippi: Conceptualization, Investigation, tion, they could inform expert raters on the boundaries of the dGM Writing - review & editing. Christian Enzinger: Conceptualization, structures, which could help training of improved automated methods, Investigation, Writing - review & editing. Jette Frederiksen: Concep­ regardless of whether they are applied with or without direct input from tualization, Investigation, Writing - review & editing. Bernard Uitde­ these methods. haag: Conceptualization, Investigation, Supervision, Writing - review & A possible limitation of this study was that we did not compare editing. Charles R.G. Guttmann: Conceptualization, Methodology, FASTSURF with other existing automated segmentation techniques Formal analysis, Investigation, Supervision, Visualization, Writing - However, two other studies that were recently published by our group original draft, Writing - review & editing. Frederik Barkhof: Concep­ already evaluated existing automated segmentations methods against tualization, Methodology, Formal analysis, Investigation, Supervision, manual references, using (partly) the same dataset (de Sitter et al., 2020; Visualization, Writing - original draft, Writing - review & editing. Hugo Burggraaff et al., 2020). Moreover, since this comparison would reveal Vrenken: Conceptualization, Methodology, Formal analysis, Investiga­ any systematic difference between methods, e.g. with respect to tion, Supervision, Visualization, Writing - original draft, Writing - re­ anatomical definitions of the structures of interest, we argued that this view & editing. would not be relevant for the value of creating accurate reference seg­ mentations. Therefore, to maintain the focus of present work, we did not perform such statistical analysis. Another limitation of our study was Declaration of Competing Interest that no statistical analysis was performed between ‘FASTSURF with de novo contours’ and manual annotations due to the limited sample size The authors declare that they have no known competing financial (N = 6). In future work, more manual delineations from trained expert interests or personal relationships that could have appeared to influence raters should be included. the work reported in this paper. To conclude, we suggest that high-quality dGM segmentations can be created based on the proposed manual delineation protocol. Together Acknowledgements with the standardized manual delineation protocol, FASTSURF can serve as an adequate tool to create accurate reference segmentations The members of the MAGNIMS Study Group Steering Committee are: with considerably less effort than full manual outlines. This opens up F. Barkhof and H. Vrenken (Amsterdam UMC, location VUmc, Amster­ possibilities for improving, training and developing algorithms for dam, Netherlands), O. Ciccarelli and T. Yousry (Queen Square MS measuring dGM atrophy in MS and other neurodegenerative diseases. Centre, UCL Institute of Neurology, London, UK), N. De Stefano (Uni­ versity of Siena, Siena, Italy), C. Enzinger (Department of Neurology, CRediT authorship contribution statement Medical University of Graz, Graz, Austria), M. Filippi and M.A. Rocca (San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Alexandra Sitter: Conceptualization, Formal analysis, Investigation, Milan, Italy), C. Gasperini (San Camillo-Forlanini Hospital, Rome, Italy), Methodology, Software, Visualization, Writing - original draft. Jessica L. Kappos (University of Basel, Basel, Switzerland), J. Palace (University Burggraaff: Conceptualization, Formal analysis, Investigation, Meth­ of Oxford Hospitals Trust, Oxford, UK), A. Rovira and J. Sastre-Garriga odology, Visualization, Writing - original draft. Fabian Bartel: Formal (Hospital Universitari Vall d’Hebron, Universitat Autònoma de Barce­ analysis, Investigation, Software, Visualization, Writing - original draft, lona, Barcelona, Spain). The study was funded by the Nauta fonds through a travel grant. The MS Center Amsterdam is supported by the 7 A. de Sitter et al. NeuroImage: Clinical 30 (2021) 102659 Dutch MS Research Foundation through a program grant (current grant Amiri, H., de Sitter, A., Bendfeldt, K., et al., 2018. Urgent challenges in quantification and interpretation of grey matter atrophy in multiple sclerosis. Neuroimage Clin. 19, 18-358f). D.B. is supported by project PI18/00823 from the “Fondo de 466–475. Investigación Sanitaria Carlos III”. F.B. is supported by the National Westlye, L.T., Walhovd, K.B., Dale, A.M., et al., 2009. Increased sensitivity to effects of Institute for Health Research University College London Hospitals normal aging and Alzheimer’s disease on cortical thickness by adjustment for local Biomedical Research Centre. The acquisition of data in London was variability in gray/white contrast: a multi-sample MRI study. Neuroimage 47: 1545- 1557. 2009/06/09. DOI: 10.1016/j.neuroimage.2009.05.084. funded by supported by the National Institute for Health Research Grimaud, J., Lai, M., Thorpe, J., Adeleine, P., Wang, L., Barker, G.J., Plummer, D.L., University College London Hospitals Biomedical Research Centre. Tofts, P.S., McDonald, W.I., Miller, D.H., 1996. Quantification of MRI lesion load in multiple sclerosis: a comparison of three computer-assisted techniques. Magn. Reson. Imaging 14 (5), 495–505. Appendix A. Supplementary data Paty, C.D., Grochowski, E., Palmer, M.R., Oger, J., Kastrukoff, L.F., 1986. Magnetic resonance imaging (MRI) in multiple sclerosis (MS): a serial study in relapsing and Supplementary data to this article can be found online at https://doi. remitting patients with quantitative measurements of lesion size. Neurology 36, 177. Fischl, Bruce, Salat, David H., Busa, Evelina, Albert, Marilyn, Dieterich, Megan, org/10.1016/j.nicl.2021.102659. Haselgrove, Christian, van der Kouwe, Andre, Killiany, Ron, Kennedy, David, Klaveness, Shuna, Montillo, Albert, Makris, Nikos, Rosen, Bruce, Dale, Anders M., References 2002. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33 (3), 341–355. Bartel, F., Vrenken, H., van Herk, M., et al., 2019. FAst Segmentation Through SURface Du Toit, G., Katz, Y., Sasieni, P., Mesher, D., Maleki, S.J., Fisher, H.R., Fox, A.T., Fairing (FASTSURF): A novel semi-automatic hippocampus segmentation method. Turcanu, V., Amir, T., Zadik-Mnuhin, G., Cohen, A., Livne, I., Lack, G., 2008. Early PLoS One 14: e0210641. 2019/01/19. DOI: 10.1371/journal.pone.0210641. consumption of peanuts in infancy is associated with a low prevalence of peanut Rocca, Maria A., Valsasina, Paola, Hulst, Hanneke E., Abdel-Aziz, Khaled, allergy. J. Allergy Clin. Immunol. 122 (5), 984–991. https://doi.org/10.1016/j. Enzinger, Christian, Gallo, Antonio, Pareto, Debora, Riccitelli, Gianna, Muhlert, Nils, jaci.2008.08.039. Ciccarelli, Olga, Barkhof, Frederik, Fazekas, Franz, Tedeschi, Gioacchino, Bagnato, V.S., Kurachi, C., Ferreira, J., et al., 2006. New photonic technologies for the Arévalo, Maria J., Filippi, Massimo, 2014. Functional correlates of cognitive treatment and diagnosis of hepatic diseases: an overview of the experimental work dysfunction in multiple sclerosis: a multicenter fMRI study. Hum. Brain Mapp. 35 performed in collaboration, between Physics Institute of Sao Carlos and Ribeirao (12), 5799–5814. https://doi.org/10.1002/hbm.v35.1210.1002/hbm.22586. Preto Faculty of Medicine of the University of Sao Paulo. Acta Cir. Bras. 21 (Suppl 1), Ropele, Stefan, Kilsdonk, Iris D, Wattjes, Mike P, Langkammer, Christian, de 3–11. Graaf, Wolter L, Frederiksen, Jette L, Larsson, Henrik B, Yiannakas, Marios, Wheeler- Geurts, J.J.G., Pouwels, P.J.W., Uitdehaag, B.M.J., Polman, C.H., Barkhof, F., Kingshott, Claudia AM, Enzinger, Christian, Khalil, Michael, Rocca, Maria A, Castelijns, J.A., 2005. Intracortical lesions in multiple sclerosis: improved detection Sprenger, Till, Amann, Michael, Kappos, Ludwig, Filippi, Massimo, Rovira, Alex, with 3D double inversion-recovery MR imaging. Radiology 236 (1), 254–260. Ciccarelli, Olga, Barkhof, Frederik, Fazekas, Franz, 2014. Determinants of iron https://doi.org/10.1148/radiol.2361040450. accumulation in deep grey matter of multiple sclerosis patients. Mult. Scler. 20 (13), Schoonheim, M.M., Hulst, H.E., Brandt, R.B., Strik, M., Wink, A.M., Uitdehaag, B.M.J., 1692–1698. https://doi.org/10.1177/1352458514531085. Barkhof, F., Geurts, J.J.G., 2015. Thalamus structure and function determine Kouwenhoven, Erik, Giezen, Marina, Struikmans, Henk, 2009. Measuring the similarity severity of cognitive impairment in multiple sclerosis. Neurology 84 (8), 776–783. of target volume delineations independent of the number of observers. Phys. Med. https://doi.org/10.1212/WNL.0000000000001285. Biol. 54 (9), 2863–2873. https://doi.org/10.1088/0031-9155/54/9/018. Bishop, C.A., Newbould, R.D., Lee, J.SZ., Honeyfield, L., Quest, R., Colasanti, A., Ali, R., Shrout, P.E., Fleiss J.L., 1979. Intraclass correlations: uses in assessing rater reliability. Mattoscio, M., Cortese, A., Nicholas, R., Matthews, P.M., Muraro, P.A., Waldman, A. Psychol. Bull. 86: 420-428. 1979/03/01. DOI: 10.1037//0033-2909.86.2.420. D., 2017. Analysis of ageing-associated grey matter volume in patients with multiple Koch G.G. Intraclass correlation coefficient. In: Samuel K. and Norman L.J. Encyclopedia sclerosis shows excess atrophy in subcortical regions. Neuroimage Clin. 13, 9–15. of Statistical Sciences 4. New York, John Wiley & Sons; 213–17. 1982. https://doi.org/10.1016/j.nicl.2016.11.005. Giavarina, D. 2015. Understanding Bland Altman analysis. Biochem. Med. (Zagreb) 25: Bermel, R.A., Innus, M.D., Tjoa, C.W., et al., 2003. Selective caudate atrophy in multiple 141-151. 2015/06/26. DOI: 10.11613/BM.2015.015. sclerosis: a 3D MRI parcellation study. NeuroReport 14, 335–339. https://doi.org/ Altman, J.M., 1983. Measurement in medicine: the analysis of method comparison 10.1097/01.wnr.0000059773.23122.ce. studies. J. Royal Statist. Soc. Ser. D (The Statistician) 32 (3), 307–317. https://doi. Houtchens, M.K., Benedict, R.H.B., Killiany, R., Sharma, J., Jaisani, Z., Singh, B., org/10.2307/2987937. Weinstock-Guttman, B., Guttmann, C.R.G., Bakshi, R., 2007. Thalamic atrophy and Bartko, J.J., 1991. Measurement and reliability: statistical thinking considerations. cognition in multiple sclerosis. Neurology 69 (12), 1213–1223. https://doi.org/ Schizophr. Bull. 17: 483-489. 1991/01/01. DOI: 10.1093/schbul/17.3.483. 10.1212/01.wnl.0000276992.17011.b5. Cicchetti, Domenic V., 1994. Multiple comparison methods: establishing guidelines for Pagani, E., Rocca, M.A., Gallo, A., et al., 2005. Regional brain atrophy evolves differently their valid application in neuropsychological research. J. Clin. Exp. Neuropsychol. in patients with multiple sclerosis according to clinical phenotype. AJNR Am. J. 16 (1), 155–161. https://doi.org/10.1080/01688639408402625. Neuroradiol. 26, 341–346. Bocchetta, Martina, Boccardi, Marina, Ganzola, Rossana, Apostolova, Liana G., Wattjes, M.P., Rovira, A., Miller, D., et al. 2015. Evidence-based guidelines: MAGNIMS Preboske, Gregory, Wolf, Dominik, Ferrari, Clarissa, Pasqualetti, Patrizio, consensus guidelines on the use of MRI in multiple sclerosis–establishing disease Robitaille, Nicolas, Duchesne, Simon, Jack, Clifford R., Frisoni, Giovanni B., prognosis and monitoring patients. Nat. Rev. Neurol. 11: 597-606. 2015/09/16. Bartzokis, George, DeCarli, Charles, deToledo-Morrell, Leyla, Fellgiebel, Andreas, DOI: 10.1038/nrneurol.2015.157. Firbank, Michael, Gerritsen, Lotte, Henneman, Wouter, Killiany, Ronald J., Sastre-Garriga, J., Pareto, D., Battaglini, M., et al. 2020. MAGNIMS consensus Malykhin, Nikolai, Pruessner, Jens C., Soininen, Hilkka, Wang, Lei, Watson, Craig, recommendations on the use of brain and spinal cord atrophy measures in clinical Wolf, Henrike, 2015. Harmonized benchmark labels of the hippocampus on practice. Nat. Rev. Neurol. 16: 171-182. 2020/02/26. DOI: 10.1038/s41582-020- magnetic resonance: the EADC-ADNI project. Alzheimers Dement 11 (2), 151–160. 0314-x. e5. https://doi.org/10.1016/j.jalz.2013.12.019. Popescu, V., Ran, N.C.G., Barkhof, F., Chard, D.T., Wheeler-Kingshott, C.A., Vrenken, H., Derakhshan, M., Caramanos, Z., Giacomini, P.S., et al., 2010. Evaluation of automated 2014. Accurate GM atrophy quantification in MS using lesion-filling with co- techniques for the quantification of grey matter atrophy in patients with multiple registered 2D lesion masks. Neuroimage Clin. 4, 366–373. https://doi.org/10.1016/ sclerosis. Neuroimage 52: 1261-1267. 2010/05/21. DOI: 10.1016/j. j.nicl.2014.01.004. neuroimage.2010.05.029. Gelineau-Morel, R., Tomassini, V., Jenkinson, M., et al. 2012. The effect of hypointense Lucas-Neto, L., Neto, D., Oliveira, E., et al. 2013. Three dimensional anatomy of the white matter lesions on automated gray matter segmentation in multiple sclerosis. human nucleus accumbens. Acta Neurochir. (Wien) 155: 2389-2398. 2013/08/06. Hum. Brain Mapp. 33: 2802-2814. 2011/10/07. DOI: 10.1002/hbm.21402. DOI: 10.1007/s00701-013-1820-z. Meijerman A., Amiri H., Steenwijk M.D., et al. Reproducibility of Deep Gray Matter Glaister, J., Carass, A., NessAiver, T., et al., 2017. Thalamus segmentation using multi- Atrophy Rate Measurement in a Large Multicenter Dataset. AJNR Am J Neuroradiol modal feature classification: validation and pilot study of an age-matched cohort. 2018; 39: 46-53. 2017/12/02. DOI: 10.3174/ajnr.A5459. Neuroimage 158: 430-440. 2017/07/04. DOI: 10.1016/j.neuroimage.2017.06.047. Popescu, V., Schoonheim, M.M., Versteeg, A., Chaturvedi, N., Jonker, M., Xavier de Feng, L., Benkert, T., Block, K.T., et al., 2017. Compressed sensing for body MRI. J. Magn. Menezes, R., Gallindo Garre, F., Uitdehaag, B.M.J., Barkhof, F., Vrenken, H., Reson. Imaging 45: 966-987. 2016/12/17. DOI: 10.1002/jmri.25547. Derfuss, T., 2016. Grey matter atrophy in multiple sclerosis: clinical interpretation Tanner, M., Gambarota, G., Kober, T., et al., 2012. Fluid and white matter suppression depends on choice of analysis method. PLoS One 11 (1), e0143942. https://doi.org/ with the MP2RAGE sequence. J. Magn. Reson. Imaging 35: 1063-1070. 2011/12/16. 10.1371/journal.pone.0143942. DOI: 10.1002/jmri.23532. Amiri, H., de Sitter, A., Bendfeldt, K., et al., 2018. Urgent challenges in quantification Kecskemeti, S., Samsonov, A., Hurley, S.A., et al., 2016. MPnRAGE: A technique to and interpretation of brain grey matter atrophy in individual MS patients using MRI. simultaneously acquire hundreds of differently contrasted MPRAGE images with Neuroimage Clin. 19: 466-475. 2018/07/10. DOI: 10.1016/j.nicl.2018.04.023. applications to quantitative T1 mapping. Magn. Reson. Med. 75: 1040-1053. 2015/ de Sitter, A., Verhoeven, T., Burggraaff, J., Liu, Y., Simoes, J., Ruggieri, S., Palotai, M., 04/18. DOI: 10.1002/mrm.25674. Brouwer, I., Versteeg, A., Wottschel, V., Ropele, S., Rocca, M.A., Gasperini, C., Burggraaff, J., Liu, Y., Prieto, J.C., et al., 2020. Manual and automated tissue Gallo, A., Yiannakas, M.C., Rovira, A., Enzinger, C., Filippi, M., De Stefano, N., segmentation confirm the impact of thalamus atrophy on cognition in multiple Kappos, L., Frederiksen, J.L., Uitdehaag, B.M.J., Barkhof, F., Guttmann, C.R.G., sclerosis: a multicenter study. Neuroimage Clin. 29 102549. 2021/01/06. DOI: Vrenken, H., 2020. Reduced accuracy of MRI deep grey matter segmentation in 10.1016/j.nicl.2020.102549. multiple sclerosis: An evaluation of four automated methods against manual reference segmentations in a multi-center cohort. J. Neurol. 267 (12), 3541–3554. 8