REVIEW OPEN ACCESS Opportunities for Understanding MS Mechanisms and Progression With MRI Using Large-Scale Data Sharing and Artificial Intelligence Hugo Vrenken, PhD, Mark Jenkinson, BSc, BE, DPhil, Dzung L. Pham, PhD, Charles R.G. Guttmann, MD, Correspondence Deborah Pareto, PhD, Michel Paardekooper, PhD, Alexandra de Sitter, PhD, Maria A. Rocca, MD, Dr. Vrenken Viktor Wottschel, PhD, M. Jorge Cardoso, PhD, and Frederik Barkhof, MD, PhD, h.vrenken@ on behalf of the MAGNIMS Study Group amsterdamumc.nl ® Neurology 2021;97:989-999. doi:10.1212/WNL.0000000000012884 Abstract RELATED ARTICLE Editorial Patients with multiple sclerosis (MS) have heterogeneous clinical presentations, symptoms, MRI, Big Data, and and progression over time, making MS difficult to assess and comprehend in vivo. The com- Artificial Intelligence: bination of large-scale data sharing and artificial intelligence creates new opportunities for Rewards vs Risks monitoring and understanding MS using MRI. First, development of validated MS-specific Page 975 image analysis methods can be boosted by verified reference, test, and benchmark imaging data. Using detailed expert annotations, artificial intelligence algorithms can be trained on such MS- specific data. Second, understanding disease processes could be greatly advanced through shared data of large MS cohorts with clinical, demographic, and treatment information. Rel- evant patterns in such data that may be imperceptible to a human observer could be detected through artificial intelligence techniques. This applies from image analysis (lesions, atrophy, or functional network changes) to large multidomain datasets (imaging, cognition, clinical dis- ability, genetics). After reviewing data sharing and artificial intelligence, we highlight 3 areas that offer strong opportunities for making advances in the next few years: crowdsourcing, personal data protection, and organized analysis challenges. Difficulties as well as specific recommen- dations to overcome them are discussed, in order to best leverage data sharing and artificial intelligence to improve image analysis, imaging, and the understanding of MS. From the MS Center Amsterdam (H.V., A.d.S., V.W.), Amsterdam Neuroscience, Department of Radiology and Nuclear Medicine, Amsterdam UMC (M.P.), the Netherlands; Wellcome Centre for Integrative Neuroimaging (WIN), FMRIB (M.J.), Nuffield Department of Clinical Neurosciences (NDCN), University of Oxford, UK; Human Imaging and Image Processing Core (D.L.P.), Center for Neuroscience and Regenerative Medicine, The Henry M. Jackson Foundation, Bethesda, MD; Center for Neurological Imaging (C.R.G.G.), Department of Radiology, Brigham and Women’s Hospital, Boston, MA; Section of Neuroradiology (Department of Radiology) (D.P.), Vall d’Hebron University Hospital and Research Institute (VHIR), Autonomous University Barcelona, Spain; Neuroimaging Research Unit (M.A.R.), Institute of Experimental Neurology, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy; AMIGO (M.J.C.), School of Biomedical Engineering and Imaging Sciences, King’s College London; and Institutes of Neurology & Healthcare Engineering (F.B.), UCL London, UK. Go to Neurology.org/N for full disclosures. Funding information and disclosures deemed relevant by the authors, if any, are provided at the end of the article. MAGNIMS Study Group coinvestigators are listed in the appendix at the end of the article. The Article Processing Charge was funded by Vrije Universiteit, Amsterdam, the Netherlands. This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0 (CC BY), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Copyright © 2021 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the American Academy of Neurology. 989 Glossary GDPR = General Data Protection Regulation; INNI = Italian Neuroimaging Network Initiative; MS = multiple sclerosis; QA = quality assessment; QC = quality control; WM = white matter. Multiple sclerosis (MS) is highly heterogeneous across pa- Data Sharing in Neuro-MRI: Non-MS tients in terms of symptoms, sites of damage, degree of re- Data sharing in MRI is becoming increasingly prevalent, large- covery, and development of the disease across time. Relevant scale, and open in the neuroimaging community. Table 1 lists patterns may be imperceptible to a human observer, but by several prominent examples, including the Alzheimer’s Dis- analyzing large amounts of imaging data with sophisticated ease Neuroimaging Initiative (ADNI), which in the neuro- artificial intelligence techniques, urgently required advances imaging field is a template for data acquisition and fostering of in understanding MS disease pathologic heterogeneity may be methodologic developments. These datasets are associated made. with a range of access policies (from free, unrestricted downloads to collaboration-only agreements) and cover var- Furthermore, to track disease progression in individual pa- ious sizes, demographics, and pathologies. They provide ac- tients with MS, MRI markers are needed. This could benefit cess to large, diverse groups of subjects, including rare from MS-specific image analysis methods, because existing diseases, and a wider range of disease stages (including pro- generalized methods tend to exhibit poorer performance in dromal cases) than is possible from single studies. In addition, patients with MS,1 as has been demonstrated quantitatively the increasingly large numbers provide greater statistical for segmentation of deep gray matter structures.2 Large power and the opportunity to apply state-of-the-art deep amounts of MS imaging data, with expert annotations as ap- learning techniques. They also allow common standards to be propriate, can be used to train and validate more accurate applied in the evaluation of methodologic tools, as pioneered measurement and analysis tools specifically for MS. by the MICCAI challenges.e67 As such, they provide the community with fair and open comparisons of methods, a Against this background, we review the possibilities of data richer set of data on which to test hypotheses, and greater sharing and artificial intelligence for improved applications of ability for assessing reliability and repeatability. There are also MRI to study MS, addressing both the need to understand MS benefits for those involved in creating and managing such disease processes and the need for MS-dedicated quantitative datasets, because the process of designing, piloting, and pre- measurement and analysis techniques for MRI assessments. processing provides impetus for novel developments in ac- We first survey relevant existing efforts regarding data sharing quisition and analysis, demonstrated by state-of-the-art and artificial intelligence, and then highlight 3 areas of interest methodologies developed within the Human Connectome in bringing the field forward: crowdsourcing, personal data Project. In addition, there are benefits in visibility, engage- protection, and organized analysis challenges (see the Figure ment, and publications. Challenges still exist (e.g., in- for the methods used to create this article). Specific recom- formation technology infrastructure, access policies, ethics mendations aim to achieve the best outcomes for patients policies) but many such datasets are already accessible, with a with MS. range of solutions to these problems, thus offering options for the creation of new datasets focusing on MS. The datasets listed also highlight that standardized magnetic resonance Data Sharing acquisition protocols can harmonize data only to a certain extent. Therefore, alternative approaches such as synthetic Data Sharing in MS: Nonimaging MRI should also be investigated (additional references in The multicenter collection of MS clinical data provides valuable eAppendix 1). information on disease prevalence, current treatment patterns, and general distribution of patients’ outcomes. Thus, clinical registries including data from several MS centers have been Data Sharing of MRI in MS strongly promoted in the past decades. National and regional MRI is one of the most important tools for diagnosing and MS registries exist in most countries, especially in Europe3 and monitoring MS.10 However, MRI data collected by clinical North America.4 Data collected by national initiatives have MS registries usually include only conventional measures or often been included in computerized platforms, such as the metadata regarding fulfilment of diagnostic criteria.5,6 Recent European Register for Multiple Sclerosis (EUReMS)5 or the collaborations (for example, the publicly funded German MSBase.6 Collaborative research studies have used these data Competence Network Multiple Sclerosise68 or the privately to define the value of prognostic indicators in different patient funded MS PATHS11) promoted the use of relatively stan- populations,7 to investigate the influence of demographic and dard conventional MRI protocols (typical T1-weighted, geographic factors on MS clinical course,8 and to evaluate the proton density–weighted, and T2-weighted or fluid- comparative efficacy of different drugs9 (additional references attenuated inversion recovery images), but did not include in eAppendix 1, https://doi.org/10.5061/dryad.2fqz612p9). advanced MRI techniques (such as quantitative mapping 990 Neurology | Volume 97, Number 21 | November 23, 2021 Neurology.org/N Recommendations on MRI Data Sharing in MS Figure Methods Used to Create the Article  Clearly define variables to be shared: this avoids ambiguity and heterogeneity at a later stage.  Set up proper QA and quality control (QC) procedures to ensure compliance with minimum standards: prefer- ably quantitative and automated, such procedures guarantee the integrity of the included data.  Implement clear policies and procedures on how access to data can be obtained.  Create a flexible data sharing system, permitting a manifold use of collected data: by choosing maximally permissive data licenses (within legal and institutional boundaries), combined with clear data storage organization, database management, and flexible access choices, data can be flexibly and easily selected, accessed, and used for a variety of purposes. techniques of tissue properties, tractography, spectroscopy, or functional MRI). Table 2 lists MS Registries identified from public sources that collect MRI information. Artificial Intelligence As an example, the Italian Neuroimaging Network Initiative Artificial Intelligence in Medical Image Analysis (INNI)12,e69 has recently been established among 4 sites Beyond MS leading MRI research in MS in Italy, with the support of the Artificial intelligence can be roughly divided into classical ma- Italian MS Society. INNI’s major goal is to determine and chine learning techniques such as support vector machines and validate novel MRI biomarkers, including biomarkers based (newer) deep learning techniques based on convolutional on more advanced, nonconventional imaging techniques, to neural networks. Classical machine learning approaches typi- be utilized as predictors or outcomes in future MS studies. cally make predictions using classifiers trained not directly on INNI aims also to standardize MRI procedures of acquisition images, but on features extracted from images.13 While this can and analysis in MS at a national level. be advantageous, it precludes the discovery of features not perceptible to or appreciated by the human observer. Deep A large population of patients with MS and healthy controls learning, when applied to classification or segmentation of (more than 1,800 participants and more than 3,000 MRI images,14 instead analyzes image data directly, without prior examinations) has been collected in the INNI platform so far. feature selection. This has given rise to excellent classifier Although MRI data had to meet some minimum require- performance in a range of medical imaging applications.13 ments in order to be included,12 a full standardization of However, as shown for example by Ghafoorian et al.,15 at least acquisition protocols was not requested from sites, at least in given the current limitations regarding sizes of available data- the first phase of the project. sets and networks, performance may be further improved by incorporating well-chosen features extracted from the images The main challenges faced at the beginning of the INNI ini- using domain knowledge and classical image analysis tech- tiative were related to ethical approvals, to the creation of the niques. Specifically, in their work, they incorporate measures online platform, to ensure proper handling of anonymity, and reflecting location in the brain to improve segmentation of age- to define guidelines to regulate database access levels and their related white matter (WM) hyperintensities.15 implementation as access procedures.12 Conversely, most of the subsequent challenges are related to the quality assess- Artificial Intelligence in Imaging of MS ment (QA) of the data collected, which will now be used for Existing studies applying artificial intelligence in MS imaging different research projects at the 4 promoting sites. Systematic can generally be divided into descriptive and predictive ex- QA (on patient positioning, image inhomogeneity, distor- periments. Descriptive studies use cross-sectional datasets in tions and artefacts, and measurement of contrast-to-noise order to segment from MRI WM lesions16 or specifically ratio) has been established to verify source data and ensure contrast-enhancing lesions,17 identify imaging patterns based maintenance of high quality. QA results will be used to pro- on MS phenotype or clinical or cognitive disease severity,18 or pose effective guidelines on acquisition protocols and scan- perform an automated diagnosis using uni- or multimodal ning options to improve harmonization of MRI data. Basic information.19 Predictive studies, on the other hand, detect analysis (e.g., T2-hyperintense lesion segmentation, T1- patterns in baseline data that allow for predicting future dis- hypointense lesion refilling, minimal preprocessing on ease outcome or severity by incorporating clinical follow-up diffusion-weighted MRI and resting-state fMRI scans) that information.20,21 The majority of studies to date used classic may be shared in the INNI platform to harmonize future machine learning techniques such as support vector machines projects is also being performed in a centralized manner. or random forests, where features have to be defined and Neurology.org/N Neurology | Volume 97, Number 21 | November 23, 2021 991 Table 1 Small Selection of Open Neuroimaging Datasets Currently Available With Partial Information on the Characteristics of Each, Highlighting the Breadth and Diversity Selected additional Web site/ Dataset name Age range, y Demographic Participants, n Modalities data/info Started references ADNI: Alzheimer’s 55–90 AD + MCI + HC >2,000 sMRI + dMRI + fMRI Genetics + clinical + neuropsych 2004 e83, e84 Disease Neuroimaging + PET tests + biosamples Initiative Harvard Aging Brain 65+ Normal aging 284 sMRI + fMRI + dMRI Longitudinal 2009 e85, e86 Study + PET DIAN: Dominantly 18+ Autosomal 430 sMRI + dMRI + fMRI Repeat visits + biosamples 2008 e87, e88 Inherited Alzheimer dominant AD + PET Network OpenNeuro (formerly Any Any >1,000 Mainly MRI, but Wide variety of different studies 2013 e76, e89 OpenfMRI) also with MEG + EEG + PET OASIS: Open Access 18–96 HC + MCI + AD 1,664 sMRI + fMRI + dMRI >1,000 with longitudinal data 2007 e90, e91 Series of Imaging + PET Studies PPMI: 33–85 PD + HC ;1,000 sMRI + dMRI + fMRI Biosamples 2010 e92, e93 Parkinson Progression + PET + SPECT Marker Initiative HCP: Human 22–35 Healthy 1,200 sMRI + dMRI + fMRI Siblings and twins; genetic + 2010 e94, e15 Connectome Project + some MEG lifestyle data dHCP: Developing HCP 20–44 wk Fetal + 1,500 sMRI + dMRI + fMRI 2013 e95, e96 neonatal (healthy and otherwise) UK Biobank 48–77 Any 100,000 sMRI + dMRI + fMRI Genetic, lifestyle, and clinical 2012 e97, e98 (currently data + nonneurologic data 25,000) Neurovault Any Any >1,000 studies fMRI + sMRI + PET Statistical results from 2014 e99, e100 publications MICCAI Challenges Any Any >160 separate Any medical Each challenge comes with a 2007 e101 (Many challenges imaging modality modest dataset and some papers) and any body part training labels/ground truth Abbreviations: AD = Alzheimer disease; dMRI = diffusion MRI; fMRI = functional MRI; HC = healthy control; MCI = mild cognitive impairment; MEG = magnetoencephalography; PD = Parkinson disease; sMRI = structural MRI. extracted from the data a priori, while more recent studies also solely on the use of local intensity patterns and contextual use deep learning methods, allowing automated detection of features to guide the image analysis process. They lack high- relevant features in the data. Deep learning has now been used level abstract thinking and have limited understanding of not only to segment WM lesions22-24 or their enhancing human anatomy and physiology. Learning from relatively subset,17 but also to quantify lesion changes,25,26 detect the small and noisy datasets, learning systems are commonly central vein sign,27 classify different lesion types based on unable to extrapolate and handle uncertain situations. Fur- diffusion basis spectrum imaging,28 predict gadolinium en- thermore, in precision medicine applications, fully automatic, hancement from other image types,29 perform MRI-based robust, and quick measurements are required for every sub- diagnosis,30,31 segment and analyze nonlesion structures,32,33 ject. Three important categories of current limitations of analyze myelin water fraction34 or quantitative susceptibility learning systems are inputs, labels, and uncertainty and mapping data,35 synthesize absent image types,36 perform confidence. automatic QC,37 improve image quality,38 or correct intensity differences between scanners39 (additional references in Inputs eAppendix 1). The main limitation of many machine learning models is the strong dependence on (good) training data. While human Challenges of Artificial Intelligence in raters may intuitively extrapolate from a few examples to new MS Imaging cases that may be very different, a supervised (deep) learning Although showing impressive performance, state-of-the-art model has to be fed with sufficient examples to cover the deep learning methods (convolutional neural networks) rely whole range of heterogeneity in the population, disease, and 992 Neurology | Volume 97, Number 21 | November 23, 2021 Neurology.org/N not fully removed. Therefore, pooling data from different Table 2 Multiple Sclerosis Registries That Collect MRI sources is advantageous, but this introduces new challenges Information due to differences between centers, scanners, and scanning Registry Start date Reference protocols, which require standardizing and postprocessing. National Registry of Croatia 2007 5 The majority of published machine learning studies in MS National Registry of Denmark 1948/1996 5 used research data rather than clinical data, which has limi- National Registry of France 1976 5 tations: patients were filtered through inclusion criteria; the number of subjects and scans is limited by the obtained National Registry of Germany 2001 5 funding; and patients with more severe disease are more likely National Registry of Greece 2011 5 to drop out, biasing data towards more benign cases. Clinical data are more representative of the general (disease) pop- National Registry of Italy 2001 5 ulation but are typically more heterogeneous and require National Registry of Norway 1998 5 additional patient consent. National Registry of Russia 2006 5 Labels National Registry of Spain (Catalonia) 2009 5 Training with high-quality labels is crucial to attaining good National Registry of Sweden 1997 5 performance of machine learning systems. Labels can be the person’s diagnosis or other overall features, or typically in National Registry of Switzerland 2012 5 image analysis tasks, manual outlines of anatomical structures National Registry of United Kingdom 2009 5 or pathologic entities like MS lesions. Distinguishing MS le- MSBase 2004 6 sions in the WM from other WM lesions and from normal- appearing WM requires skill and expertise. The variability of Expression, Proteomics, Imaging and Clinical 2004 e105 (EPIC) labeling protocols and inter- and intrarater variability in- troduce errors when training machine learning systems. Such Kompetenznetz Multiple Sklerose (KKNMS) 2005 e68 errors degrade the performance of learning systems and limit Comprehensive Longitudinal Investigation 2006 e104 to what extent that performance can be validated. Those er- of MS (CLIMB) rors could be quantified by expanding training sets, based on Italian Neuroimaging Network Initiative 2011 12 common protocols applied by larger numbers of raters, and (INNI) subsequently overcome by modeling or machine learning OPTIMISE-MS 2017 e103 approaches. North American Registry for Care and 2017 e102 Uncertainty and Confidence Research in MS (NARCRMS) Algorithms commonly solve a categorical hard problem, but clinical decisions are rarely categorical, and involve intrinsic uncertainty. The introduction of biomarker- and subject- scanning parameters. Differences in imaging devices, acqui- specific error bars, and the development of novel ways to sition parameters, tissue contrasts, artefacts, and noise pat- convey and introduce this information into the clinical terns can degrade algorithm performance if not handled workflow, will present challenges to clinical adoption. Recent appropriately. To overcome between-scanner or between- work addresses that uncertainty for MS lesion detection and acquisition image differences, many approaches for segmentation.43 Other areas of medicine with inherently postprocessing-based data harmonization have been pro- uncertain predictions may suggest ways of introducing this posed, including traveling phantoms,40 which requires phys- uncertainty into the clinical workflow in the context of MS ical travel of objects or subjects,39 limiting scalability; data imaging (additional references in eAppendix 1). augmentation,41 which requires sufficiently accurate signal simulating models; and domain adaptation,42 which has Recommendations on Machine Learning in shown promising results. Ultimately, the most robust results MS Imaging may come from combinations of the above, together with  Compile large, annotated datasets for training. To obtain basic steps such as intensity normalization (additional refer- sufficient amounts of training data, large-scale data ences in eAppendix 1) sharing of MS imaging data is required, both for homogeneous datasets (for generating new knowledge), Given the relatively low prevalence of MS, lack of training data and heterogeneous datasets (for deriving more generaliz- is an issue of particular relevance here. Insufficient variability able classifiers). in training data can lead to overfitted models that do not  Create methods that are robust to data variability. perform well on new data, and single-center datasets seldom Harmonize data using both classical and machine exceed a few hundred MS cases. This effect can be reduced learning techniques to improve robustness to unseen with regularization, augmentation, and cross-validation, but datasets. Neurology.org/N Neurology | Volume 97, Number 21 | November 23, 2021 993  Include nonresearch data in training. Train machine enjoy participating in crowdsourced research and remain learning methods also on data acquired in a real-life committed to finishing their contribution. clinical setting, to increase robustness to heterogeneity and to improve applicability in the clinical population. Potential for Crowdsourcing in MS Imaging  Create high-quality labels. Validating algorithms for For processing large amounts of image data on a regular clinical use will require large multicenter labeling efforts basis, as in a clinical setting, automation of analysis methods yielding, depending on the aims, consensus-based is key. The training that goes into such automated methods, “ground truth” labels or collections of individual raters’ whether based on deep learning or using other approaches, labels. plays a large role in their performance. Ideally, reference  Allow more subtle information in labels than global yes/ labels, for example, of specific imaging features such as MS no answers. The use of soft labels (e.g., image-wide lesions or particular anatomical structures, would be gener- disease classification) to model the intrinsic anatomical ated by intensively trained expert raters. However, if this is and pathologic variability should also be investigated. not possible, for example due to the associated costs,  Incorporate the uncertainty of classifier predictions. crowdsourcing such training labels could provide a realistic Algorithms should learn the intrinsic uncertainty and alternative, if properly used. A potential issue concerns the confidence of every decision they make. Diagnostic and quality of crowdsourced image annotations. For the example prognostic guidelines should be modified to enable of image segmentation, this has been addressed. Specifically, clinical usage of biomarker- and subject-specific un- Bogovic et al.45 demonstrated that for cerebellar parcella- certainty metrics. tion, it is possible to achieve high-quality labels from a group of nonexpert raters. Opportunity 1: Crowdsourcing This suggests that, provided that training and QA procedures are adequately employed, a large group of nonexpert volun- Crowdsourcing in Research teers could create reference labels on a large enough dataset to Crowdsourcing, not to be confused with crowdfunding, refers train deep learning or other methods robust to data variability. to people donating their time and skills to complete certain Nevertheless, as the task becomes more complex, the degree tasks. In the context of scientific research, it is sometimes of communication required between the participants in order called “citizen science.” Its premise is that there are many to achieve an adequate performance is likely to increase, with enthusiastic members of the general public who are willing to the risk of the efforts becoming an outsourcing initiative in- donate some of their time to science. By making it easy for stead of a crowdsourcing one. Thus, the project to be carried them to contribute, the scientific community can reward their out should be clearly defined in terms of tasks and expecta- enthusiasm and willingness by letting them help the field tions, with the inclusion of tutorials and support by an ex- forward. Successful projects have been conducted this way, perienced professional in the field. including examples in astronomy on differentiating different galaxy types (Galaxy Zooe70), in organic chemistry on the Besides providing training labels for image segmentation, topic of protein folding (Foldit, a game with tens of thousands crowdsourcing may also assist in other tasks such as (pro- of playerse71), in biology on identifying bat calls (Bat Detec- viding training labels for) image artifact detection, QC, or tivee72), and in paleontology on dinosaur limb bone mea- disease classification. Especially niche applications such as MS surements (Open Dinosaur Projecte73). The potential for imaging, where the costs of expert training labels can be brain imaging applications has been noted, and a successful prohibitive, can provide a “sweet spot” where crowdsourcing approach to interface-building, data management, and anal- can make a crucial contribution that advances the field. A first ysis has been described. SPINE,e74 Open Neuroimaging such approach has been proposed recently.46 Laboratory,e75 and OpenNeuroe76 are examples of Web-based infrastructures for crowdsourced brain imaging research. Recommendations for Crowdsourcing in MS Imaging While the potential benefits of crowdsourcing to the re-  Ensure high-quality instruction of volunteers. In order to searchers are clear, the benefits to the participants (the help the crowd participate effectively, especially for “crowd”) may be less obvious. There is the potential gratifi- longer and more complex tasks, comprehensive tutorials cation of contributing to science, and in the case of MS re- are essential. search, these volunteers may be people with an interest in brain  Define clear tasks and expectations. In order to allow imaging or neuroscience, or they may know someone who has volunteers to make contributions to research, their tasks MS and want to help develop a solution. Furthermore, well- should be clearly defined, and generally limited in scope designed crowdsourcing activities can also carry the reward of and time investment. being entertaining to perform. The field of “gamification” is a  Enforce rigorous QC. In order to ensure high quality of rapidly developing area of research and development in its own the crowdsourced contributions, QC procedures such as right that has already been applied in the radiologic field,44 repeatability and agreement with experts on selected which creates important opportunities for helping volunteers samples are essential. 994 Neurology | Volume 97, Number 21 | November 23, 2021 Neurology.org/N Opportunity 2: Solutions to Personal different sources by many companies and organizations poses another hurdle to achieving full anonymization. New tech- Data Protection and nologies, notably artificial intelligence techniques, allow, for Consent Requirements example, reconstruction of faces from low-resolution pic- tures.48 In brain imaging, structural images allow 3D face Personal Data Protection and Consent in the reconstructions, possibly enabling identification. Face re- GDPR Framework moval and face scrambling49,50 do not fully solve this, as these When sharing data, protecting personal data is a crucial procedures may affect subsequent image analysis, for example, guarantee to the participants. Because the MAGNIMS Study radiotherapeutic dose distributions or EEG signals, or further Group is a European collaboration, the expertise available on brain image analyses.51 Removed faces can be (partially) data protection laws in other jurisdictions is limited and we reconstructed, and the structure of each brain may soon be therefore focus here mainly on the situation in the European enough to identify the person.52 Union. Personal data protection is required by law in the European Union, with very specific rules set out in the Gen- It may therefore be difficult to reach full anonymization at all. eral Data Protection Regulation (GDPRe77). While the spe- Hence, a second option may be more viable: to request, cific legal requirements vary, concerns about preserving upfront, informed consent of the persons to share their data in confidentiality of data of participants exist around the globe, an identifiable or not directly identifiable way. This informed and different legal frameworks to address these concerns exist consent should comply with national laws including those in different countries. Differences between the GDPR and the based on GDPR, where applicable, as well as indicate the US Health Insurance Portability and Accountability Act various options for planned or as yet unforeseen data sharing, (HIPAA) framework include the more limited scope of the such as with researchers in countries outside the European latter, and have been discussed in detail elsewhere.47 Personal Union, or perhaps with members of the general public data is understood as any information relating to an identified through crowdsourcing initiatives as discussed above. In ad- or identifiable natural person. An identifiable natural person is dition, to further ensure the protection of personal data, an one who can be identified, either directly or indirectly. To agreement on the use of the personal data must be made determine whether someone is identifiable, all the means between the institutions or organizations sharing the data. An reasonably likely to be used have to be taken into account, and increasingly important means of generating large cohorts is by for each of these, the costs and the amount of time required sharing the data across large groups of many different centers for identification, the available technology at the time of the from many countries, which can lead to additional legal un- processing, and expected technological developments have to certainties about personal data protection, data ownership, be assessed. This is important because GDPR holds the and data usage. The extreme case of this informed consent controller, that is, the researcher’s organization, accountable. approach, where legally allowed, would be to ask the partici- Security of personal data must be demonstrated through the pants to consent to sharing their data without restricting that existence of both technical and organizational measures.e77 sharing to specific parties or applications. GDPR places the persons whose data it concerns (referred to as “data subjects”) in full control of what happens to their A third way to share research data is by using an infrastructure data. If personal data are meant to be shared or the possibility that allows researchers to analyze the data remotely.53 Such of identification cannot be excluded, written informed con- “trusted data ecosystems” at their core are similar to a fed- sent must be obtained from all participants for data sharing, erated database but are much more comprehensive and en- including whether data will be shared with countries where compass not just data management but all features necessary EU regulations do not apply. Furthermore, procedures must to perform analyses on the data including the computing be in place for removal of data when participants ask for that infrastructure and audit trail. An example of such an in- removal. The data protection authorities consider coded or frastructure, currently under development in the Netherlands, pseudonymized data as personal data. If personal data are to is the Health-RI infrastructure.e78 The approach taken by be shared, written data protection agreements are necessary to Health-RI is that the personal data are on the inside and the demonstrate compliance with the GDPR. platform performs the analyses, so researchers only receive the outcome measures but have no access to the actual data. Challenges Related to Personal Data Examples focused on federated deep learning, in which model Protection and Consent parameters but not data are transferred between sites, are Ensuring protection of personal data while providing ade- described by Chang et al.54 and Remedios et al.55 A limitation quate access for research purposes is challenging. Full ano- of such a federated approach is that the careful scrutiny of nymization, meaning that the person cannot be identified analysis pipeline success and inspection of intermediate re- from the data at all, may be difficult to achieve in several cases. sults is less directly feasible than in the more standard data In “extreme” groups (e.g., rare diseases, extremely tall per- sharing approach, which would hamper not just any particular sons), some basic information accompanying the imaging research project but also use of those data for further meth- data may help reveal the identity of the person. The increasing odologic improvements. An advantage is that their compre- accumulation of big data on people’s behavior from many hensive approach, including technological, legal, and business Neurology.org/N Neurology | Volume 97, Number 21 | November 23, 2021 995 layers, ensures compliance with regulations, governance, legal their approach to the test dataset and submit results for evaluation. issues for data sharing, security issues, and accountability. Finally, the teams and organizers discuss performance of the different algorithms, as well as the evaluation and related issues. The trade-off between protecting privacy and allowing access remains the main challenge that needs to be addressed There have been 3 segmentation challenges focused specifically properly. In this context, “differential privacy” may offer so- on the segmentation of MS brain lesions: the 2008 MICCAI lutions. Limiting how much algorithms can learn from each Challenge,57,e106 the 2015 ISBI Longitudinal Challenge,58 and data point prevents algorithms from learning enough to the 2016 MICCAI Challenge.59 These provide clear examples identify individuals, yet allows them to learn the relevant in- of what can be achieved through this kind of approach. An formation at the population level.56 enduring key benefit of these challenges, beyond the papers, is that the organizers have continued to make the data available Recommendations Related to Personal Data after the meeting, and have set up Web-based systems for Protection and Consent continually benchmarking new algorithms. In this way, all 3  Protect personal data. Implement technical, legal, and challenges continue to actively make an impact, aiding software organizational guarantees for protecting personal data. For developers in developing improved methods.e80-e82 MRI, these include DICOM anonymization and face removal. These advances notwithstanding, challenges thus far have only  Always request consent for data sharing. Maximize released portions of the full datasets for training, with the possibilities for reuse by requesting participant consent testing data reserved by the challenge organizers for algorithm for subsequent sharing and aiming for a broad scope of evaluation. Furthermore, data use licenses have been restricted future projects. Invest in standardization of the necessary to research or educational use. Those previous MS challenges data protection agreements. have also focused rather narrowly on different aspects of MS  Invest in developing optimized infrastructure. Investi- WM lesion segmentation. For example, the 2015 challenge gate how the strengths of Trusted Data Ecosystems can focused on longitudinal data,58 while the 2016 data focused on be combined with access to raw data and intermediate multicentric data, including those acquired at different field results. strengths.59 Continued organization of challenges could target benchmarking algorithms for applications directly relevant to patient care, such as clinical trials or patient monitoring. In- Opportunity 3: Organized stead of metrics based on lesion segmentation accuracy, al- Analysis Challenges gorithms could be evaluated based on predicting the efficacy of therapies or clinical measures. Besides white matter lesion Organized Analysis Challenges as a Tool for segmentation, a number of other promising imaging bio- Accelerating Methodologic Developments markers could be tested: cortical lesions, cortical gray matter With image analysis and machine learning algorithms ad- measurements, and thalamic volumes have all been found to vancing at a rapid pace, there exists a need to understand the be promising predictors of disease progression.60 An MS da- performance and limitations of state-of-the-art approaches. tabase with whole-brain labels, currently not available, would Evaluating the performance of an automated algorithm, such aid training and validation of algorithms to more accurately as lesion segmentation, is a fundamental part of methods extract such biomarkers. Other grand challenges could ex- development that can require significant resources. Grand amine MRI of spinal cord morphology and pathology and challenges are organized, competitive events that provide data characterization of retinal morphology using optical co- to be analyzed, an analysis objective, ground truth data, and herence tomography. There is ample opportunity for chal- evaluation metrics for achieving the objective.e79 They provide lenges to contribute to further improvements in methods for a means to compare the performance of multiple algorithms, studying MS, as well as proof from previous years that chal- to which a single laboratory would not typically have access. lenges can be a successful approach. Furthermore, by providing these critical resources, research laboratories may participate in the challenge that might not Recommendations on Organized Analysis otherwise engage in MS research. Challenges for MS Image Analysis  Include additional aspects of MS image analysis other than The format of a grand challenge typically involves several key WM lesions, such as cortical lesions and measures of brain steps. Participating teams are first provided with a training dataset volume. that includes both imaging data and the ground truth. For the  Evaluate algorithms also against clinical outcomes, instead aforementioned lesion segmentation challenges, this consisted of of just against imaging data. multicontrast MRI data from multiple patients and a set of manual  Ensure challenge datasets contain large numbers of images lesion delineations. The training data allow the teams to optimize and labels, to improve robustness and generalizability. performance of their algorithms and achieve results consistent  Reduce restrictions on challenge data, to allow more with the ground truth. Next, teams are provided with a test dataset diverse applications and to build more expansive data that includes imaging data but no ground truth. The teams apply resources for algorithm development and evaluation. 996 Neurology | Volume 97, Number 21 | November 23, 2021 Neurology.org/N Discussion plus royalties from the licensing of FSL to commercial en- tities, and consultancy from Oxford Brain Diagnostics. D. L. To maximize improvements of both the understanding of MS Pham reports no financial disclosures. C.R.G. Guttmann has disease processes and in vivo MRI methods to study those, received support from Mobilengine (free use of platform and using big data and machine learning, specific recommenda- programming by Mobilengine Engineers), as well as the tions were provided on data sharing, machine learning, National Multiple Sclerosis Society, the International Pro- crowdsourcing, personal data protection, and organized gressive Multiple Sclerosis Alliance, and the US Office for analysis challenges. Naval Research, as well as travel support from Roche Phar- maceuticals. C.R.G. Guttmann owns stock in Roche, Novartis, GSK, Alnylam, Protalix Biotherapeutics, Arrow- Acknowledgment head Pharmaceuticals, Cocrystal Pharma, and Sangamo This article is based on a workshop of the MAGNIMS Study Therapeutics. D. Pareto has received speaking honoraria Group that was made possible through financial support from from Novartis and Genzyme. M. Paardekooper reports no the Dutch MS Research Foundation, Amsterdam Neurosci- disclosures. A. de Sitter has been employed on a research ence, VU University Medical Center, Merck KGaA, and grant from Teva. M.A. Rocca received speakers honoraria Novartis. from Biogen Idec, Novartis, Genzyme, Teva, Merck Serono, Roche, Celgene, and Bayer and receives research support Study Funding from the MS Society of Canada and Fondazione Italiana H. Vrenken has received funding support from the Dutch Sclerosi Multipla. V. Wottschel reports no disclosures. M. J. MS Research Foundation (grant 14-876 MS), ZonMW Cardoso is a Founder of BrainMiner, plc. F. Barkhof has jointly with the Dutch MS Research Foundation (grant 40- received compensation for consulting services and/or 44600-98-326), and HealthHolland (grant LSHM19053). speaking activities from Bayer, Biogen Idec, Merck Serono, The MS Center Amsterdam is supported by the Dutch MS Novartis, Roche, Teva, Bracco, and IXICO. Go to Neurol- Research Foundation through a series of program grants ogy.org/N for full disclosures. (current grant 18-358f). M. Jenkinson is supported by the National Institute for Health Research (NIHR) Oxford Publication History Biomedical Research Centre (BRC), and this research was Received by Neurology August 10, 2020. Accepted in final form funded by the Wellcome Trust (215573/Z/19/Z). The September 9, 2021. Wellcome Centre for Integrative Neuroimaging is sup- ported by core funding from the Wellcome Trust (203139/Z/16/Z). D.L. Pham received funding support from National Multiple Sclerosis Society Grants RG-1507- Appendix 1 Authors 05243 and RG-1907-34570, Congressionally Directed Medical Research Programs Grant W81XWH-20-1-0912, Name Location Contribution and the Department of Defense in the Center for Neuro- Hugo Vrenken, Amsterdam UMC, Manuscript science and Regenerative Medicine. C.R.G. Guttmann PhD Amsterdam, the conceptualization, Netherlands manuscript structuring, acknowledges support from the National Multiple Scle- combining author rosis Society (grant identifier RG-1501-03141), the In- contributions, manuscript writing, editing and ternational Progressive Multiple Sclerosis Alliance (grant reviewing identifier PA-1412-02420), the Foundation of the Uni- Mark University of Oxford, UK Writing manuscript text versity of Bordeaux, Roche Pharmaceuticals, and Talan. D. Jenkinson, Bsc, sections, composing Table 1, Pareto received support from Instituto de Salud Carlos III BE, DPhil revising full manuscript, reviewing and editing final (PI18/00823). V. Wottschel has received funding from the manuscript before European Union’s Horizon 2020 research and innovation submission programme under grant agreement 666992. M.J. Cardoso Dzung Pham, The Henry M. Jackson Writing manuscript text is supported by the Wellcome/EPSRC Centre for Medical PhD Foundation, Bethesda, sections, revising full Engineering (WT203148/Z/16/Z) and the Wellcome MD manuscript, reviewing and editing final manuscript Flagship Programme (WT213038/Z/18/Z). F. Barkhof is before submission supported by the NIHR biomedical research center at Charles R.G. Brigham and Women’s Writing manuscript text UCLH. Guttmann, MD Hospital, Boston, MA sections, revising full manuscript, reviewing and editing final manuscript Disclosure before submission H. Vrenken has received research grants from Pfizer, Deborah Vall d’Hebron University Writing manuscript text MerckSerono, Novartis, and Teva; speaker honoraria from Pareto, PhD Hospital and Research sections, revising full Novartis; and consulting fees from MerckSerono; all funds Institute (VHIR), manuscript, reviewing and Autonomous University editing final manuscript were paid directly to his institution. M. Jenkinson has re- Barcelona, Spain before submission ceived research grants from Novartis, paid to his institution, Continued Neurology.org/N Neurology | Volume 97, Number 21 | November 23, 2021 997 Appendix 1 (continued) Appendix 2 (continued) Name Location Contribution Name Location Role Contribution Michel Amsterdam UMC, the Providing intellectual Claudio San Camillo- Coinvestigator Revising full Paardekooper, Netherlands content on data sharing, Gasperini, Forlanini Hospital, (steering manuscript, PhD reviewing and editing final MD, PhD Rome, Italy committee reviewing and manuscript before member) editing final submission manuscript before submission Alexandra de Amsterdam UMC, the Writing manuscript text Sitter, PhD Netherlands sections, revising full Ludwig University Hospital, Coinvestigator Revising full manuscript, reviewing and Kappos, Basel, Switzerland (steering manuscript, editing final manuscript MD, PhD committee reviewing and before submission member) editing final manuscript before Maria A. Rocca, IRCCS San Raffaele Writing manuscript text submission MD Scientific Institute, Milan, sections, composing Table 2, Italy revising full manuscript, Jacqueline University of Coinvestigator Revising full reviewing and editing final Palace, Oxford, UK (steering manuscript, manuscript before submission MD, PhD committee reviewing and member) editing final Viktor Amsterdam UMC, the Writing manuscript text manuscript before Wottschel, PhD Netherlands sections, revising full submission manuscript, reviewing and editing final manuscript Àlex Vall d’Hebron Coinvestigator Revising full before submission Rovira, Hospital, Barcelona, (steering manuscript, MD, PhD Spain; San Raffaele, committee reviewing and M. Jorge King’s College London, UK Writing manuscript text Milan, Italy member) editing final Cardoso, PhD sections, revising full manuscript before manuscript, reviewing and submission editing final manuscript before submission Tarek University College Coinvestigator Revising full Yousry, London, UK; San (steering manuscript, Frederik Amsterdam UMC, the Manuscript MD, PhD Raffaele, Milan, Italy committee reviewing and Barkhof, MD, Netherlands; University conceptualization, revising member) editing final PhD College London, UK full manuscript, reviewing manuscript before and editing final manuscript submission before submission References Appendix 2 Coinvestigators 1. Amiri H, de Sitter A, Bendfeldt K, et al. Urgent challenges in quantification and interpretation of brain grey matter atrophy in individual MS patients using MRI. Name Location Role Contribution Neuroimage Clin. 2018;19:466-475. 2. de Sitter A, Verhoeven T, Burggraaff J, et al. Reduced accuracy of MRI deep grey Nicola de University of Siena, Coinvestigator Revising full matter segmentation in multiple sclerosis: an evaluation of four automated methods Stefano, Italy (steering manuscript, against manual reference segmentations in a multi-center cohort. J Neurol 2020; MD, PhD committee reviewing and editing 267(12):3541-3554. member) final manuscript 3. Glaser A, Stahmann A, Meissner T, et al. Multiple sclerosis registries in Europe: an before submission updated mapping survey. Mult Scler Relat Disord. 2018;27:171-178. 4. Hurwitz BJ. Registry studies of long-term multiple sclerosis outcomes: description of Jaume Vall d’Hebron Coinvestigator Revising full key registries. Neurology. 2011;76(1 suppl 1):S3–S6. Sastre- Hospital, Barcelona, (steering manuscript, 5. Flachenecker P, Buckow K, Pugliatti M, et al. Multiple sclerosis registries in Europe: Garriga Spain committee reviewing and editing results of a systematic survey. Mult Scler. 2014;20(11):1523-1532. member) final manuscript 6. Butzkueven H, Chapman J, Cristiano E, et al. MSBase: an international, online reg- before submission istry and platform for collaborative outcomes research in multiple sclerosis. Mult Scler. 2006;12(6):769-774. Olga University College Coinvestigator Revising full 7. Jokubaitis VG, Spelman T, Kalincik T, et al. Predictors of long-term disability accrual Ciccarelli, London, UK (steering manuscript, in relapse-onset multiple sclerosis. Ann Neurol. 2016;80(1):89-100. MD, PhD committee reviewing and 8. Spelman T, Gray O, Trojano M, et al. Seasonal variation of relapse rate in multiple member) editing final sclerosis is latitude dependent. Ann Neurol. 2014;76(6):880-890. manuscript before 9. Kalincik T, Manouchehrinia A, Sobisek L, et al. Towards personalized therapy for submission multiple sclerosis: prediction of individual treatment response. Brain. 2017;140(9): 2426-2443. Christian Medical University Coinvestigator Revising full 10. Filippi M, Preziosa P, Rocca MA. MRI in multiple sclerosis: what is changing? Curr Enzinger, Graz, Austria (steering manuscript, Opin Neurol. 2018;31(4):386-395. MD, PhD committee reviewing and 11. Bermel R, Mowry E, Krupp L, et al. Multiple sclerosis partners advancing technology member; co- editing final and health solutions (MS PATHS): initial launch experience. Neurology. 2017;88(16 chair) manuscript before suppl):P1 submission 12. Filippi M, Tedeschi G, Pantano P, De Stefano N, Zaratin P, Rocca MA. The Italian Neuroimaging Network Initiative (INNI): enabling the use of advanced MRI tech- Massimo San Raffaele, Milan, Coinvestigator Revising full niques in patients with MS. Neurol Sci. 2017;38(6):1029-1038. Filippi, Italy (steering manuscript, 13. Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image MD, PhD committee reviewing and analysis. Med Image Anal. 2017;42:60-88. member) editing final 14. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolu- manuscript before tional neural networks. Commun Acm. 2017;60:84-90. submission 15. Ghafoorian M, Karssemeijer N, Heskes T, et al. Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Sci Rep. 2017;7(1):5110. 998 Neurology | Volume 97, Number 21 | November 23, 2021 Neurology.org/N 16. Steenwijk MD, Pouwels PJ, Daams M, et al. Accurate white matter lesion segmen- 38. Zhao C, Shao M, Carass A, et al. Applications of a deep learning method for anti- tation by k nearest neighbor classification with tissue type priors (kNN-TTPs). aliasing and super-resolution in MRI. Magn Reson Imaging. 2019;64(12): Neuroimage Clin. 2013;3:462-469. 132-141. 17. Coronado I, Gabr RE, Narayana PA. Deep learning segmentation of gadolinium- 39. Dewey BE, Zhao C, Reinhold JC, et al. DeepHarmony: a deep learning approach to enhancing lesions in multiple sclerosis. Mult Scler J. 2020;27(4):519-527. contrast harmonization across scanner changes. Magn Reson Imaging. 2019;64:160-170. 18. Kocevar G, Stamile C, Hannoun S, et al. Graph theory-based brain connectivity for 40. Dewey BE, Zhao C, Carass A, et al. Deep Harmonization of Inconsistent MR Data for automatic classification of multiple sclerosis clinical courses. Front Neurosci. 2016;10:478. Consistent Volume Segmentation. Springer International Publishing; 2018:20-30. 19. Eshaghi A, Wottschel V, Cortese R, et al. Gray matter MRI differentiates neuromyelitis 41. Jog A, Fischl B. Pulse Sequence Resilient Fast Brain Segmentation. Springer International optica from multiple sclerosis using random forest. Neurology. 2016;87(23):2463-2470. Publishing; 2018:654-662. 20. Bendfeldt K, Taschler B, Gaetano L, et al. MRI-based prediction of conversion from 42. Valverde S, Salem M, Cabezas M, et al. One-shot domain adaptation in multiple clinically isolated syndrome to clinically definite multiple sclerosis using SVM and sclerosis lesion segmentation using convolutional neural networks. Neuroimage Clin. lesion geometry. Brain Imaging Behav. 2018;13(5):1361-1374. 2019;21:101638. 21. Wottschel V, Chard DT, Enzinger C, et al. SVM recursive feature elimination analyses 43. Nair T, Precup D, Arnold DL, Arbel T. Exploring uncertainty measures in deep of structural brain MRI predicts near-term relapses in patients with clinically isolated networks for multiple sclerosis lesion detection and segmentation. Med Image Anal. syndromes suggestive of multiple sclerosis. Neuroimage Clin. 2019;24:102011. 2020;59:101557. 22. Valverde S, Cabezas M, Roura E, et al. Improving automated multiple sclerosis lesion 44. Winkel DJ, Brantner P, Lutz J, Korkut S, Linxen S, Heye TJ. Gamification of electronic segmentation with a cascaded 3D convolutional neural network approach. Neuro- learning in radiology education to improve diagnostic confidence and reduce error Image. 2017;155:159-168. rates. AJR Am J Roentgenol. 2020;214(3):618-623. 23. Aslani S, Dayan M, Storelli L, et al. Multi-branch convolutional neural network for 45. Bogovic JA, Jedynak B, Rigg R, et al. Approaching expert results using a hierarchical multiple sclerosis lesion segmentation. NeuroImage. 2019;196(5):1-15. cerebellum parcellation protocol for multiple inexpert human raters. NeuroImage. 24. Brosch T, Tang LY, Youngjin Yoo Y, Li DK, Traboulsee A, Tam R. Deep 3D con- 2013;64:616-629. volutional encoder networks with shortcuts for multiscale feature integration applied to 46. Damangir S, de Sitter A, Brouwer I, et al. A distributed platform for making large scale multiple sclerosis lesion segmentation. IEEE Trans Med Imaging. 2016;35(5):1229-1239. manual reference datasets for MS lesion segmentation. Mult Scler J. 2018;24:864. 25. McKinley R, Wepfer R, Grunder L, et al. Automatic detection of lesion load change in 47. Forcier MB, Gallois H, Mullan S, Joly Y. Integrating artificial intelligence into health Multiple Sclerosis using convolutional neural networks with segmentation confi- care through data access: can the GDPR act as a beacon for policymakers? J Law Biosci. dence. Neuroimage Clin. 2020;25:102104. 2019;6:317-335. 26. Salem M, Valverde S, Cabezas M, et al. A fully convolutional neural network for new 48. Dahl RN M, Shlens J. Pixel recursive super resolution. Proc IEEE Int Conf Computer T2-w lesion detection in multiple sclerosis. Neuroimage Clin. 2020;25:102149. Vis. 2017:5439-5448. 27. Maggi P, Fartaria MJ, Jorge J, et al. CVSnet: a machine learning approach for automated 49. Bischoff-Grethe A, Ozyurt IB, Busa E, et al. A technique for the deidentification of central vein sign assessment in multiple sclerosis. NMR Biomed. 2020;33(5):e4283. structural brain MR images. Hum Brain Mapp. 2007;28:892-903. 28. Ye Z, George A, Wu AT, et al. Deep learning with diffusion basis spectrum imaging for 50. Milchenko M, Marcus D. Obscuring surface anatomy in volumetric imaging data. classification of multiple sclerosis lesions. Ann Clin Transl Neurol. 2020;7(5):695-706. Neuroinformatics. 2013;11(1):65-75. 29. Narayana PA, Coronado I, Sujit SJ, Wolinsky JS, Lublin FD, Gabr RE. Deep learning 51. de Sitter A, Visser M, Brouwer I, et al. Facing privacy in neuroimaging: removing facial for predicting enhancing lesions in multiple sclerosis from noncontrast MRI. Radi- features degrades performance of image analysis methods. Eur Radiol. 2020;30: ology. 2020;294(2):398-404. 1062-1074. 30. Eitel F, Soehler E, Bellmann-Strobl J, et al. Uncovering convolutional neural network 52. Abramian DERefacing: reconstructing anonymized facial features using GANS. 2019 decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019); 2019: relevance propagation. Neuroimage Clin. 2019;24:102003. 1104-1108. 31. Yoo Y, Tang LYW, Brosch T, et al. Deep learning of joint myelin and T1w MRI 53. Kaissis GA, Makowski MR, Rückert D, Braren RF. Secure, privacy-preserving and features in normal-appearing brain tissue to distinguish between multiple sclerosis federated machine learning in medical imaging. Nat Machine Intelligence. 2020;2: patients and healthy controls. Neuroimage Clin. 2018;17:169-178. 305-311. 32. Gabr RE, Coronado I, Robinson M, et al. Brain and lesion segmentation in multiple 54. Chang K, Balachandar N, Lam C, et al. Distributed deep learning networks sclerosis using fully convolutional neural networks: a large-scale study. Mult Scler. among institutions for medical imaging. J Am Med Inform Assoc. 2018;25(8): 2019;26(10):1217-1226. 945-954. 33. Sander L, Pezold S, Andermatt S, et al. Accurate, rapid and reliable, fully automated 55. Remedios SW, Roy S, Bermudez C, et al. Distributed deep learning across multisite MRI brainstem segmentation for application in multiple sclerosis and neurodegen- datasets for generalized CT hemorrhage segmentation. Med Phys. 2020;47(1):89-98. erative diseases. Hum Brain Mapp. 2019;40(14):4091-4104. 56. Azencott CA. Machine learning and genomics: precision medicine versus patient 34. Liu H, Xiang QS, Tam R, et al. Myelin water imaging data analysis in less than one privacy. Philos Trans A Math Phys Eng Sci. 2018;376(2128):20170350. minute. NeuroImage. 2020;210:116551. 57. Styner M, Lee J, Chin B, et al. 3D segmentation in the clinic: a grand challenge II: MS 35. Yoon J, Gong E, Chatnuntawech I, et al. Quantitative susceptibility mapping using lesion segmentation. MIDAS J. 2008. deep neural network: QSMnet. NeuroImage. 2018;179:199-206. 58. Carass A, Roy S, Jog A, et al. Longitudinal multiple sclerosis lesion segmentation: 36. Wei W, Poirion E, Bodini B, et al. Fluid-attenuated inversion recovery MRI synthesis resource and challenge. NeuroImage. 2017;148:77-102. from multisequence MRI using three-dimensional fully convolutional networks for 59. Commowick O, Istace A, Kain M, et al. Objective evaluation of multiple sclerosis lesion multiple sclerosis. J Med Imaging. 2019;6(1):014005. segmentation using a data management and processing infrastructure. Sci Rep. 2018;8:13650. 37. Sreekumari A, Shanbhag D, Yeo D, et al. A deep learning-based approach to reduce 60. Ontaneda D, Fox RJ. Imaging as an outcome measure in multiple sclerosis. Neuro- rescan and recall rates in clinical MRI examinations. AJNR Am J Neuroradiol. 2019; therapeutics. 2017;14(1):24-34. 40(2):217-223. Data available from Dryad (eReferences): doi.org/10.5061/dryad.2fqz612p9 Subspecialty Alerts by E-mail! Customize your online journal experience by signing up for e-mail alerts related to your subspecialty or area of interest. Access this free service by clicking on the “My Alerts” link on the home page. An extensive list of subspecialties, methods, and study design choices will be available for you to choose from—allowing you priority alerts to cutting-edge research in your field! Neurology.org/N Neurology | Volume 97, Number 21 | November 23, 2021 999