Received: 2 March 2020 Revised: 26 May 2020 Accepted: 31 May 2020 DOI: 10.1002/hbm.25096 REVIEW ARTICLE Mega-analysis methods in ENIGMA: The experience of the generalized anxiety disorder working group André Zugman1 | Anita Harrewijn1 | Elise M. Cardinale1 | Hannah Zwiebel1 | Gabrielle F. Freitag1 | Katy E. Werwath1 | Janna M. Bas-Hoogendam2,3,4 | Nynke A. Groenewold5 | Moji Aghajani6,7 | Kevin Hilbert8 | Narcis Cardoner9,10,11 | Daniel Porta-Casteràs9,10,11 | Savannah Gosnell12 | Ramiro Salas12 | Karina S. Blair13 | James R. Blair13 | Mira Z. Hammoud14 | Mohammed Milad14 | Katie Burkhouse15 | K. Luan Phan16 | Heidi K. Schroeder17 | Jeffrey R. Strawn17 | Katja Beesdo-Baum18 | Sophia I. Thomopoulos19 | Hans J. Grabe20,21 | Sandra Van der Auwera20,21 | Katharina Wittfeld20,21 | Jared A. Nielsen22,23 | Randy Buckner22,23,24 | Jordan W. Smoller24 | Benson Mwangi25 | Jair C. Soares25 | Mon-Ju Wu25 | Giovana B. Zunta-Soares25 | Andrea P. Jackowski26 | Pedro M. Pan26 | Giovanni A. Salum27 | Michal Assaf28,29 | Gretchen J. Diefenbach30,31 | Paolo Brambilla32 | Eleonora Maggioni32 | David Hofmann33 | Thomas Straube33 | Carmen Andreescu34 | Rachel Berta34 | Erica Tamburo34 | Rebecca Price35 | Gisele G. Manfro36,37 | Hugo D. Critchley38 | Elena Makovac39 | Matteo Mancini38 | Frances Meeten40 | Cristina Ottaviani41 | Federica Agosta42,43 | Elisa Canu42 | Camilla Cividini42 | Massimo Filippi42,43,44 | Milutin Kostic45,46 | Ana Munjiza45 | Courtney A. Filippi1 | Ellen Leibenluft1 | Bianca A. V. Alberton47 | Nicholas L. Balderston48 | Monique Ernst1 | Christian Grillon1 | Lilianne R. Mujica-Parodi49 | Helena van Nieuwenhuizen50 | Gregory A. Fonzo51 | Martin P. Paulus52 | Murray B. Stein53 | Raquel E. Gur54 | Ruben C. Gur54 | Antonia N. Kaczkurkin54 | Bart Larsen54 | Theodore D. Satterthwaite54 | Jennifer Harper55 | Michael Myers55 | Michael T. Perino55 | Qiongru Yu55 | Chad M. Sylvester55 | Dick J. Veltman6 | Ulrike Lueken8 | Nic J. A. Van der Wee2,3 | Dan J. Stein5,56 | Neda Jahanshad19 | Paul M. Thompson19 | Daniel S. Pine1 | Anderson M. Winkler1 1 National Institute of Mental Health (NIMH), National Institutes of Health (NIH), Bethesda, Maryland 2 Leiden University Medical Center, Department of Psychiatry, Leiden, The Netherlands 3 Leiden Institute for Brain and Cognition (LIBC), Leiden, The Netherlands This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2020 The Authors. Human Brain Mapping published by Wiley Periodicals LLC. Hum Brain Mapp. 2022;43:255–277. wileyonlinelibrary.com/journal/hbm 255 256 ZUGMAN ET AL. 4 Leiden University, Institute of Psychology, Developmental and Educational Psychology, Leiden, The Netherlands 5 Department of Psychiatry & Neuroscience Institute, University of Cape Town, Cape Town, South Africa 6 Department. of Psychiatry, Amsterdam UMC/VUMC, Amsterdam, The Netherlands 7 GGZ InGeest, Department of Research & Innovation, Amsterdam, The Netherlands 8 Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany 9 Department of Mental Health, University Hospital Parc Taulí-I3PT, Barcelona, Spain 10 Department of Psychiatry and Forensic Medicine, Universitat Autònoma de Barcelona, Barcelona, Spain 11 Centro de Investigación Biomédica en Red de Salud Mental, Carlos III Health Institute, Madrid, Spain 12 Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, Texas 13 Center for Neurobehavioral Research, Boys Town National Research Hospital, Boys Town, Nebraska 14 Department of Psychiatry, New York University, New York, New York 15 Department of Psychiatry, University of Illinois at Chicago, Chicago, Illinois 16 Department of Psychiatry and Behavioral Health, The Ohio State University, Columbus, Ohio 17 Department of Psychiatry & Behavioral Neuroscience, University of Cincinnati, Cincinnati, Ohio 18 Behavioral Epidemiology, Institute of Clinical Psychology and Psychotherapy, Technische Universität Dresden, Dresden, Germany 19 Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, University of Southern California, Marina del Rey, California 20 Department of Psychiatry and Psychotherapy, University Medicine Greifswald, Greifswald, Germany 21 German Center for Neurodegenerative Diseases (DZNE), Site Rostock/Greifswald, Greifswald, Germany 22 Department of Psychology, Harvard University, Cambridge, Massachusetts 23 Center for Brain Science, Harvard University, Cambridge, Massachusetts 24 Department of Psychiatry, Massachusetts General Hospital, Boston, Massachusetts 25 Center Of Excellence On Mood Disorders, Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at Houston, Houston, Texas 26 LiNC, Department of Psychiatry, Federal University of S~ao Paulo, S~ ao Paulo, S~ao Paulo, Brazil 27 Section on Negative Affect and Social Processes, Hospital de Clínicas de Porto Alegre, Universidade Federal do Rio Grande do Sul, Porto Alegre, Rio Grande do Sul, Brazil 28 Olin Neuropsychiatry Research Center, Institute of Living, Hartford Hospital, Hartford, Connecticut 29 Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 30 Anxiety Disorders Center, Institute of Living, Hartford Hospital, Hartford, Connecticut 31 Yale School of Medicine, New Haven, Connecticut 32 Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milan, Italy 33 Institute of Medical Psychology and Systems Neuroscience, University of Muenster, Muenster, Germany 34 Department of Psychiatry, University of Pittsburgh, Pittsburgh, Pennsylvania 35 Department of Psychiatry & Psychology, University of Pittsburgh, Pittsburgh, Pennsylvania 36 Anxiety Disorder Program, Hospital de Clínicas de Porto Alegre, Porto Alegre, Rio Grande do Sul, Brazil 37 Department of Psychiatry, Federal University of Rio Grande do Sul, Porto Alegre, Rio Grande do Sul, Brazil 38 Department of Neuroscience, Brighton and Sussex Medical School, University of Sussex, Brighton, UK 39 Centre for Neuroimaging Science, Kings College London, London, UK 40 School of Psychology, University of Sussex, Brighton, UK 41 Department of Psychology, Sapienza University of Rome, Rome, Italy 42 Neuroimaging Research Unit, Institute of Experimental Neurology, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy 43 Vita-Salute San Raffaele University, Milan, Italy 44 Neurology and Neurophysiology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy 45 Institute of Mental Health, University of Belgrade, Belgrade, Serbia 46 Department of Psychiatry, School of Medicine, University of Belgrade, Belgrade, Serbia 47 Graduate Program in Electrical and Computer Engineering, Universidade Tecnológica Federal do Paraná, Curitiba, Puerto Rico, Brazil 48 Center for Neuromodulation in Depression and Stress, University of Pennsylvania, Philadelphia, Pennsylvania 49 Department of Biomedical Engineering, Stony Brook University, Stony Brook, New York 50 Department of Physics, Stony Brook University, Stony Brook, New York ZUGMAN ET AL. 257 51 Department of Psychiatry, The University of Texas at Austin Dell Medical School, Austin, Texas 52 Laureate Institute for Brain Research, Tulsa, Oklahoma 53 Department of Psychiatry & Family Medicine and Public Health, University of California, La Jolla, California 54 Department of Psychiatry, University of Pennsylvania, Philadelphia, Pennsylvania 55 Department of Psychiatry, Washington University, St. Louis, Missouri 56 SAMRC Unite on Risk & Resilience in Mental Disorders, Department of Psychiatry & Neuroscience Institute, University of Cape Town, Cape Town, South Africa Correspondence André Zugman and Anderson M. Winkler, Abstract National Institute of Mental Health (NIMH), The ENIGMA group on Generalized Anxiety Disorder (ENIGMA-Anxiety/GAD) is part National Institutes of Health (NIH), Bethesda, MD, USA. of a broader effort to investigate anxiety disorders using imaging and genetic data Email: andre.zugman@nih.gov (A. Z.) and across multiple sites worldwide. The group is actively conducting a mega-analysis of a anderson.winkler@nih.gov (A. M. W.) large number of brain structural scans. In this process, the group was confronted with Funding information many methodological challenges related to study planning and implementation, Fondazione Cariplo, Grant/Award Number: 2016-0908; Hartford HealthCare Research between-country transfer of subject-level data, quality control of a considerable Funding Initiative, Grant/Award Number: amount of imaging data, and choices related to statistical methods and efficient use #129522; Federal Ministry of Education and Research, Grant/Award Numbers: 01ZZ0403, of resources. This report summarizes the background information and rationale for 01ZZ0103, 01ZZ9603, 01ER1703, 01ER1303; the various methodological decisions, as well as the approach taken to implement National Institutes of Health (NIH), Grant/ Award Numbers: K23MH109983, them. The goal is to document the approach and help guide other research groups T32MH100019, R01MH101486, working with large brain imaging data sets as they develop their own analytic pipe- U54-EB020403, ZIA-MH002782, ZIA- MH002781 lines for mega-analyses. KEYWORDS data sharing, generalized anxiety disorder, mega-analyses, meta-analyses, neuroimaging 1 | I N T RO DU CT I O N respect country-level differences in data privacy and data protection procedures, discussed below. In the case of ENIGMA-GAD, as The ENIGMA (Enhancing NeuroImaging Genetics through Meta Anal- detailed in the final section of this article, the vast majority of sites ysis) Consortium, started in 2009, with the aim of performing large- contributed raw, T1-weighted magnetic resonance imaging (MRI) scale neuroimaging genetics research using meta-analytic methods by scans, as opposed to processed scans or results of subsequent ana- pooling data from around the world. ENIGMA has since expanded to lyses. These raw data could then be processed centrally using an include many working groups, resources, and expertise to answer fun- imaging processing software, in this case FreeSurfer1 (Dale, Fischl, & damental questions in neuroscience, psychiatry, neurology, and genet- Sereno, 1999; Fischl et al., 2002; Fischl, Sereno, & Dale, 1999). Having ics (Thompson et al., 2014; Thompson et al., 2020). One of these access to raw IPD provided unique opportunities to review methods groups is the ENIGMA-Anxiety working group, created in 2016 (Bas- for handling and harmonizing such data, defining processing pipelines, Hoogendam et al., 2020), focused on anxiety related disorders. Such and implementing analytic strategies. Crucially, this led ENIGMA-GAD disorders, that include social anxiety disorder, specific phobia, panic to prioritize a mega-analysis approach. This approach consists of ana- disorder, generalized anxiety disorder (GAD), and agoraphobia, share lyzing IPD from all sites in one stage. This contrasts with two-stage substantive phenomenological features and are often comorbid. approaches, which consist of analyzes of site-specific results in a sec- Within the ENIGMA-Anxiety working group, a subgroup devoted to ond step after each site generates processed data in an initial step the study of GAD was formed, the ENIGMA-Anxiety/GAD (detailed below). “subgroup,” which for simplicity is referred to here as This paper presents some of the challenges posed by the decision, “ENIGMA-GAD.” by the ENIGMA-GAD group, to use a mega-analysis, and discusses Because the ENIGMA-Anxiety working group was formed rela- the rationale for the choices that were made to establish the analysis tively recently, it has benefited from the experience and work per- plan. The discussion is broadly applicable to mega-analyses in the con- formed by earlier groups, particularly in terms of collaborative text of ENIGMA and other international neuroimaging efforts. Below, methods. In more recent years, research groups have become increas- differences between meta-analytic vs. mega-analytic approaches, ben- ingly favorable toward sharing and transferring de-identified individual efits of preregistration, issues concerning data sharing and data reuse participant data (IPD), often as part of cooperative agreements that are discussed. Methods for quality control and choices with respect to 258 ZUGMAN ET AL. measurements and statistical analyses are also presented. Finally, spe- performed by the coordinating facility, while addressing site- cific choices in the ENIGMA-GAD group with respect to each of these related heterogeneity; this would be a one stage meta-analysis, or issues are described. simply “mega-analysis.” With imaging data, such mega-analyses could start with the raw images being sent to the coordinating facility where they then undergo batch processing using identical 2 | META-ANALYSIS VERSUS MEGA- methods and computing environments. Alternatively, mega- ANALYSIS analyses could start with image-derived measurements, such as the volumes of brain structures or cortical surface area, already As collaborative and coordinated endeavors, ENIGMA meta-analyses computed and furnished by the participating sites to the coordi- studies operate differently from literature-based meta-analyses. In the nating facility, for each individual participant; the coordinating latter, results from published studies are compiled to draw conclusions facility then proceeds to the statistical analysis. Combination of on a certain question. In most cases, such pooled studies have been approaches for some projects (e.g., some sites sending raw data conducted and published over many years, with high sample and for processing whereas others sending processed data) are also methodological heterogeneity, encompassing diverse statistical possible. approaches. Such diversity is aggravated in meta-analyses that exam- Analyses using IPD offer several advantages (Riley, Lambert, & ine neuroimaging studies. In neuroimaging studies, substantial chal- Abo-Zaid, 2010): they improve consistency in inclusion criteria lenges for combined inference result from the use of statistical maps across sites, better treatment of confounds and of missing data, limited to significant p-values or test statistics, tables with coordinates verification of assumptions of statistical models, standardization of in relation to some standard (but not always the same) stereotaxic procedures, increases in statistical power, reductions in biases for space, and different representations of the brain (volume-based or not depending on previous publications of (invariably significant) surface-based; Fox, Lancaster, Laird, & Eickhoff, 2014; Müller results. Access to IPD further allows other strategies for investiga- et al., 2018; Tahmasian et al., 2019). Moreover, because of publication tion that are not limited to hypotheses testing. For example, it may biases, there can be a misrepresentation of negative results (the “file- allow classification at the individual participant level using drawer” problem; Rosenthal, 1979) or study selection (Roseman machine-learning methods (Nunes et al., 2018). In a mega-analysis et al., 2011). starting with raw imaging data, all data can be processed identi- In ENIGMA, these issues are minimized through analysis of IPD cally in the same facility, thus minimizing the chance for errors or using an agreed upon processing strategy. Briefly, three approaches, variability that can arise when each site conducts these aspects of that relate to data location, are currently used by different projects the analysis. One major challenge to this approach is that mega- within ENIGMA working groups: (a) all raw data and all derived IPD analysis requires at least one site to possess the necessary remote in relation to the coordinating facility; (b) all raw data remote resources and expertise to handle large datasets. Additionally, this in relation to the coordinating facility, but derived data centralized; approach is only possible when IPD are shared with a central facil- (c) all raw data centralized. These approaches are not mutually exclu- ity. Data exchange on an IPD level often is limited as data protec- sive within a working group, and different projects conducted by the tion is regulated differently among research projects, consortia, same working group may each use a different strategy, depending on and countries. Barriers on data exchange and limitation of available the project goals and considerations about data availability, computa- resources can, in effect, restrict the participation to few well- tional resources, and expertise. These approaches are summarized equipped centers. schematically in Figure 1. Multiple studies have compared meta- and mega-analysis (Belias, For meta-analysis with access to IPD, the strategy includes quality Rovers, Reitsma, Debray, & IntHout, 2019; Riley et al., 2010; checks and statistical analysis over mostly coetaneous data. Summary Simmonds, Stewart, & Stewart, 2015), suggesting superiority of mega- statistics (such as effect sizes, standard errors, and/or confidence analyses with IPD when compared to meta-analyses in terms of higher intervals) are pooled by a coordinating facility that then uses meta- statistical power and acceptable false positive rates. In the context of analytic methods for inference across sites. Such a coordinated, two- ENIGMA, comparisons have likewise tended to favor mega-analyses stage meta-analysis approach has been pursued by most ENIGMA (Boedhoe et al., 2019; Kochunov et al., 2014; Koshiyama et al., 2020). working groups (Hibar et al., 2015; Hibar et al., 2016; Schmaal However, if individual sites use identical processing strategies with et al., 2016; Stein et al., 2012; van Erp et al., 2016), particularly due to IPD, a random-effects two-stage approach leads to the same esti- privacy concerns regarding genetic data. ENIGMA genome-wide asso- mates as a (one-stage) mega-analysis. This is well-established in the ciation studies still use a meta-analysis approach (Hibar et al., 2017; neuroimaging literature, which uses similar statistical methods for Satizabal et al., 2019); sites analyze their own data with an agreed multi-level inference for analysis of functional magnetic resonance upon protocol, which avoids the need to transfer individual participant imaging data (Beckmann, Jenkinson, & Smith, 2003; Worsley genomic data, and allows distributed analysis of computationally et al., 2002). Rarely such identical processing can be accomplished, intense approaches. though, given the usually large number of sites and, and the need that Other strategies can be considered if the coordinating facility all engage in approaches intended to ensure consistency (discussed has access to all IPD: a single-stage statistical analysis can be below). ZUGMAN ET AL. 259 F I G U R E 1 Differences between classical, literature-based meta-analyses, conducted without access to individual participant data (IPD) (upper panel) versus approaches used by different ENIGMA working groups, in which researchers, collectively, have access to IPD (lower panel). The latter encompasses three main approaches (top) data are processed using common methods at each site, then summary statistics are computed and sent to a coordinating facility which then conducts a meta-analysis; (middle) data are processed using common methods at each site, then sent to the coordinating facility which then conducts a mega-analysis; and (bottom) raw data are sent to the coordinating facility which then processes the data in batch and conducts a mega-analysis, while taking site-specific effects into account 3 | ANALYSIS PLAN AND and an international directory was created by the World Health Orga- PREREGISTRATION nization (WHO), the International Clinical Trials Registry Platform (ICTRP).3 However, broadly similar efforts did not emerge in other Preregistration of clinical trials has been emphasized for many years, research areas for decades. Defining a hypothesis, an associated anal- 2 and a registry was established by law in the United States through ysis plan, and preregistering these ideas before conducting any ana- the Food and Drug Administration Modernization Act of 1997 lyses is important in many ways (Chambers, 2013). It helps to (Dickersin & Rennie, 2003). Similar registries exist in other countries, conceptually separate specific, previously formulated hypotheses 260 ZUGMAN ET AL. from exploratory analyses that have potential to generate new transferred parsimoniously among sites without substantive risk of hypotheses based on the data (Nosek, Ebersole, DeHaven, & reidentification. It should be noted, however, that without precaution- Mellor, 2018; Wagenmakers, Wetzels, Borsboom, van der Maas, & ary measures, repeated computation of aggregate results using slightly Kievit, 2012). Likewise, it helps to separate a priori and exploratory varying subsets of participants can expose information about individ- hypotheses and the analytic plans used for their investigation uals (Dwork, 2006). This risk can be minimized through agreements (Ledgerwood, 2018). The benefits, however, stretch well beyond epis- among researchers on the nature and amount of aggregated data to temological advantages by reducing the potential for questionable be transferred. For mega-analyses, in which IPD are transferred, fur- research practices (Chambers, Feredoes, Muthukumaraswamy, & ther attention is needed, due to differences across sites in the regula- Etchells, 2014). For example, preregistration reduces problems that tions that protect the confidentiality, integrity, and security of the IPD follow when negative results remain unreported (Rosenthal, 1979; and their use in human research. In international collaborations, such Sterling, 1959), reduces the chances of selective reporting (Macleod as ENIGMA, accommodating such requirements necessitates that the et al., 2014) and maximizes transparency in analytic approaches, strictest regulations are followed. While compliance with the law must thereby facilitating replication (Simmons, Nelson, & be integral, three points are particularly relevant for ENIGMA projects: Simonsohn, 2011). Without preregistration, these problems remain (a) protection of data and privacy of research subjects, (b) data reuse, prevalent, possibly due to the structure of incentives in academic and (c) international transfers of data. environments (Neuroskeptic, 2012; Nosek, Spies, & Motyl, 2012). Pre- In the United States (US), research must follow the Federal Policy registration also reduces hypothesizing after the results are known for the Protection of Human Subjects (the “Common Rule”; Arellano, (Kerr, 1998), and protects scientists from other biases Dai, Wang, Jiang, & Ohno-Machado, 2018). This requires that specific (Chambers, 2013), such as confirmation bias, hindsight bias, and consent be obtained from participants before their data and/or speci- anchoring effects (Moreau, 2019). mens can be used not only for the research project for which they are For ENIGMA, specific details and challenges need to be consid- enrolling, but also for future research that may use such material, ered when preregistering a study. First, an analytic plan must be dis- which often is the case of ENIGMA projects. Privacy in the US is cussed with participating centers. The plan should include who access governed by the Health Insurance Portability and Accountability Act the data, roles of each participating site and their personnel, compli- (HIPAA) of 1996, which requires patient data to be de-identified; ance with supervening laws and regulations, funding sources, as well reuse requires approval by an Institutional Review Board. as authorship expectation. This ensures that pooled data from differ- Regulations differ, however, across countries. In the US, there is a ent cohorts are analyzed in a way acceptable by all investigators. Sec- presumption that processing personal data is lawful unless it is ond, many ENIGMA sites may have already analyzed the data they expressly forbidden. In the European Union (EU), in contrast, the share for meta or mega-analysis, often to test similar hypotheses as processing of such data is prohibited unless there is a lawful basis that those being considered for the ENIGMA combined analyses. permits it (Dove, 2018). Legal provision for data protection and use in Obtaining credible results requires an analytic plan free of influences research comes from the General Data Protection Regulation (GDPR), from findings known by the investigators, and that remains inclusive adopted in 2016, which also covers the use of data from EU residents of all relevant data. Preregistration mitigates such concerns by outside the Union (Chassang, 2017). While HIPAA emphasizes subject supporting reasonable hypotheses of broad interest and with well- privacy, the GDPR makes no direct mention of privacy whatsoever, defined inclusion and exclusion criteria of subjects, both of which are dealing instead with data protection, as established in the EU Charter unlikely to be swayed by prior knowledge of outcomes. These analytic of Fundamental Rights along with the right to a private life. Privacy is plans are formalized into “project proposals,” which can be distributed extremely difficult to define (Alfino & Mayes, 2003), and may be to members for approval and participation, and are often considered a understood in this context as a state of nonaccess to data pertaining form of preregistration for working group members. to an individual (Dove, 2018). Data protection, in turn, is a less ambig- Many platforms support preregistration, though the platform pro- uous definition and can be understood as a set of rules that aim to vided by the Open Science Foundation4 stands out for its comprehen- protect the rights, freedoms, and interests of individuals whose per- siveness and user-friendliness. The process is remarkably simple, with sonal data are handled and used (Tzanou, 2013). the site offering detailed instructions and preregistration templates. The GDPR establishes that data reuse should only be allowed Specifying an embargo period before the registration becomes public where new purposes are compatible with those for which the data is possible, and a digital object identifier (DOI) can be generated. were initially collected. This is usually the case for ENIGMA analyses. International data transfers are not allowed unless the country to which data are sent has been found by the European Commission to 4 | D A T A S HA R I N G A N D R E U S E provide “adequate” data protection; at the time of this writing, the list of countries for which an adequacy decision has been provided Both meta- and mega-analysis require that individual sites transfer includes, for example, Argentina, Israel, Japan, New Zealand, and Swit- data to the coordinating facility. Aggregated data, such as histograms zerland.5 While the list does include the US and Canada, in the case of of quality metrics, effect sizes, confidence intervals, and standard these two it does so for commercial uses of data that do not broadly errors, are not identifiable at the individual level and can be cover research by universities and research institutes as needed for ZUGMAN ET AL. 261 ENIGMA. In the absence of such adequacy decision, or of specific der- Neuroimaging Informatics Technology Initiative (NIFTI). This format ogations, an alternative path to data transfer is through specific provi- stores no personally identifiable information but contains two sion of safeguards concerning data protection. These require the general-purpose fields (“descrip” and “intent_name,” with 80 and signing of legally binding agreements between authorities, or binding 16 bytes, respectively) that could hold such information. The format corporate or institutional rules approved by competent supervisory can also accommodate extensions, and can be paired with a JavaScript authorities (Dove, 2018; Staunton, Slokenberga, & Mascalzoni, 2019). Object Notation text file (JSON), both of which may contain informa- If none of these paths are viable, a possible solution to still allow tion that may allow subject identification. Any field with information research is to determine that the coordinating facility for a given that could lead to reidentification must be erased or removed before ENIGMA Working Group will be in the EU itself; then no data from data can be shared between ENIGMA sites and the coordinating facil- EU subjects need to be transferred to outside the Union. However, ity, or other safeguards must be in place to ensure no reidentification such a workaround is limited in scope and time: countries that are in will be attempted or possible. A popular tool for conversion from the process of adopting legislation modeled after GDPR (such as the DICOM to NIFTI, “dcm2niix” (Rorden, 2014) allows removal of such United Kingdom through the Data Protection Act of 2018) will be information during format conversion. under broadly similar rules; these countries might, nonetheless, Moreover, the data portion of DICOM and NIFTI files may be quickly receive an adequacy decision by the European Commission, edited to ensure that facial features will be removed (defacing). such that transfers between the EU and these countries should ulti- Reidentification of participants based on scan data had been consid- mately be facilitated. ered a remote possibility, which motivated the creation of defacing algorithms (Alfaro-Almagro et al., 2018; Bischoff-Grethe et al., 2007; Milchenko & Marcus, 2013; Schimke, Kuehler, & Hale, 2011). Such 4.1 | De-identification reidentification, however, has recently been demonstrated to be feasi- ble (Schwarz et al., 2019), which now renders defacing mandatory for Regardless of specific legislation, data de-identification is a crucial publicly available data. Moreover, two recent developments further step. De-identification consists of removal of personally identifiable complicate matters. First, even defaced data may be reidentified, par- information that allows data to be traced back to individuals, thus ren- ticularly if facial features are only blurred, as opposed to completely dering such identification impossible or extremely difficult or unlikely. removed (Abramian & Eklund, 2019). Second, recent research indi- In the context of HIPAA, unless otherwise determined by an expert, cates that defacing unfortunately may degrade the performance of removal of information such as names, locations with granularity image processing algorithms, possibly affecting the quality of mea- smaller than that of a state, dates related to an individual (such as surements obtained (de Sitter et al., 2020). For ENIGMA mega-ana- birth date, admission date, etc.), and other identifying details, is con- lyses, reconciling data protection with maximum scientific value that sidered to provide a reasonable basis to assume that the information data can provide may ultimately require bilateral agreements to avoid cannot be used to identify an individual. Full-face photographs and data breaches that could allow for unintended or malicious use. In this any comparable images must likewise be removed for HIPAA compli- case the participating institutions can reach an understanding (usually ance. For ENIGMA data, this means that MRI scans may need to have in the terms of a data use agreement—DUA) that all shared data is to facial features of subjects removed before data are shared (see remain securely stored with limited access to researchers who are below). conducting relevant ENIGMA work. Unlike HIPAA, the GDPR does not specify de-identification methods. Instead, researchers are expected to remain mindful that de- identified data might become reidentifiable through the development 4.2 | Encryption and transfer of new technologies or use of ancillary data. Thus, the GDPR requires vigilance to ensure that data remain anonymous (Dove, 2018). Manag- Encryption reduces the possibility that data might be misappropriated ing the risk of reidentification is crucial, and safeguards should be put when stored, or intercepted during transfer, and thus reduces the in place as if the data were not anonymous. Pseudonymized chances that data can be used in ways that are not in the best interest (e.g., tokenized or key-coded) data are subject to the GDPR, even if of research participants. Data encryption is always compatible with the codes are not shared and remain within different organizations. both HIPAA and GDPR, and in the case of the former, it can be con- For ENIGMA, this means that sites that handle information of EU resi- sidered “a reasonable and appropriate measure” to ensure confidenti- dents must ensure complete de-identification as well as take into ality, which renders it mandatory for all practical purposes. Even account the risk that de-identified data becomes reidentifiable, or pur- without specific regulations, data encryption is good practice insofar sue GDPR compliance by treating data as if not anonymous. as the confidentiality, integrity, and security of data of participants are Imaging data stored in the standard Digital Imaging and Commu- concerned. nications in Medicine (DICOM) file format are accompanied by a host A basic scheme consists of encrypting the data using a reasonably of personally identifiable information. Tools exist to anonymize such secure cipher (algorithm), with a key (password) that can also be used files, by erasing fields from the file header that could contain such for decrypting. Such a key is transmitted from an individual site to the information. Another popular file format used in brain imaging is the ENIGMA coordinating facility through means other than those used 262 ZUGMAN ET AL. to transfer the encrypted data. A more sophisticated approach uses of imaging pipelines, such as the brain imaging data structure (BIDS; pairs of public/private keys: the site encrypts the data using the public Gorgolewski et al., 2016). BIDS prescribes a hierarchy of files and (not secret) key provided by the coordinating facility; data can be directories that is simple and intuitive, yet powerful enough to accom- decrypted by the coordinating facility using the private (secret) key. modate a diverse set of imaging modalities collected in varied circum- Various tools enable encryption of individual files or the generation of stances. The scheme is intended to minimize efforts related to data encrypted containers, which are files that emulate a file system and curation, to reduce the number of errors due to incorrect organization can hold multiple other files. Examples of such tools that operate of data files, and to facilitate the development and usage of software, across multiple platforms include VeraCrypt,6 CipherShed,7 and which can be written to parse the file structure directly (Gorgolewski, GnuPG.8 Alfaro-Almagro, Auer, Bellec, & Capot, 2017). Data transfer can be performed in different ways: if data are Processing of the whole dataset using one operating system and strongly encrypted, transmission does not require further encryption; software version can help avoid inconsistencies. It has been demon- if data are not encrypted, transmission should use a form of secure strated that differences in operating systems can have a small effect communication. For small amounts of data, such as for analyses using on, for example, FreeSurfer metrics (Gronenschild et al., 2012); such imaging-derived measurements, which tend to be smaller than with metrics have been used in many ENIGMA analyses to date, including imaging data in full resolution, a plain email with encrypted attach- in ENIGMA-GAD analyses. Scientists may benefit from monitoring ments containing the (possibly compressed) data, or encrypted emails their computing environment and run analyses in batches that are not with nonencrypted attachments, are sufficient. For large volumes of interspersed with periodic software updates. data, as for mega-analyses that start from raw data, data transfer Options to ensure software consistency include the use of virtual using methods such as SSH file transfer protocol (SFTP) are more con- machines (such as QEMU/KVM,15 VirtualBox,16 or VMware17) or con- venient. However, connection to institutional servers (or even per- tainerized environments (such as Docker18 or Singularity19). In virtual sonal laptops) hosting the data may require the potentially machines, the whole system—including emulated hardware and the problematic opening of firewall exceptions. Two alternative methods “guest” operating system—can be kept static and be shared. Con- are straightforward to implement. One uses strong ciphers to encrypt tainers use a layer of compatibility between the “host” operating sys- the data, which are then stored in a physical portable medium, such as tem and the desired applications. They tend to run faster and have external hard drives, thumbsticks, or even secure digital (SD) cards, to simpler maintenance than virtual machines. In either case, the be sent by post or courier; decryption keys are negotiated ahead of researcher can keep tight control over software versions, libraries, and time and shared through different means. The second method uses dependencies. Neither of the two methods, however, is ideal. Virtual peer-to-peer secure transfers using a service such as Globus,9 a non- machines can be heavier to run and offer less flexible integration with profit, free service provided by the University of Chicago the host operating system (which in turn may have access to a large (Ananthakrishnan, Chard, Foster, & Tuecke, 2015) for data exchange computing cluster, such that integration is something often desirable). among academic or research institutions. Containers address this problem but introduce others: troubleshoot- Cloud storage systems (e.g., Dropbox,10 Box,11 Amazon Web ing experimental software may be difficult because it is not always Services,12 Google Drive,13 and Microsoft Azure14) should be used clear whether a given problem has arisen because of the software with caution. Even though most cloud providers offer some level of itself, or because of the container or its interaction with the host sys- encryption, compliance with data protection and privacy laws may tem. Regardless, such solutions improve reproducibility of results by only be offered with high tier subscriptions or specific security set- allowing researchers to share not only their code and information tings, if offered at all. Users should be aware of the level of compli- about their computing environment, but also their actual computing ance that their choice of cloud system provides. environment. Encryption and transfer have sometimes to be established on a site-specific basis. Some sites may have particular expertise and/or infrastructure in place to allow transfer of large amounts of data using 5 | Q U A L I T Y CO N T R O L particular methods, which would be favored over others. Laws governing transfer of technology and geopolitics may also impact For ENIGMA meta-analyses, each site can perform a quality assess- choices: furnishing encryption software to some countries is illegal in ment of its own data using a previously agreed protocol. Sites can some jurisdictions, whereas receiving hard drives may also pose diffi- report the quality metrics to the coordinating facility, which then can culties in countries that heavily tax or delay the delivery of imported use the information in the statistical model by, for example, giving less goods. weight to sites contributing lower-quality data. ENIGMA protocols provide consistent, streamlined strategies for visual inspection of imaging data; these strategies involve inspection of the cortical border 4.3 | Organization and processing between gray and white matter, parcellations of the cortex, and seg- mentation of subcortical structures. For mega-analyses, while the Before or after being transferred to the coordinating facility, the data same kind of visual inspection could be advantageous, the amount of can be organized into a scheme that facilitates processing and the use data may render this process difficult. Although there is no standard ZUGMAN ET AL. 263 triage or similar requirement before sharing raw data, it is usually the can assess the quality of their own images versus those collected by case that images will have already been seen by at least one investiga- others or elsewhere (Esteban et al., 2019). tor before sharing, and as such, might have been excluded from con- sideration and not sent to the coordinating facility. Moreover, while using the same raters may give higher consistency on selection of par- 5.2 | Euler characteristic ticipants across sites given imaging features, the same process might introduce unwanted bias toward selection, for example, if imaging fea- One particular metric has been found to be a good predictor of the tures used to visually define inclusion or exclusion are unknowingly quality of FreeSurfer outputs: the Euler characteristic (χ; sometimes related to the variables investigated, a risk that may be present even if also called Euler number) of the cortical surface produced before quality criteria are consistent across sites. topological correction (Rosen et al., 2018). To conceptualize the Euler characteristic, consider a polyhedron whose spatial configuration is determined by its vertices, edges, and faces. It can be shown 5.1 | Automated methods (Lakatos, 1976) that if the polyhedron is convex, the number of verti- ces (V), minus the number of edges (E), plus the number of faces (F), is Biases arising from manual inspection can be minimized through auto- always equal to 2; this quantity is the Euler characteristic, that is, mated quality control methods. In the UK Biobank, for example, a χ = V − E + F. If the polyhedron is crossed by a single hole, χ is supervised learning classifier identifies problematic images with decreased by 1; if crossed by two holes, decreased by 2; if hollow, χ is acceptable accuracy (Alfaro-Almagro et al., 2018). The UK Biobank, increased by 1. More generally, for every hole that crosses a polyhe- however, benefits from the fact that data collection is limited to only dron, its χ is decreased by one, whereas for every hollow, it is three sites, all of which use identical equipment (Miller et al., 2016). In increased by one. The Euler characteristic is well-known in neuroim- ENIGMA, data come from many sites, with MRI scanners from differ- aging as a key metric for multiple-testing correction using the random ent vendors and models, with different field and gradient strengths, field theory (RFT; Worsley et al., 1996). Here, however, it serves an different coils, acquisition sequences, and software versions. Using a entirely different purpose: it acts as a metric to quantify topological quality control classifier with such heterogeneous data is challenging deviation of the initial cortical surface from a sphere, as an increas- (Chen et al., 2014; Focke et al., 2011; Han et al., 2006; Jovicich ingly large number of holes in the initial surface generates an increas- et al., 2006), although methods with good performance have been ingly negative Euler characteristic. As these values become more proposed (Klapwijk, van de Kamp, van der Meulen, Peters, & negative, the more likely it is that the original T1-weighted scans had Wierenga, 2019). low quality in ways that negatively impacts the surface reconstruction. Tools such PCP-QAP20 (Zarrar et al., 2015) and MRIQC21 FreeSurfer treats such holes as topological defects and corrects them (Esteban et al., 2017) compute a host of image quality metrics that automatically to create a cortical surface that reaches a χ = 2 (Fischl, consider signal, noise, image smoothness and contrast, as well as spe- Liu, & Dale, 2001). However, initial surfaces that have too many cific artifacts (Atkinson, Hill, Stoyle, Summers, & Keevil, 1997; Die- defects are less likely to be ever usable, even after topology trich, Raya, Reeder, Reiser, & Schoenberg, 2007; Ganzetti, correction. Wenderoth, & Mantini, 2016; Magnotta & Friedman, 2006; Mortamet The Euler characteristic was found to be highly correlated with et al., 2009). In particular, MRIQC operates on data organized manual quality ratings, discriminating accurately unusable from usable according to BIDS, and produces detailed reports of these metrics. scans, and outperforming other data quality measures (Klapwijk This tool does not, however, classify images as having high or low et al., 2019). However, the precise threshold to be applied to χ quality; instead, it provides an interface for a rater to make that deter- remains unknown when deciding whether a surface is usable or not; mination based on the computed quality metrics and possibly other such a threshold may be site or scanner specific. Moreover, it is not features; these metrics may, in turn, be used to train a classifier. Such currently known whether, as a general rule, the Euler characteristics classification, however, can be difficult to generalize, given the diver- for each brain hemisphere should be combined as their mean, or the sity of data from multiple sites (Esteban et al., 2017). Even so, derived worst (minimum, most negative) of the two, nor whether other met- metrics may be insufficient to predict successful generation of cortical rics related to surface topology could be helpful for quality assess- surfaces and segmentation of subcortical structures with FreeSurfer, ment. For subcortical structures, specific quality metrics are currently from which image-derived measurements of interest are often com- missing from the literature. puted. Notwithstanding these considerations, it is good practice to investigate quality using this kind of tool, which includes boxplots (Figure 2), and mosaics that show multiple slices color-coded so as to 5.3 | Manual edits highlight potential defects. The output from these tools are useful to assist in flagging images that, even if successful at FreeSurfer Image processing pipelines may allow manual edits when automated processing, may require specific decisions whether or not they should approaches fail to generate processed images of desirable quality. This remain in the sample. Moreover, these tools provide summary metrics is also the case with FreeSurfer, whereby the user can employ “con- that can be returned to the contributing sites, where local researchers trol points” to establish final cortical surface placements; 264 ZUGMAN ET AL. F I G U R E 2 Example screenshot of a report of image quality for the subjects of one site. Box plots of various metrics are shown. The report is produced by the tool MRIQC, available, along with documentation that details all the metrics (many more than shown in the figure), at https:// mriqc.readthedocs.io segmentation also can be hand edited to exclude nonbrain tissue and harnessed. As an example, a recent ENIGMA meta-analysis using data fix mislabeled regions. Different ENIGMA working groups have from 51,665 subjects identified 187 loci influencing cortical surface decided differently on whether or not to do manual editing, as the area and 12 others influencing thickness (Grasby et al., 2020); in process is time consuming and requires expert knowledge. Crucially, another example, the recent UK Biobank analysis used 3,144 imaging- while manual editing may improve validity of measurements, it intro- derived traits (Elliott et al., 2018) to find 148 replicable clusters of duces variance to the data unrelated to the images themselves, but associations between single nucleotide polymorphisms and these related to the manual operator; multiple operators potentially com- traits. Not all imaging traits are sufficiently well defined and stable to pound the problem. Taken to an extreme, such undesired variance can allow reliable quantification for ENIGMA meta or mega-analyses, reduce power, though research has found that such editing may have though. While the problem can be partially mitigated in mega-analyses little impact on the final results, in either a beneficial or deleterious that use common processing schemes, stable, reliable measurements manner (McCarthy et al., 2015). should be the first line of research. Region-based or vertex-based measures of cortical thickness, cortical surface area, and cortical and subcortical volumes are easily obtained, and measurement workflows 6 | MEASUREMENTS are established across multiple research sites (Hibar et al., 2015; Stein et al., 2012; Thompson et al., 2014). Diffusion-weighted imaging also Imaging generates a myriad of measurements. Analyses can reveal allows measurements that are robust to variations on processing pipe- genetic and environmental influences on healthy and pathological var- lines, and workflows for ENIGMA have been developed (Jahanshad iability in the human brain, providing great potential currently not fully et al., 2013; Kochunov et al., 2015). Likewise, a resting-state ZUGMAN ET AL. 265 functional MRI processing pipeline was proposed recently for use in These considerations, however, do not imply superiority of ROI- ENIGMA meta-analyses (Adhikari et al., 2018). based measurements over voxelwise or vertexwise approaches. While The goal of these pipelines is to ensure consistency in methods noisier, vertexwise and voxelwise data are typically smoothed, across sites, not affecting the relationship between the imaging mea- thereby increasing the signal-to-noise ratio while retaining localizing surements and their underlying biological processes. In effect, such power. Moreover, statistical power of ROI-based measurements is measurements may be influenced by a myriad of physiological and maximal when the space spanned by the true effects matches per- pathological processes. For cortical thickness and surface area, for fectly the borders of the ROI; otherwise, true signal is diluted within example, measurements depend to some extent on neuronal and glial an ROI, or split across multiple ROIs. Multiplicity of testing, while cell volume and density, dendritic complexity, degree of myelination, more severe with voxelwise or vertexwise data, may not necessarily inflammatory processes, and other factors. Evidence has been compromise power: the tests are largely nonindependent and reported for changes in cortical thickness associated with learning methods to accommodate such nonindependence exist, both in para- (Zatorre, Fields, & Johansen-Berg, 2012) and even with stimuli during metric (Worsley et al., 1996) and nonparametric cases (Winkler, Web- structural MRI data collection (Månsson et al., 2020); the latter find- ster, et al., 2016). Finally, it is not always obvious how to aggregate ing, if confirmed, could indicate that information about what the sub- measurements for ROIs, nor what the ROIs should be. For example, jects viewed would need to be considered as a confound at the time while the surface area of an ROI can be trivially obtained by summing of the statistical analysis. together the areas assigned to all vertices within that region, thickness Furthermore, the scale with which measurements are obtained within the same ROI could be computed either as an average of all does not necessarily correspond to the scale in which linear effects vertices, or as a weighted average using the areas of the vertices as manifest; such linear effects constitute the backbone of most statisti- weighting factor; for functional MRI, aggregate measurements could cal brain-imaging analyses as encapsulated in the general linear model be the simple average, or the first principal component. Moreover, a (GLM). The GLM assumes that modeled factors (e.g., diagnostic group, host of different parcellation schemes exist (Craddock, James, age, and sex) possess additive effects over the dependent variable Holtzheimer, Hu, & Mayberg, 2012; Desikan et al., 2006; Destrieux, (e.g., an imaging-derived measurement); this may not always hold. Fischl, Dale, & Halgren, 2010; Glasser et al., 2016; Ji et al., 2019; Mag- Some notable examples include fine-resolution area of the cortex, gioni, Tana, Arrigoni, Zucca, & Bianchi, 2014; Power et al., 2011; which follows a lognormal distribution potentially reflecting exponen- Schaefer et al., 2018; Tzourio-Mazoyer et al., 2002; Yeo et al., 2011), tial influences; fractional anisotropy of water diffusion, a quantity based on various macroscopic, microscopic, and functional aspects of bounded between 0 and 1, also could be considered not a sum of mul- the brain, none of which is clearly superior to others for all possible tiple small effects, nor functional connectivity assessments bounded investigations (Arslan et al., 2018; Brett, Johnsrude, & Owen, 2002; between −1 and 1. Cases such as these may be accommodated Messé, 2019). For instance, the use of an anatomical parcellation in through the use of a data transformation, such as logarithmic, power, fMRI studies might hide possibly relevant functional inhomogeneities Fisher's r-to-z, logit, or probit transformations; generalized linear within each cluster (Maggioni et al., 2014). models and nonparametric statistics can also be considered. Pooling raw IPD for mega-analysis creates many data analytic opportunities. Since the coordinating facility has access to all data, mass-univariate analyses are possible without the constraints imposed 6.1 | Choice of resolution by the limited data exchanges of meta-analyses. Mass-univariate methods support easier, more reliable forms of vertexwise/voxelwise Researchers need to consider whether imaging analyses should use analyses, performed by processing all data in an identical manner, measures obtained at every point of an image (e.g., voxelwise or regardless of site provenance, an approach that has already been used vertexwise data) or aggregate measures computed over regions of in ENIGMA (Wang et al., 2019). However, this process still can be interest or parcellations, broadly termed as “ROIs.” Although computationally difficult. FreeSurfer default surface data, for example, vertexwise analyses have been performed in recent ENIGMA research uses 163,842 vertices per hemisphere for between-subject compari- (Chye et al., in press; Ho et al., 2020), most previous ENIGMA studies sons, which is denser than the number of voxels that pass through the used meta-analyses. In these cases, an ROI-based approach is more pia mater or the interface between gray and white matter in a typical robust to small deviations from a common image registration scheme. MRI scan, given the cortical convolutions; moreover, data are usually Moreover, voxelwise and vertexwise measurements represent small smoothed, further lowering the effective resolution. Computational pieces of tissue in relation to the resolution inherent to the equipment savings that do not substantively sacrifice localizing power may be or scanning sequence. As such, these measures are intrinsically noisier accomplished by downsampling the surface. In the case of FreeSurfer, than ROI-based quantities. Furthermore, because the number of this downsampling can be done by using an icosahedron recursively voxels/vertices is usually many times larger than the number of ROIs subdivided fewer times (n) than the default 7 (Winkler et al., 2012, under potential consideration, their use is computationally more appendix A) as the target for interpolation. Data from vertices not intensive, and leads to an exacerbation of the multiple testing explicitly included are still represented when smoothing is applied problem. before interpolation, thus before downsampling. The number of 266 ZUGMAN ET AL. F I G U R E 3 Surface reconstructions of the cortex of the right hemisphere based on different resolutions of a recursively subdivided icosahedron. The default in FreeSurfer uses n = 7 recursions, resulting in a total of 163,842 vertices. Considerable computational savings can be obtained with lower resolutions (such as with n = 4 or 5) without substantial losses in localizing power. V, number of vertices; E, number of edges; F, number of triangular faces vertices is given by V = 10  4n + 2 if the grid is based on an icosahe- effects of interest; neglecting such nuisance effects can reduce power dron; using n = 4 or 5 still allows good cortical coverage (Figure 3). or generate false positives and low reproducibility (Baggerly, Coombes, & Neeley, 2008; Leek et al., 2010). These confounds can be accommodated at the time of the statistical modeling and analysis, at 6.2 | Harmonization the penalty of increasing model complexity, or the data may be modi- fied before analysis so as to remove such unwanted effects. In a mega-analysis, pooling data from numerous cohorts requires ComBat (“combining batches”) is such an approach, that allows addressing nuisance factors. Site, scanner and cohort-specific effects harmonization of data across sites. The method originated in genetics of no interest can be manifest as effects larger than diagnosis or other for correcting batch effects in microarrays, and is described in detail in ZUGMAN ET AL. 267 (Johnson, Li, & Rabinovic, 2007). In brief, ComBat incorporates sys- to an idealized “population of sites”, but instead to the actual popula- tematic biases common across voxels/vertices, under the mild tion. Thus, between-site variability is a nuisance that should either be assumption that phenomena resulting in such “batch” effects modeled by including random intercepts to accommodate different (e.g., site, scanner, and/or cohort effects) affect voxels/vertices in sim- site means, or be addressed through data harmonization, as discussed ilar ways (e.g., stronger mean values, higher variability). In the method, above. Effects of interest, such as differences between individuals location (additive) and scale (multiplicative) model parameters that with a specific condition and comparison individuals, can be assumed represent these batch effects are estimated. This estimation is done to be fixed across sites (thus, would be modeled as a single regressor, by pooling information across voxels or vertices from participants i.e., with fixed slopes), or assumed to vary across sites (thus, would be from each site so as to shrink such unwanted effects toward an over- modeled with multiple regressors, i.e., with random slopes), thus all group effect (i.e., across batches and voxels/vertices). These esti- implying the possibility of an interaction of site by effect of interest. mates are then used to adjust the data, robustly discounting The latter would accommodate, for example, site differences due to unwanted effects. Variability of interest or related to known nuisance clinical characteristics or recruitment setting. Unwanted effects other or confounds (e.g., age or sex) can be retained. In brain imaging, the than the intercept can be modeled either considering fixed or random approach has been an effective method to harmonize diffusion tensor slopes, the consideration being left on a per variable basis. For exam- imaging data (Fortin et al., 2017), cortical thickness measures (Fortin ple, age effects may be modeled using fixed slopes if all sites have par- et al., 2018), rest and task-based functional MRI (Nielson et al., 2018), ticipants within similar age ranges, but using random slopes if some and functional connectivity (Yu et al., 2018). ComBat has been used in sites have only young participants whereas others have only elderly, ENIGMA studies (Hatton et al., in press; Villalón-Reina et al., in press), as age is often not expected to have a linear effect across lifespan. although it has been argued that it leads to similar results as random effects linear regression (Zavaliangos-Petropulu et al., 2019). Which statistical harmonization model is optimal remains an active discussion 7.2 | Confounds at the time of this writing. Unwanted data variability may arise due to procedural factors includ- ing site or scanner features, or due to factors that affect both depen- 7 | STATISTICAL ANALYSIS dent and independent variables. Variables representing the former case are termed nuisance; those representing the latter, confounds. Statistical analyses can proceed once data have been processed, and Variables such as age or sex may be nuisance in some analyses or con- measurements obtained and possibly harmonized. Such analyses esti- founds in others, depending on the relationship between these vari- mate the effects of interest and compare them to expected observa- ables and the other variables studied; here we broadly call nuisance tions should there be no real effect to compute a p-value. It is at the and confound variables covariates. stage of the statistical analysis that the differences between meta- The large sample size of ENIGMA increases statistical power in and mega-analysis become most pronounced. general, however, this may result in erroneous labeling confounding effects (Smith & Nichols, 2018); ignoring such confounds may reduce power or identify spurious associations. Addressing these concerns can 7.1 | Fixed versus random effects be challenging, as decisions regarding confounding variables affect interpretation of the relationship between dependent and independent For all cases discussed in Figure 1, analyses may assume that true variables (Gordon, 1968; Lord, 1967). For example, if a confounding var- effects are fixed (constant) across sites, and therefore any differences iable causes at least part of the variation observed in the imaging data in effects among sites are solely due to random experimental error, or across participants and in the variation of the independent variable may assume that the true effects themselves may be random (i.e., it is a collider), adjustment for the undesired effect induces a false (i.e., varying) across sites. For meta-analyses without access to IPD, association (Berkson, 1946; Luque-Fernandez et al., 2019; Pearl, 2009, the above distinction between fixed and random effects holds rela- chapter 6), which can happen in either direction (positive or negative). tively without ambiguity, and distinct methods to summarize literature Moreover, controlling for poorly reliable measures may not findings for either of the two cases exist (Borenstein, Hedges, completely remove their putative effects, leading to false conclusions Higgins, & Rothstein, 2009). For other cases, unfortunately, these about effects (J. Westfall & Yarkoni, 2016). While this is a greater con- terms have multiple meanings that sometimes conflict cern for social and psychological constructs that often are measured (Gelman, 2005). For research using IPD, less ambiguous definitions with relatively low reliability, the same can apply to imaging measure- apply to slopes and intercepts, which can be treated as constant (thus, ments. Examples include segmentation of structures where tissue fixed) or allowed to vary (thus, random) across sites. This distinction contrast is minimal, or for structures that are small for the image reso- between fixed and random becomes then an attribute not of the sta- lution; for functional MRI, false conclusions can occur through effects tistical model, but of each independent variable. of signal fluctuations that are poorly associated with task performance As for the level of inference, in the case of ENIGMA, the selection or of weak functional connectivity among regions. All such measures of sites is seldom a random quantity, and generalization is sought not can produce variables that, if used as confounds, may increase the 268 ZUGMAN ET AL. chance of false positives. Perhaps counterintuitively, here too large assumptions that are sometimes difficult to justify. For example, data sample sizes may exacerbate the problem. have to be assumed to be independent and normally distributed with A special kind of confound in brain imaging is a composite measure- identical variances after all nuisance variables and confounds have ment formed by pooling together values of all voxels/vertices or regions been taken into account. These assumptions may hold for some ana- of interest, with the goal of discounting unwanted global effects. For lyses, but not for others. When the variety of imaging modalities pos- example, in a vertexwise analysis of surface area, it might be of interest to sible for ENIGMA studies is considered, these assumptions cannot consider the total cortical surface area as a confounding variable. Like- hold for all of them. The consequence is that results will be incorrect wise, for studies of subcortical volume, total brain size—or a related quan- in at least some cases. Nonparametric tests, such as permutation tests, tity, the intracranial volume (Buckner et al., 2004)—can be considered a on the other hand, require very few assumptions about the data prob- confound; for cortical thickness, the average thickness across the cortex; ability distribution, and therefore can be applied to a wider variety of for functional MRI, at the subject level, a measurement of global signal, situations than parametric tests. For permutation tests, the only key though controversial, might be considered in a similar manner (Murphy & assumption is that any random instantiation of permuted data must Fox, 2017). The rationale for inclusion of a global measurement as a be as likely to have been observed as the original, unpermuted. In regressor within the model stems from interest in enhancing the localizing other words, the data must be exchangeable. If exchangeability holds, power afforded by imaging methods, and reducing sources of noise that permutation tests are exact, in the sense that the probability of affect measures globally. From this perspective, the scientist seeks to observing a p-value smaller than a predefined significance level α is α learn where, specifically, in the brain some phenomenon may occur. In itself when there are no true effects (Holmes, Blair, Watson, & this context, arguably, global effects would be of lesser interest, unless a Ford, 1996; T. E. Nichols & Holmes, 2002; Winkler, Ridgway, Web- research hypothesis is specifically about them. In addition, for functional ster, Smith, & Nichols, 2014). MRI, some sources of noise, such as movement and respiration, result in For ENIGMA mega-analyses, permutation tests can pose practical artifactual global signal changes, and so removal of the global signal is also challenges, though. Large sample sizes and the multiplicity of sites, an effective means of reducing artifacts (Ciric et al., 2017). combined with modeling that include random slopes and random What makes these confounds special is that, being composites of intercepts for covariates, leads to large design matrices that can be all other local (voxelwise/vertexwise) or regional quantities, they are slow to process repeatedly as needed for permutations. Moreover, almost certainly correlated with these measurements, and thus, are unless data have been harmonized in ways that accommodate poten- likely to also be associated with variables of interest in the model if tial different variances across sites, statistics that are robust to these are associated with the local or regional measurements. These heteroscedasticity (DiCiccio & Romano, 2017; Guillaume, Hua, global variables are more likely to impact results where local or regional Thompson, Waldorp, & Nichols, 2014; Winkler et al., 2014) can like- effects of interest are present, even more so if these are widespread wise add to the computational burden; here, permutations may be across the brain. Options for taking into account such global effects in restricted to blocks of exchangeable observations that have been col- the statistical analysis have been studied (Andersson, 1997; Barnes lected within each site or within scanner. Another increase in compu- et al., 2010; Nordenskjöld et al., 2013; Sanfilipo, Benedict, Zivadinov, & tational expense occurs if more powerful, yet nonstandard test Bakshi, 2004). The main approaches are: (a) convert each local or statistics, such as pseudo-t (T. E. Nichols & Holmes, 2002), or spatial regional measurement into a proportion over the global quantity; statistics such as cluster extent, cluster mass, or threshold-free cluster (b) residualize the dependent variable with respect to the global; and enhancement (TFCE) (Smith & Nichols, 2009) are used. In all these (c) include the global in the model. Among these three, the latter option cases, speed can be increased using fast, parallel implementation of should always be favored as it accounts for effects that the con- permutation algorithms (Eklund, Dufort, Villani, & Laconte, 2014), or founding variable may have over both dependent and independent vari- using accelerations based on various mathematical and statistical ables. The least preferable is the proportion method (a), one of the properties of these same tests (Winkler, Ridgway, et al., 2016), reasons being that noisier (unreliable) measurements compromise the or both. measurements to a much greater extent than the others. If confounding variables are meant to be included in the model, it is often appropriate, considering all the above, to present results with 7.4 | Multiple testing and without these variables in the model (Hyatt et al., 2020; Simmons et al., 2011). Ideally, these would also be corrected for multiple test- As with any imaging experiment that uses one statistical test per ing, as the number of opportunities for falsely significant results has imaging element (voxel, vertex, ROI), correction for multiple testing is now doubled (see below more on multiple testing). necessary (T. Nichols & Hayasaka, 2003). For parametric inference, and under a series of additional assumptions, it is possible to control the familywise error rate (FWER) using the RFT (Worsley et al., 1996); 7.3 | Inference methods and software exist for both voxelwise and vertexwise data. However, this method cannot be used for ROIs, as these cannot be Choices for inference can be broadly divided into parametric and non- represented as a regular lattice, or for voxelwise data that do not parametric. Parametric methods are computationally faster but require meet all the assumptions of the theory, such as tract-based spatial ZUGMAN ET AL. 269 statistics (Smith et al., 2006). A valid approach for all these cases, but and/or p-values in logarithmic scale become then necessary, one per that controls a different error quantity, is the false discovery rate site, as well as for the overall results. (FDR) (Benjamini & Hochberg, 1995; Genovese, Lazar, & Nichols, 2002). For permutation inference, correction for multiple testing that controls the FWER can be accomplished in a straightfor- 8.1 | Authorship ward manner for all the above cases using the distribution of the max- imum statistic obtained across all tests in each permutation Given the large number of involved sites and investigators, authorship (P. H. Westfall & Young, 1993). Uncorrected permutation p-values of published reports are an important aspect of ENIGMA projects. can also be subjected to FDR correction. While there are no enforceable rules to determine the authorship of a Correction should consider not only the multiplicity of imaging scientific paper, a number of organizations have provided guidelines elements as voxels or ROIs, but also the multiple imaging-derived and recommendations intended to ensure that substantial contributors measurements eventually tested in the same ENIGMA study are credited as authors; for a review, see (Claxton, 2005). One such (e.g., cortical thickness and cortical area), as well as multiple hypothe- organization is the International Committee of Medical Journal Editors ses formulated in terms of contrasts of parameters of the model (ICMJE, also known as the “Vancouver group”), which recommends that (Alberton, Nichols, Gamba, & Winkler, 2019), or multiple models for authorship are based on the following four criteria: (a) substantial contri- the same data, for example, with and without a global measurement butions to the conception or design of the work; or the acquisition, as confounding variable. Failing to consider these issues exposes a analysis, or interpretation of data for the work; (b) drafting the work or study to the risk of excess false positives. Correcting across multiple revising it critically for important intellectual content; (c) final approval tests for multiple hypotheses in the same study is challenging with of the version to be published; (d) agreement to be accountable for all parametric tests given the existing but invariably unknown depen- aspects of the work in ensuring that questions related to the accuracy dence structure among tests; Bonferroni correction, while valid, is or integrity of any part of the work are appropriately investigated and unduly conservative given that dependence. Correction using the dis- resolved. It is recommended that all four conditions are satisfied. More- tribution of the maximum statistic, assessed via permutations, solves over, all authors should be able to identify which co-authors are respon- the problem, regardless of the dependence structure, yielding exact sible for other specific parts of the work (ICMJE, 2019). results, thus that are neither conservative (thus not less powerful Most ENIGMA studies have tried as much as possible to adhere to given the multiplicity of tests) nor invalid. these recommendations. For example, while not all investigators from all sites may contribute directly to the planning or execution of the meta- or mega-analysis, intellectual formulation of research hypotheses and study 8 | REPORTING RESULTS design that led to the data collection at a given site often are the same for which the data or results were pooled across multiple ENIGMA sites, Classical meta-analyses results are often reported with the aid of for- which, together with data collection itself, satisfies the first criterion. Col- est plots (Borenstein et al., 2009), which show effect sizes and confi- laborative, real-time text editing tools, such as Google Docs,22 Authorea,23 dence intervals for each study separately (or for each site in the case and Overleaf,24 allow many authors to work simultaneously on the same of ENIGMA), along with a combined effect size that considers the document, editing and providing each other with comments, thus satisfy- effects from all studies after some sensible weighting. ENIGMA stud- ing the second and third criterion. The fourth recommended criterion may ies that used meta-analyses adopted a similar approach where possi- be satisfied implicitly through the communication established during the ble, for example, when imaging metrics were collapsible into single editing process, by assenting upon the publication of a preprint, at a time numbers, such as asymmetry (Guadalupe et al., 2017; Kong in which an author may choose to opt out before submission to a journal et al., 2018) or indices for specific structures (Hibar et al., 2015; Stein and where eventual rectifications are more complex, or by signing a form et al., 2012). For mega-analyses, while such plots may be of lesser in which sugh agreement is made explicit. ENIGMA working group mem- value as the ultimate conclusions come from pooling all IPD into a sin- bers who do not satisfy the Vancouver criteria may be presented with gle analysis, reporting forest plots may still be helpful for showing alternatives, such as (a) be named under a consortium author if they potentially distinct effects at each site, as well as identifying outlier worked on overarching conception of the project but not on the specific sites and qualitatively revealing how disperse the data are. However, paper, (b) be named in the Acknowledgments section of the paper, or for this purpose, the mega-analysis may need to be broken down into (c) not named; in the absence of guidelines, participating members are free separate analyses, one per site, or contrasts tested separately for each to choose what best represents their contribution. site in the case of random slopes (which accommodate interactions of effects of interest by site). While these two approaches are equivalent if there are no fixed slopes or fixed intercept anywhere in the model, 9 | M E G A - A N A L Y S I S I N TH E E NI G M A - G A D running analyses separately for each site is computationally less inten- GR O UP sive (and can be done in parallel in a straightforward manner). Voxelwise and vertexwise results cannot, however, be feasibly shown Having discussed the above, we are now in a position to better with forest plots, and the usual, color-coded maps for effect sizes describe the specifics of the ENIGMA-GAD analyses. In this group, 270 ZUGMAN ET AL. sites were contacted based on their publication and funding record as to give a mark “pass” or “fail.” This process would require an excess using imaging data of subjects with a history of anxiety disorders, and of person-hours not available to the group. Instead, a semi-automated who could meet criteria for GAD. Virtually all sites that were con- method was used. First, a script27 to quickly allow visual inspection of tacted and that did have structural imaging data were able to partici- FreeSurfer cortical surfaces and subcortical segmentations of many pate. While imaging offers a large range of measurements, the group subjects in a single report page was used (Figure 4). One of the bene- began by examining structural, T1-weighted imaging scans and per- fits is that, by showing many subjects at the same time, the page per- forming an analysis based on FreeSurfer. These choices reflected the mits one to quickly learn how good quality surface and segmentations popularity of these scans, which are nearly universally collected, should look like. Reconstructions with clear defects that grossly regardless of other imaging modalities that each site may have used affected anatomy were marked for exclusion by a single researcher for their investigations. Furthermore, FreeSurfer-based pipelines and for all the data; all other reconstructions were, at this stage, not yet quality control protocols were already available from previous marked for exclusion. Second, from the results of FreeSurfer, the ENIGMA studies. Having these as a starting point facilitated the Euler characteristic of the surfaces before topology was obtained (it is establishment of a new working group. stored in the log file “recon-all.log”), as well as the number of vertices As described above, IPD were made available by the participating of these surfaces. Receiver operating characteristic curves relating sites to the coordinating facility—in this case, the National Institute of variation on the Euler characteristic threshold and the ability to reject Mental Health (NIMH), part of the US National Institutes of Health the surfaces marked for exclusion after visual inspection were con- (NIH)—such that a mega-analysis could be conducted. While the supe- structed so as to identify site-specific thresholds. Ultimately, the Euler riority of mega-analyses is established, another factor was the propor- characteristic was replaced by the ratio between the Euler character- tion of sites contributing IPD: nearly all sites preferred to send raw, istic and the number of vertices in the surfaces before topology cor- anonymized imaging scans, as opposed to individual level processed rection, as this measure had a better ability to discriminate between data or simply summary results. Another early decision involved good and bad surfaces in this dataset. Finally, the identified threshold preregistering the analytic plan, which was deposited at the Open Sci- was used to determine which subjects would be included and which 25 ence Foundation, where it remains publicly accessible. The plan was would not, on a per site basis. registered after data had been received by the NIMH and processed Measurements considered for analyses, as indicated in the pre- using FreeSurfer, such that sample sizes were known (eventually registration and as in previous ENIGMA studies, included cortical more sites could contribute data after overcoming institutional bar- measurements of thickness and surface area for each of the parcels of riers; these were included in the analysis), and before statistical ana- the Desikan–Killiany atlas (Desikan et al., 2006), as well as volumes of lyses. Data from some sites may have been part of previous subcortical structures. Cortical vertexwise thickness and surface area publications; the team analyzing the data at the coordinating facility were also measured, and downsampled to the resolution of an icosa- did not deliberately check the existence of previous results, nor hedron recursively subdivided four times, with 2,562 vertices per whether previous analyses on subsets used similar or different hemisphere. Because sites differed widely in variables such as age, methods. Instead, inclusion and exclusion criteria for data were modeling age with random slopes (with additional quadratic effects) based on diagnoses given to the subjects in relation to the research seemed more appropriate than merely assuming that, across all ages hypotheses, and the ability of the sites to make their data available and sites, age effects would be exactly the same. Models with and to the coordinating facility. In the preregistered plan, research without a global measure (total surface area, average thickness, and hypotheses were specified, inclusion and exclusion criteria were intracranial volume) were considered. Correction for multiple testing defined, dependent and independent variables were indicated, and used the distribution of the maximum statistic, assessed via permuta- statistical methods outlined. Exploratory analyses that were not the tions. ComBat was not used for the main analyses; instead, scanner- main focus of the study were also listed. specific effects were modeled (random intercepts) and a test statistic A de-identification agreement was signed between the NIMH robust to heteroscedasticity was used, along with variance groups and each participating center whereby any eventual identifiable data (one per site) and exchangeability blocks. ComBat is, however, being would not be disclosed or requested by any of the parties. De- assessed with the same data as a potential option for future studies; identified data were transferred using the Globus service to a stor- results will be reported opportunely. Statistical analysis for this mega- age partition of the high performance computing (HPC) systems of analysis used the tool Permutation Analysis of Linear Models the NIH. Each site had its own endpoint for transfer, such that no (PALM).28 At the time of this writing, the analysis is being finished and information of any kind could leak from one contributing site to the manuscript is being prepared for publication (Harrewijn et al., in another. prep.). Authorship, like with the present paper, was defined according The received data were organized according to BIDS, passed to Vancouver criteria, generally with early career investigators, mem- through an initial quality check using MRIQC, and processed using the bers of the coordinating facility and who worked directly with the FreeSurfer 6.0.0. Given the large number of subjects, it was not viable data handling and the bulk of the writing appearing first, and with lead 26 to follow the ENIGMA-QC protocol, according to which every corti- investigators appearing last; in between, the contributing sites in cal parcel and every subcortical region of every participant would alphabetic order, and, within each site, early investigators appearing need to be visually inspected and annotated where usable or not, so first; lead investigators last. ZUGMAN ET AL. 271 F I G U R E 4 Example report pages with multiple views of the cortical surfaces (front) and slices of subcortical volumes (back). Pial surfaces are shown, but inspection can use white and inflated; slices with subcortical volumes can be complemented with surface overlays. The script that generates these pages uses FreeSurfer scripting to automate the operation of the tools “tkmedit” and “tksurfer,” and is available at https:// brainder.org 10 | CO NC LUSIO N Gabrielle F. Freitag, Daniel S. Pine, and Anderson M. Winkler are supported by NIH through ZIA-MH002781 and ZIA-MH002782. Paul This overview described the analytic choices across the various stages M. Thompson is supported in part by NIH grant U54-EB020403. Katy of an ENIGMA mega-analysis, setting out the reasoning behind these E. Werwath received support from the NIH Office of Intramural Train- choices. Aspects related to data protection and privacy, and how to ing and Education (OITE). Nynke A. Groenewold was supported by a handle confounds, along with other challenges that inevitably occur Claude Leon Postdoctoral Fellowship. Nick J. A. Van der Wee and when large-scale data from multiple sites are analyzed were also dis- Dan J. Stein are supported by the EU 7th Framework Marie Curie cussed. The various choices made by ENIGMA-GAD when facing each Actions International Staff Exchange Scheme grant “European and of the discussed topics were presented. The hope is that the resulting South African Research Network in Anxiety Disorders” (EUSARNAD). survey of these practical considerations will be useful to others Dan J. Stein is supported by the SA Medical Research Council. Ulrike embarking on similar multi-site neuroimaging studies, especially those Lueken was funded by the Deutsche Forschungsgemeinschaft (DFG, integrating data across multiple countries and data modalities. German Research Foundation), Projektnummer 44541416, TRR 58. Katja Beesdo-Baum was supported by the Federal Ministry of Educa- ACKNOWLEDGMENTS tion and Research (BMBF, grants 01ER1303 & 01ER1703). SHIP The authors thank all the many research sites worldwide who consid- (Study of Health in Pomerania) is part of the Community Medicine ered contributing data to the ENIGMA-Anxiety/GAD project. André Research net of the University of Greifswald, Germany, which is Zugman, Anita Harrewijn, Elise M. Cardinale, Hannah Zwiebel, funded by the Federal Ministry of Education and Research (grants 272 ZUGMAN ET AL. 11 no. 01ZZ9603, 01ZZ0103, and 01ZZ0403), the Ministry of Cultural https://box.com 12 Affairs and the Social Ministry of the Federal State of Mecklenburg- https://aws.amazon.com 13 West Pomerania; MRI scans in SHIP have been supported by a joint https://drive.google.com grant from Siemens Healthineers, Erlangen, Germany and the Federal 14 https://azure.microsoft.com State of Mecklenburg-West Pomerania. Jared A. Nielsen, Randy Buck- 15 http://www.linux-kvm.org ner, and Jordan W. Smoller were supported by NIH grant 16 https://www.virtualbox.org R01MH101486. Gretchen Diefenbach and Michal Assaf received sup- 17 https://www.vmware.com port from the Hartford HealthCare Research Funding Initiative grant 18 https://www.docker.com (#129522). Eleonora Maggioni received support from Fondazione 19 https://sylabs.io/singularity Cariplo, grant n 2016-0908. Michael Perino received support from 20 http://preprocessed-connectomes-project.org NIH grant T32MH100019 (Developmental Neuroscience and Child 21 https://mriqc.readthedocs.io Psychopathology). Chad M. Sylvester received support from NIH 22 https://docs.google.com grant K23MH109983. 23 https://authorea.com 24 https://overleaf.com DIS CLOSURES 25 https://doi.org/10.17605/osf.io/yxajs Paul M. Thompson received partial grant support from Biogen, Inc., 26 http://enigma.ini.usc.edu (Boston, MA) for research unrelated to this manuscript. Anderson 27 http://brainder.org M. Winkler received in the past support from the Government of Bra- 28 https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/PALM zil through the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq: 211534/2013–7). Straube was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Founda- RE FE RE NCE S tion), Projects C07/C08, SFB-TRR 58. Dan J. Stein has received Abramian, D., & Eklund, A. (2019). Refacing: reconstructing anonymized research grants and/or honoraria from Lundbeck and Sun. Hans facial features using GANS. IEEE 16th International Symposium on Bio- medical Imaging, Venice, Italy, pp. 1104–1108. J. Grabe has received travel grants and speakers honoraria from Adhikari, B. M., Jahanshad, N., Shukla, D., Turner, J., Grotegerd, D., Fresenius Medical Care, Neuraxpharm, Servier, and Janssen Cilag as Dannlowski, U., … Kochunov, P. (2018). A resting state fMRI analysis well as research funding from Fresenius Medical Care. pipeline for pooling inference across diverse cohorts: An ENIGMA rs- fMRI protocol. Brain Imaging and Behavior, 13(5), 1453–1467. https:// doi.org/10.1007/s11682-018-9941-x DATA AVAI LAB ILITY S TATEMENT Alberton Bianca A. V., Nichols Thomas E., Gamba Humberto R., Data sharing is not applicable to this article as no new data were cre- Winkler Anderson M. (2020). Multiple testing correction over con- ated or analyzed in this study. trasts for brain imaging. NeuroImage, 216, 116760. http://dx.doi.org/ 10.1016/j.neuroimage.2020.116760. Alfaro-Almagro, F., Jenkinson, M., Bangerter, N. K., Andersson, J. L. R., ORCID Griffanti, L., Douaud, G., … Smith, S. M. (2018). Image processing and André Zugman https://orcid.org/0000-0002-6126-7759 quality control for the first 10,000 brain imaging datasets from Janna M. Bas-Hoogendam https://orcid.org/0000-0001-8982-1670 UKbiobank. NeuroImage, 166, 400–424. https://doi.org/10.1016/j. Ramiro Salas https://orcid.org/0000-0002-1105-566X neuroimage.2017.10.034 David Hofmann https://orcid.org/0000-0003-3494-634X Alfino, M., & Mayes, G. R. (2003). Reconstructing the right to privacy. Social Theory and Practice, 29(1), 1–18. https://doi.org/10.5840/ Carmen Andreescu https://orcid.org/0000-0003-3767-5127 soctheorpract20032915 Elena Makovac https://orcid.org/0000-0002-7018-1906 Ananthakrishnan, R., Chard, K., Foster, I., & Tuecke, S. (2015). Globus Massimo Filippi https://orcid.org/0000-0002-5485-0479 platform-as-a-service for collaborative science applications. Concur- Anderson M. Winkler https://orcid.org/0000-0002-4169-9781 rency and Computation: Practice and Experience, 27(2), 290–305. https://doi.org/10.1002/cpe.3262 Andersson, J. L. R. (1997). How to estimate global activity independent of ENDNOTES changes in local activity. NeuroImage, 6(4), 237–244. https://doi.org/ 1 10.1006/nimg.1997.0302 http://freesurfer.net 2 Arellano, A. M., Dai, W., Wang, S., Jiang, X., & Ohno-Machado, L. (2018). https://www.clinicaltrials.gov 3 Privacy policy and technology in biomedical data science. Annual https://www.who.int/ictrp Review of Biomedical Data Science, 1(1), 115–129. https://doi.org/10. 4 https://osf.io 1146/annurev-biodatasci-080917-013416 5 The complete and updated list can be found at https://ec.europa.eu. Arslan, S., Ktena, S. I., Makropoulos, A., Robinson, E. C., Rueckert, D., & 6 Parisot, S. (2018). Human brain mapping: A systematic comparison of https://www.veracrypt.fr parcellation methods for the human cerebral cortex. NeuroImage, 170, 7 https://www.ciphershed.org 5–30. https://doi.org/10.1016/j.neuroimage.2017.04.014 8 Atkinson, D., Hill, D. L. G., Stoyle, P. N. R., Summers, P. E., & Keevil, S. F. https://www.gnupg.org 9 (1997). Automatic correction of motion artifacts in magnetic reso- https://www.globus.org 10 nance images using an entropy focus criterion. IEEE Transactions on https://dropbox.com Medical Imaging, 16(6), 903–910. https://doi.org/10.1109/42.650886 ZUGMAN ET AL. 273 Baggerly, K. A., Coombes, K. R., & Neeley, E. S. (2008). Run batch effects Ciric, R., Wolf, D. H., Power, J. D., Roalf, D. R., Baum, G. L., Ruparel, K., … potentially compromise the usefulness of genomic signatures for ovar- Satterthwaite, T. D. (2017). Benchmarking of participant-level con- ian cancer. Journal of Clinical Oncology, 26(7), 1186–1187. https://doi. found regression strategies for the control of motion artifact in studies org/10.1200/JCO.2007.15.1951 of functional connectivity. NeuroImage, 154, 174–187. https://doi. Barnes, J., Ridgway, G. R., Bartlett, J., Henley, S. M. D., Lehmann, M., org/10.1016/j.neuroimage.2017.03.020 Hobbs, N., … Fox, N. C. (2010). Head size, age and gender adjustment Claxton, L. (2005). Scientific authorship—Part 2. History, recurring issues, in MRI studies: A necessary nuisance? NeuroImage, 53(4), 1244–1255. practices, and guidelines. Mutation Research/Reviews in Mutation https://doi.org/10.1016/j.neuroimage.2010.06.025 Research, 589(1), 31–45. https://doi.org/10.1016/j.mrrev.2004. Bas-Hoogendam, J. M., Groenewold, N. A., Aghajani, M., Freitag, G., 07.002 Harrewijn, A., Hilbert, K., … Stein, D. J. (2020). ENIGMA-anxiety work- Craddock, R. C., James, G. A., Holtzheimer, P. E., Hu, X. P., & ing group: rationale for and organization of large-scale neuroimaging Mayberg, H. S. (2012). A whole brain fMRI atlas generated via spatially studies of anxiety disorders. PsyArXiv. https://doi.org/10.31234/osf. constrained spectral clustering. Human Brain Mapping, 33(8), io/n62vc 1914–1928. https://doi.org/10.1002/hbm.21333 Beckmann, C. F., Jenkinson, M., & Smith, S. M. (2003). General multilevel Dale, A. M., Fischl, B., & Sereno, M. I. (1999). Cortical surface-based analy- linear modeling for group analysis in FMRI. NeuroImage, 20(2), sis. I. Segmentation and surface reconstruction. NeuroImage, 9(2), 1052–1063. https://doi.org/10.1016/S1053-8119(03)00435-X 179–194. https://doi.org/10.1006/nimg.1998.0395 Belias, M., Rovers, M. M., Reitsma, J. B., Debray, T. P. A., & IntHout, J. de Sitter, A., Visser, M., Brouwer, I., Cover, K. S., van Schijndel, R. A., (2019). Statistical approaches to identify subgroups in meta-analysis of Eijgelaar, R. S., … Vrenken, H. (2020). Facing privacy in neuroimaging: individual participant data: A simulation study. BMC Medical Research Removing facial features degrades performance of image analysis Methodology, 19(1), 183. https://doi.org/10.1186/s12874-019- methods. European Radiology, 30(2), 1062–1074. https://doi.org/10. 0817-6 1007/s00330-019-06459-3 Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., A practical and powerful approach to multiple testing. Journal of the Blacker, D., … Killiany, R. J. (2006). An automated labeling system for Royal Statistical Society. Series B (Methodological), 57(1), 289–300. subdividing the human cerebral cortex on MRI scans into gyral based Berkson, J. (1946). Limitations of the application of fourfold table analysis regions of interest. NeuroImage, 31(3), 968–980. https://doi.org/10. to hospital data. Biometrics, 2(3), 47–53. 1016/j.neuroimage.2006.01.021 Bischoff-Grethe, A., Ozyurt, I. B., Busa, E., Quinn, B. T., Fennema- Destrieux, C., Fischl, B., Dale, A., & Halgren, E. (2010). Automatic Notestine, C., Clark, C. P., … Fischl, B. (2007). A technique for the parcellation of human cortical gyri and sulci using standard anatomical deidentification of structural brain MR images. Human Brain Mapping, nomenclature. NeuroImage, 53(1), 1–15. https://doi.org/10.1016/j. 28(9), 892–903. https://doi.org/10.1002/hbm.20312 neuroimage.2010.06.010 Boedhoe, P. S. W., Heymans, M. W., Schmaal, L., Abe, Y., Alonso, P., DiCiccio, C. J., & Romano, J. P. (2017). Robust permutation tests for corre- Ameis, S. H., … Twisk, J. W. R. (2019). An empirical comparison of lation and regression coefficients. Journal of the American Statistical meta- and mega-analysis with data from the ENIGMA obsessive- Association, 112(519), 1211–1220. https://doi.org/10.1080/ compulsive disorder working group. Frontiers in Neuroinformatics, 12, 01621459.2016.1202117 102. https://doi.org/10.3389/fninf.2018.00102 Dickersin, K., & Rennie, D. (2003). Registering clinical trials. JAMA, 290(4), Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). 516–523. Introduction to meta-analysis. West Sussex, UK: John Wiley & Sons. Dietrich, O., Raya, J. G., Reeder, S. B., Reiser, M. F., & Schoenberg, S. O. Brett, M., Johnsrude, I. S., & Owen, A. M. (2002). The problem of func- (2007). Measurement of signal-to-noise ratios in MR images: Influence tional localization in the human brain. Nature Reviews. Neuroscience, 3 of multichannel coils, parallel imaging, and reconstruction filters. Jour- (3), 243–249. https://doi.org/10.1038/nrn756 nal of Magnetic Resonance Imaging, 26(2), 375–385. https://doi.org/10. Buckner, R. L., Head, D., Parker, J., Fotenos, A. F., Marcus, D., 1002/jmri.20969 Morris, J. C., & Snyder, A. Z. (2004). A unified approach for morpho- Dove, E. S. (2018). The EU general data protection regulation: Implications metric and functional data analysis in young, old, and demented adults for international scientific research in the digital era. The Journal of using automated atlas-based head size normalization: Reliability and Law, Medicine & Ethics, 46(4), 1013–1030. https://doi.org/10.1177/ validation against manual measurement of total intracranial volume. 1073110518822003 NeuroImage, 23(2), 724–738. https://doi.org/10.1016/j.neuroimage. Dwork, C. (2006). Differential privacy. Proceedings of the 33rd Interna- 2004.06.018 tional Colloquium on Automata, Languages and Programming, Chambers, C. D. (2013). Registered reports: A new publishing initiative at pp. 1–12. https://doi.org/10.1007/11787006_1 cortex. Journal Devoted to the Study of the Nervous System and Behav- Eklund, A., Dufort, P., Villani, M., & Laconte, S. (2014). BROCCOLI: Soft- ior, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016 ware for fast fMRI analysis on many-core CPUs and GPUs. Frontiers in Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. J. Neuroinformatics, 8(24), 1–19. https://doi.org/10.3389/fninf.2014. (2014). Instead of “playing the game” it is time to change the rules: 00024 Registered reports at AIMS neuroscience and beyond. AIMS Neurosci- Elliott, L. T., Sharp, K., Alfaro-Almagro, F., Shi, S., Miller, K. L., Douaud, G., ence, 1(1), 4–17. https://doi.org/10.3934/Neuroscience.2014.1.4 … Smith, S. M. (2018). Genome-wide association studies of brain imag- Chassang, G. (2017). The impact of the EU general data protection regula- ing phenotypes in UKbiobank. Nature, 562(7726), 210–216. https:// tion on scientific research. Ecancermedicalscience, 11(709). https://doi. doi.org/10.1038/s41586-018-0571-7 org/10.3332/ecancer.2017.709 Esteban, O., Birman, D., Schaer, M., Koyejo, O. O., Poldrack, R. A., & Chen, J., Liu, J., Calhoun, V. D., Arias-Vasquez, A., Zwiers, M. P., Gorgolewski, K. J. (2017). MRIQC: Advancing the automatic prediction Gupta, C. N., … Turner, J. A. (2014). Exploration of scanning effects in of image quality in MRI from unseen sites. PLoS One, 12(9), e0184661. multi-site structural MRI studies. Journal of Neuroscience Methods, 230, https://doi.org/10.1371/journal.pone.0184661 37–50. https://doi.org/10.1016/j.jneumeth.2014.04.023 Esteban, O., Blair, R. W., Nielson, D. M., Varada, J. C., Marrett, S., Chye, Y., Mackey, S., Gutman, B. A., Ching, C. R. K., Batalla, A., Blaine, S., … Thomas, A. G., … Gorgolewski, K. J. (2019). Crowdsourced MRI quality Garavan, H. (in press). Subcortical surface morphometry in substance metrics and expert quality annotations for training of humans and dependence: An ENIGMA addiction working group study. Addiction machines. Scientific Data, 6(1), 30. https://doi.org/10.1038/s41597- Biology. https://doi.org/10.1111/adb.12830 019-0035-4 274 ZUGMAN ET AL. Fischl, B., Liu, A., & Dale, A. M. (2001). Automated manifold surgery: Con- asymmetries in 15,847 people worldwide reveal effects of age and structing geometrically accurate and topologically correct models of sex. Brain Imaging and Behavior, 11(5), 1497–1514. https://doi.org/10. the human cerebral cortex. IEEE Transactions on Medical Imaging, 20(1), 1007/s11682-016-9629-z 70–80. https://doi.org/10.1109/42.906426 Guillaume, B., Hua, X., Thompson, P. M., Waldorp, L., & Nichols, T. E. Fischl, B., Salat, D. H., Busa, E., Albert, M. S., Dieterich, M., Haselgrove, C., (2014). Fast and accurate Modelling of longitudinal and repeated mea- … Dale, A. M. (2002). Whole brain segmentation: Automated labeling sures neuroimaging data. NeuroImage, 94, 287–302. https://doi.org/ of neuroanatomical structures in the human brain. Neuron, 33(3), 10.1016/j.neuroimage.2014.03.029 341–355. Han, X., Jovicich, J., Salat, D., van der Kouwe, A., Quinn, B., Czanner, S., … Fischl, B., Sereno, M. I., & Dale, A. M. (1999). Cortical surface-based analy- Fischl, B. (2006). Reliability of MRI-derived measurements of human sis. II: Inflation, flattening, and a surface-based coordinate system. cerebral cortical thickness: The effects of field strength, scanner NeuroImage, 9(2), 195–207. https://doi.org/10.1006/nimg.1998.0396 upgrade and manufacturer. NeuroImage, 32(1), 180–194. https://doi. Focke, N. K., Helms, G., Kaspar, S., Diederich, C., Tóth, V., Dechent, P., … org/10.1016/j.neuroimage.2006.02.051 Paulus, W. (2011). Multi-site voxel-based morphometry—Not quite Harrewijn, A., Cardinale, E. M., Groenewold, N. A., Bas-Hoogendam, J. M., there yet. NeuroImage, 56(3), 1164–1170. https://doi.org/10.1016/j. Aghajani, M., Cardoner, N., … Pine, D. S. (in prep.). Comparing cortical neuroimage.2011.02.029 and subcortical brain structure between patients with generalized anx- Fortin, J.-P., Cullen, N., Sheline, Y. I., Taylor, W. D., Aselcioglu, I., iety disorder and healthy comparisons—findings from the ENIGMA Cook, P. A., … Shinohara, R. T. (2018). Harmonization of cortical thick- Generalized Anxiety Disorder Working Group. ness measurements across scanners and sites. NeuroImage, 167, Hatton, S. N., Huynh, K. H., Bonilha, L., Abela, E., Alhusaini, S., Altmann, A., 104–120. https://doi.org/10.1016/j.neuroimage.2017.11.024 … McDonald, C. R. (in press). White matter abnormalities across differ- Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M. A., Ruparel, K., … ent epilepsy syndromes in adults: An ENIGMA epilepsy study. BioRxiv. Shinohara, R. T. (2017). Harmonization of multi-site diffusion tensor https://doi.org/10.1101/2019.12.19.883405 imaging data. NeuroImage, 161, 149–170. https://doi.org/10.1016/j. Hibar, D. P., Adams, H. H. H., Jahanshad, N., Chauhan, G., Stein, J. L., neuroimage.2017.08.047 Hofer, E., … Ikram, M. A. (2017). Novel genetic loci associated with Fox, P. T., Lancaster, J. L., Laird, A. R., & Eickhoff, S. B. (2014). Meta- hippocampal volume. Nature Communications, 8, 13624. https://doi. analysis in human neuroimaging: Computational modeling of large- org/10.1038/ncomms13624 scale databases. Annual Review of Neuroscience, 37(1), 409–434. Hibar, D. P., Stein, J. L., Renteria, M. E., Arias-Vasquez, A., Desrivières, S., https://doi.org/10.1146/annurev-neuro-062012-170320 Jahanshad, N., … Medland, S. E. (2015). Common genetic variants Ganzetti, M., Wenderoth, N., & Mantini, D. (2016). Intensity inhomogene- influence human subcortical brain structures. Nature, 520(7546), ity correction of structural MR images: A data-driven approach to 224–229. https://doi.org/10.1038/nature14101 define input algorithm parameters. Frontiers in Neuroinformatics, 10, Hibar, D. P., Westlye, L. T., van Erp, T. G. M., Rasmussen, J., 1–14. https://doi.org/10.3389/fninf.2016.00010 Leonardo, C. D., Faskowitz, J., … Andreassen, O. A. (2016). Subcortical Gelman, A. (2005). Analysis of variance—Why it is more important than volumetric abnormalities in bipolar disorder. Molecular Psychiatry, 21 ever. The Annals of Statistics, 33(1), 1–53. https://doi.org/10.1214/ (12), 1710–1716. https://doi.org/10.1038/mp.2015.227 009053604000001048 Ho, T. C., Gutman, B., Pozzi, E., Grabe, H. J., Hosten, N., Wittfeld, K., … Genovese, C. R., Lazar, N. A., & Nichols, T. (2002). Thresholding of statisti- Schmaal, L. (2020). Subcortical shape alterations in major depressive cal maps in functional neuroimaging using the false discovery rate. disorder: Findings from the ENIGMA major depressive disorder work- NeuroImage, 15(4), 870–878. https://doi.org/10.1006/nimg.2001. ing group. Human Brain Mapping, 1–11. https://doi.org/10.1002/hbm. 1037 24988 Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Holmes, A. P., Blair, R. C., Watson, J. D., & Ford, I. (1996). Nonparametric Yacoub, E., … Van Essen, D. C. (2016). A multi-modal parcellation of analysis of statistic images from functional mapping experiments. Jour- human cerebral cortex. Nature, 536(7615), 171–178. https://doi.org/ nal of Cerebral Blood Flow and Metabolism: Official Journal of the Inter- 10.1038/nature18933 national Society of Cerebral Blood Flow and Metabolism, 16(1), 7–22. Gordon, R. A. (1968). Issues in Multiple Regression. American Journal of https://doi.org/10.1097/00004647-199601000-00002 Sociology, 73(5), 592–616. https://doi.org/10.1086/224533 Hyatt, C. S., Owens, M. M., Crowe, M. L., Carter, N. T., Lynam, D. R., & Gorgolewski, K. J., Alfaro-Almagro, F., Auer, T., Bellec, P., & Capot, M. Miller, J. D. (2020). The quandary of covarying: A brief review and (2017). BIDS apps: Improving ease of use, accessibility, and reproduc- empirical examination of covariate use in structural neuroimaging ibility of neuroimaging data analysis methods. PLoS Computational Biol- studies on psychological variables. NeuroImage, 205, 116225. https:// ogy, 13(3), e1005209. doi.org/10.1016/j.neuroimage.2019.116225 Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., ICMJE. (2019). International Committee of Medical Journal Editors— Duff, E. P., … Poldrack, R. A. (2016). The brain imaging data structure, Recommendations for the Conduct, Reporting, Editing, and Publication a format for organizing and describing outputs of neuroimaging experi- of Scholarly Work in Medical Journals. ments. Scientific Data, 3(1), 1–9. https://doi.org/10.1038/sdata. Jahanshad, N., Kochunov, P. V., Sprooten, E., Mandl, R. C., Nichols, T. E., 2016.44 Almasy, L., … Glahn, D. C. (2013). Multi-site genetic analysis of diffu- Grasby, K. L., Jahanshad, N., Painter, J. N., Colodro-Conde, L., Bralten, J., sion images and voxelwise heritability analysis: A pilot project of the Hibar, D. P., … Medland, S. E. (2020). The genetic architecture of the ENIGMA-DTI working group. NeuroImage, 81, 455–469. https://doi. human cerebral cortex. Science, 367(6484), eaay6690. https://doi.org/ org/10.1016/j.neuroimage.2013.04.061 10.1126/science.aay6690 Ji, J. L., Spronk, M., Kulkarni, K., Repovš, G., Anticevic, A., & Cole, M. W. Gronenschild, E. H. B. M., Habets, P., Jacobs, H. I. L., Mengelers, R., (2019). Mapping the human brain's cortical-subcortical functional net- Rozendaal, N., van Os, J., & Marcelis, M. (2012). The effects of work organization. NeuroImage, 185, 35–57. https://doi.org/10.1016/ FreeSurfer version, workstation type, and Macintosh operating system j.neuroimage.2018.10.006 version on anatomical volume and cortical thickness measurements. Johnson, W. E., Li, C., & Rabinovic, A. (2007). Adjusting batch effects in PLoS One, 7(6), e38234. https://doi.org/10.1371/journal.pone. microarray expression data using empirical Bayes methods. Biostatis- 0038234 tics, 8(1), 118–127. https://doi.org/10.1093/biostatistics/kxj037 Guadalupe, T., Mathias, S. R., vanErp, T. G. M., Whelan, C. D., Jovicich, J., Czanner, S., Greve, D., Haley, E., Van Der Kouwe, A., Zwiers, M. P., Abe, Y., … Francks, C. (2017). Human subcortical brain Gollub, R., … Dale, A. (2006). Reliability in multi-site structural MRI ZUGMAN ET AL. 275 studies: Effects of gradient non-linearity correction on phantom and enlargement in the human visual cortex. Cerebral Cortex, 30(3), human data. NeuroImage, 30(2), 436–443. https://doi.org/10.1016/j. 851–857. https://doi.org/10.1093/cercor/bhz131 neuroimage.2005.09.046 McCarthy, C. S., Ramprashad, A., Thompson, C., Botti, J.-A., Coman, I. L., & Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Kates, W. R. (2015). A comparison of FreeSurfer-generated data with Personality and Social Psychology Review, 2(3), 196–217. https://doi. and without manual intervention. Frontiers in Neuroscience, 9(379), org/10.1207/s15327957pspr0203_4 1–18. https://doi.org/10.3389/fnins.2015.00379 Klapwijk, E. T., van de Kamp, F., van der Meulen, M., Peters, S., & Messé, A. (2019). Parcellation influence on the connectivity-based Wierenga, L. M. (2019). Qoala-T: A supervised-learning tool for quality structure–function relationship in the human brain. Human Brain Map- control of FreeSurfer segmented MRI data. NeuroImage, 189, ping, 41(5), 1167–1180. https://doi.org/10.1002/hbm.24866 116–129. https://doi.org/10.1016/j.neuroimage.2019.01.014 Milchenko, M., & Marcus, D. (2013). Obscuring surface anatomy in volu- Kochunov, P., Jahanshad, N., Marcus, D., Winkler, A., Sprooten, E., metric imaging data. Neuroinformatics, 11(1), 65–75. https://doi.org/ Nichols, T. E., … van Essen, D. C. (2015). Heritability of fractional 10.1007/s12021-012-9160-3 anisotropy in human White matter: A comparison of human Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Connectome project and ENIGMA-DTI data. NeuroImage, 111, Xu, J., … Smith, S. M. (2016). Multimodal population brain imaging in 300–311. https://doi.org/10.1016/j.neuroimage.2015.02.050 the UKbiobank prospective epidemiological study. Nature Neurosci- Kochunov, P., Jahanshad, N., Sprooten, E., Nichols, T. E., Mandl, R. C., ence, 19(11), 1523–1536. https://doi.org/10.1038/nn.4393 Almasy, L., … Glahn, D. C. (2014). Multi-site study of additive genetic Moreau, D. (2019). Preregistration in the context of expertise research: effects on fractional anisotropy of cerebral white matter: Comparing Benefits, challenges, and recommendations. Journal of Expertise, 2(4), meta and mega analytical approaches for data pooling. NeuroImage, 210–216. https://doi.org/10.31234/osf.io/v7xrb 95, 136–150. https://doi.org/10.1016/j.neuroimage.2014.03.033 Mortamet, B., Bernstein, M. A., Jack, C. R., Gunter, J. L., Ward, C., Kong, X.-Z., Mathias, S. R., Guadalupe, T., ENIGMA Laterality Working Britson, P. J., … Krueger, G. (2009). Automatic quality assessment in Group, Glahn, D. C., Franke, B., … Francks, C. (2018). Mapping cortical structural brain magnetic resonance imaging. Magnetic Resonance in brain asymmetry in 17,141 healthy individuals worldwide via the Medicine, 62(2), 365–372. https://doi.org/10.1002/mrm.21992 ENIGMA consortium. Proceedings of the National Academy of Sciences, Müller, V. I., Cieslik, E. C., Laird, A. R., Fox, P. T., Radua, J., Mataix-Cols, D., 115(22), E5154–E5163. https://doi.org/10.1073/pnas.1718418115 … Eickhoff, S. B. (2018). Ten simple rules for neuroimaging meta-analy- Koshiyama, D., Fukunaga, M., Okada, N., Morita, K., Nemoto, K., Usui, K., sis. Neuroscience & Biobehavioral Reviews, 84, 151–161. https://doi. … Hashimoto, R. (2020). White matter microstructural alterations org/10.1016/j.neubiorev.2017.11.012 across four major psychiatric disorders: Mega-analysis study in 2937 Murphy, K., & Fox, M. D. (2017). Towards a consensus regarding global individuals. Molecular Psychiatry, 25(4), 883–895. https://doi.org/10. signal regression for resting state functional connectivity MRI. 1038/s41380-019-0553-7 NeuroImage, 154, 169–173. https://doi.org/10.1016/j.neuroimage. Lakatos, I. (1976). Proofs and refutations. Cambridge: Cambridge University 2016.11.052 Press. Neuroskeptic. (2012). The nine circles of scientific hell. Perspectives on Psy- Ledgerwood, A. (2018). The preregistration revolution needs to distinguish chological Science, 7(6), 643–644. between predictions and analyses. Proceedings of the National Academy Nichols, T., & Hayasaka, S. (2003). Controlling the familywise error rate in of Sciences, 115(45), E10516–E10517. https://doi.org/10.1073/pnas. functional neuroimaging: A comparative review. Statistical Methods in 1812592115 Medical Research, 12(5), 419–446. Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Nichols, T. E., & Holmes, A. P. (2002). Nonparametric permutation tests Johnson, W. E., … Irizarry, R. A. (2010). Tackling the widespread and for functional neuroimaging: A primer with examples. Human Brain critical impact of batch effects in high-throughput data. Nature Reviews Mapping, 15(1), 1–25. Genetics, 11(10), 733–739. https://doi.org/10.1038/nrg2825 Nielson, D. M., Pereira, F., Zheng, C. Y., Migineishvili, N., Lee, J. A., Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Thomas, A. G., & Bandettini, P. A. (2018). Detecting and harmonizing Psychological Bulletin, 68(5), 304–305. https://doi.org/10.1037/ scanner differences in the ABCD study—Annual release 1.0. https:// h0025105 doi.org/10.1101/309260 Luque-Fernandez, M. A., Schomaker, M., Redondo-Sanchez, D., Jose Nordenskjöld, R., Malmberg, F., Larsson, E.-M., Simmons, A., Brooks, S. J., Sanchez Perez, M., Vaidya, A., & Schnitzer, M. E. (2019). Educational Lind, L., … Kullberg, J. (2013). Intracranial volume estimated with com- note: Paradoxical collider effect in the analysis of non-communicable monly used methods could introduce bias in studies including brain disease epidemiological data: A reproducible illustration and web volume measurements. NeuroImage, 83, 355–360. https://doi.org/10. application. International Journal of Epidemiology, 48(2), 640–653. 1016/j.neuroimage.2013.06.068 https://doi.org/10.1093/ije/dyy275 Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., preregistration revolution. Proceedings of the National Academy of Sci- Ioannidis, J. P. A., … Glasziou, P. (2014). Biomedical research: Increas- ences, 115(11), 2600–2606. https://doi.org/10.1073/pnas. ing value, reducing waste. The Lancet, 383(9912), 101–104. https:// 1708274114 doi.org/10.1016/S0140-6736(13)62329-6 Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: Maggioni, E., Tana, M. G., Arrigoni, F., Zucca, C., & Bianchi, A. M. (2014). II. Restructuring incentives and practices to promote truth over Constructing fMRI connectivity networks: A whole brain functional Publishability. Perspectives on Psychological Science, 7(6), 615–631. parcellation method for node definition. Journal of Neuroscience Nunes, A., Schnack, H. G., Ching, C. R. K., Agartz, I., Akudjedu, T. N., Methods, 228, 86–99. https://doi.org/10.1016/j.jneumeth.2014. Alda, M., … Hajek, T. (2018). Using structural MRI to identify bipolar 03.004 disorders – 13 site machine learning study in 3020 individuals from Magnotta, V. A., & Friedman, L. (2006). Measurement of signal-to-noise the ENIGMA bipolar disorders working group. Molecular Psychiatry. and contrast-to-noise in the fBIRN multicenter imaging study. Journal https://doi.org/10.1038/s41380-018-0228-9 of Digital Imaging, 19(2), 140–147. https://doi.org/10.1007/s10278- Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed. Cam- 006-0264-x bridge, UK: Cambridge University Press. Månsson, K. N. T., Cortes, D. S., Manzouri, A., Li, T.-Q., Hau, S., & Power, J. D., Cohen, A. L., Nelson, S. M., Wig, G. S., Barnes, K. A., Fischer, H. (2020). Viewing pictures triggers rapid morphological Church, J. A., … Petersen, S. E. (2011). Functional network organization 276 ZUGMAN ET AL. of the human brain. Neuron, 72(4), 665–678. https://doi.org/10.1016/ Staunton, C., Slokenberga, S., & Mascalzoni, D. (2019). The GDPR and the j.neuron.2011.09.006 research exemption: Considerations on the necessary safeguards for Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010). Meta-analysis of individ- research biobanks. European Journal of Human Genetics, 27(8), ual participant data: Rationale, conduct, and reporting. BMJ, 340, 1159–1167. https://doi.org/10.1038/s41431-019-0386-5 c221–c221. https://doi.org/10.1136/bmj.c221 Stein, J. L., Medland, S. E., Vasquez, A. A., Hibar, D. P., Senstad, R. E., Rorden, C. (2014). Dcm2niix. Retrieved from https://github.com/ Winkler, A. M., … Thompson, P. M. (2012). Identification of common rordenlab/dcm2niix variants associated with human hippocampal and intracranial volumes. Roseman, M., Milette, K., Bero, L. A., Coyne, J. C., Lexchin, J., Nature Genetics, 44(5), 552–561. https://doi.org/10.1038/ng.2250 Turner, E. H., & Thombs, B. D. (2011). Reporting of conflicts of interest Sterling, T. D. (1959). Publication decisions and their possible effects on in meta-analyses of trials of pharmacological treatments. JAMA, 305 inferences drawn from tests of significance—Or vice versa. Journal of (10), 1008–1017. the American Statistical Association, 54(285), 30–34. https://doi.org/ Rosen, A. F. G., Roalf, D. R., Ruparel, K., Blake, J., Seelaus, K., Villa, L. P., … 10.2307/2282137 Satterthwaite, T. D. (2018). Quantitative assessment of structural Tahmasian, M., Sepehry, A. A., Samea, F., Khodadadifar, T., image quality. NeuroImage, 169, 407–418. https://doi.org/10.1016/j. Soltaninejad, Z., Javaheripour, N., … Eickhoff, C. R. (2019). Practical neuroimage.2017.12.059 recommendations to conduct a neuroimaging meta-analysis for neuro- Rosenthal, R. (1979). The file drawer problem and tolerance for null psychiatric disorders. Human Brain Mapping, 40(17), 5142–5154. results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10. https://doi.org/10.1002/hbm.24746 1037//0033-2909.86.3.638 Thompson, P. M., Jahanshad, N., Ching, C. R. K., Salminen, L., Sanfilipo, M. P., Benedict, R. H. B., Zivadinov, R., & Bakshi, R. (2004). Cor- Thomopoulos, S. I., Bright, J., … Zelman, V. (2020). ENIGMA and global rection for intracranial volume in analysis of whole brain atrophy in neuroscience: A decade of large-scale studies of the brain in health multiple sclerosis: The proportion vs. residual method. NeuroImage, 22 and disease across more than 40 countries. Translational Psychiatry. (4), 1732–1743. https://doi.org/10.1016/j.neuroimage.2004.03.037 https://doi.org/10.1038/s41398-020-0705-1 Satizabal, C. L., Adams, H. H. H., Hibar, D. P., White, C. C., Knol, M. J., Thompson, P. M., Stein, J. L., Medland, S. E., Hibar, D. P., Vasquez, A. A., Stein, J. L., … Ikram, M. A. (2019). Genetic architecture of subcortical Renteria, M. E., … Drevets, W. (2014). The ENIGMA consortium: brain structures in 38,851 individuals. Nature Genetics, 51(11), Large-scale collaborative analyses of neuroimaging and genetic data. 1624–1636. https://doi.org/10.1038/s41588-019-0511-y Brain Imaging and Behavior, 8(2), 153–182. https://doi.org/10.1007/ Schaefer, A., Kong, R., Gordon, E. M., Laumann, T. O., Zuo, X.-N., s11682-013-9269-5 Holmes, A. J., … Yeo, B. T. T. (2018). Local-global Parcellation of the Tzanou, M. (2013). Data protection as a fundamental right next to privacy? human cerebral cortex from intrinsic functional connectivity MRI. “Reconstructing” a not so new right. International Data Privacy Law, 3 Cerebral Cortex, 28(9), 3095–3114. https://doi.org/10.1093/cercor/ (2), 88–99. https://doi.org/10.1093/idpl/ipt004 bhx179 Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Schimke, N., Kuehler, M., & Hale, J. (2011). Preserving privacy in structural Etard, O., Delcroix, N., … Joliot, M. (2002). Automated anatomical Neuroimages. In Y. Li (Ed.), Data and Applications Security and Privacy labeling of activations in SPM using a macroscopic anatomical XXV. DBSec 2011, Lecture Notes in Computer Science, (Vol. 6818, parcellation of the MNI MRI single-subject brain. NeuroImage, 15(1), pp. 301–308). Berlin, Heidelberg: Springer. https://doi.org/10.1007/ 273–289. https://doi.org/10.1006/nimg.2001.0978 978-3-642-22348-8_26 van Erp, T. G. M., Hibar, D. P., Rasmussen, J. M., Glahn, D. C., Schmaal, L., Veltman, D. J., van Erp, T. G. M., Sämann, P. G., Frodl, T., Pearlson, G. D., Andreassen, O. A., … Turner, J. A. (2016). Subcortical Jahanshad, N., … Hibar, D. P. (2016). Subcortical brain alterations in brain volume abnormalities in 2028 individuals with schizophrenia and major depressive disorder: Findings from the ENIGMA major depres- 2540 healthy controls via the ENIGMA consortium. Molecular Psychia- sive disorder working group. Molecular Psychiatry, 21(6), 806–812. try, 21(4), 547–553. https://doi.org/10.1038/mp.2015.63 https://doi.org/10.1038/mp.2015.69 Villalón-Reina, J. E., Martínez, K., Qu, X., Ching, C. R. K., Nir, T. M., Schwarz, C. G., Kremers, W. K., Therneau, T. M., Sharp, R. R., Gunter, J. L., Kothapalli, D., … Bearden, C. E. (in press). Altered white matter micro- Vemuri, P., … Jack, C. R. (2019). Identification of anonymous MRI structure in 22q11.2 deletion syndrome: A multisite diffusion tensor research participants with face-recognition software. The New England imaging study. Molecular Psychiatry. https://doi.org/10.1038/s41380- Journal of Medicine, 381(17), 1684–1686. 019-0450-0 Simmonds, M., Stewart, G., & Stewart, L. (2015). A decade of individual Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & participant data meta-analyses: A review of current practice. Contem- Kievit, R. A. (2012). An agenda for purely confirmatory research. Per- porary Clinical Trials, 45, 76–83. https://doi.org/10.1016/j.cct.2015. spectives on Psychological Science, 7(6), 632–638. 06.012 Wang, X., Xie, H., Chen, T., O'Leary, B., Cotton, A., White, D., … Liberzon, I. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psy- (2019). 109. Mega-analysis of cortical morphometric differences chology: Undisclosed flexibility in data collection and analysis allows between PTSD patients and non-PTSD controls. Biological Psychiatry, presenting anything as significant. Psychological Science, 22, 85(10), S45–S46. https://doi.org/10.1016/j.biopsych.2019.03.123 1359–1366. https://doi.org/10.1177/0956797611417632 Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding Smith, S. M., Jenkinson, M., Johansen-Berg, H., Rueckert, D., Nichols, T. E., constructs is harder than you think. PLoS One, 11(3), e0152719. Mackay, C. E., … Behrens, T. E. J. (2006). Tract-based spatial statistics: https://doi.org/10.1371/journal.pone.0152719 Voxelwise analysis of multi-subject diffusion data. NeuroImage, 31(4), Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing. 1487–1505. https://doi.org/10.1016/j.neuroimage.2006.02.024 New York, NY: John Wiley & Sons. Smith, S. M., & Nichols, T. E. (2009). Threshold-free cluster enhancement: Winkler, A. M., Ridgway, G. R., Douaud, G., Nichols, T. E., & Smith, S. M. Addressing problems of smoothing, threshold dependence and (2016). Faster permutation inference in brain imaging. NeuroImage, localisation in cluster inference. NeuroImage, 44(1), 83–98. https://doi. 141, 502–516. https://doi.org/10.1016/j.neuroimage.2016.05.068 org/10.1016/j.neuroimage.2008.03.061 Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Smith, S. M., & Nichols, T. E. (2018). Statistical challenges in “big data” Nichols, T. E. (2014). Permutation inference for the general linear human neuroimaging. Neuron, 97(2), 263–268. https://doi.org/10. model. NeuroImage, 92, 381–397. https://doi.org/10.1016/j. 1016/j.neuron.2017.12.018 neuroimage.2014.01.060 ZUGMAN ET AL. 277 Winkler, A. M., Sabuncu, M. R., Yeo, B. T. T., Fischl, B., Greve, D. N., Zarrar, S., Steven, G., Qingyang, L., Yassine, B., Chaogan, Y., Zhen, Y., … Kochunov, P., … Glahn, D. C. (2012). Measuring and comparing brain Cameron, C. (2015). The preprocessed Connectomes project quality cortical surface area and other areal quantities. NeuroImage, 61(4), assessment protocol—A resource for measuring the quality of MRI 1428–1443. https://doi.org/10.1016/j.neuroimage.2012.03.026 data. INCF Neuroinformatics, 9. https://doi.org/10.3389/conf.fnins. Winkler, A. M., Webster, M. A., Brooks, J. C., Tracey, I., Smith, S. M., & 2015.91.00047 Nichols, T. E. (2016). Non-parametric combination and related permu- Zatorre, R. J., Fields, R. D., & Johansen-Berg, H. (2012). Plasticity in gray tation tests for neuroimaging. Human Brain Mapping, 37(4), and white: Neuroimaging changes in brain structure during learning. 1486–1511. https://doi.org/10.1002/hbm.23115 Nature Neuroscience, 15(4), 528–536. https://doi.org/10.1038/nn. Worsley, K. J., Liao, C. H., Aston, J., Petre, V., Duncan, G. H., Morales, F., & 3045 Evans, A. C. (2002). A general statistical analysis for fMRI data. Zavaliangos-Petropulu, A., Nir, T. M., Thomopoulos, S. I., Reid, R. I., NeuroImage, 15(1), 1–15. https://doi.org/10.1006/nimg.2001.0933 Bernstein, M. A., Borowski, B., … Thompson, P. M. (2019). Diffusion Worsley, K. J., Marrett, S., Neelin, P., Vandal, A. C., Friston, K. J., & MRI indices and their relation to cognitive impairment in brain aging: Evans, A. C. (1996). A unified statistical approach for determining sig- The updated multi-protocol approach in ADNI3. Frontiers in Neu- nificant signals in images of cerebral activation. Human Brain Mapping, roinformatics, 13(2), 1–18. https://doi.org/10.3389/fninf.2019. 4(1), 58–73. https://doi.org/10.1002/(SICI)1097-0193(1996)4:1<58:: 00002 AID-HBM4>3.0.CO;2-O Yeo, B. T. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., … Buckner, R. L. (2011). The organization of the human cerebral cortex estimated by intrinsic functional connectivity. How to cite this article: Zugman A, Harrewijn A, Journal of Neurophysiology, 106(3), 1125–1165. https://doi.org/10. Cardinale EM, et al. Mega-analysis methods in ENIGMA: The 1152/jn.00338.2011 experience of the generalized anxiety disorder working group. Yu, M., Linn, K. A., Cook, P. A., Phillips, M. L., McInnis, M., Fava, M., … Hum Brain Mapp. 2022;43:255–277. https://doi.org/10.1002/ Sheline, Y. I. (2018). Statistical harmonization corrects site effects in func- tional connectivity measurements from multi-site fMRI data. Human Brain hbm.25096 Mapping, 39(11), 4213–4227. https://doi.org/10.1002/hbm.24241