Received: 21 April 2020 Revised: 13 July 2020 Accepted: 19 July 2020 DOI: 10.1002/hbm.25154 RESEARCH ARTICLE Reproducibility in the absence of selective reporting: An illustration from large-scale brain asymmetry research Xiang-Zhen Kong1,2 | ENIGMA Laterality Working Group | Clyde Francks1,3 1 Language and Genetics Department, Max Planck Institute for Psycholinguistics, Abstract Nijmegen, The Netherlands The problem of poor reproducibility of scientific findings has received much attention 2 Department of Psychology and Behavioral over recent years, in a variety of fields including psychology and neuroscience. The Sciences, Zhejiang University, Hangzhou, China problem has been partly attributed to publication bias and unwanted practices such 3 Donders Institute for Brain, Cognition and as p-hacking. Low statistical power in individual studies is also understood to be an Behaviour, Radboud University, Nijmegen, The Netherlands important factor. In a recent multisite collaborative study, we mapped brain anatomical left–right asymmetries for regional measures of surface area and cortical thickness, in Correspondence Xiang-Zhen Kong, Language and Genetics 99 MRI datasets from around the world, for a total of over 17,000 participants. In the Department, Max Planck Institute for present study, we revisited these hemispheric effects from the perspective of repro- Psycholinguistics, Nijmegen, The Netherlands. Email: xiangzhen.kong@outlook.com ducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect Funding information This research was funded by the Max Planck direction and significance threshold. In this sense, the results within each dataset were Society (Germany). Funding information for viewed as coming from separate studies in an “ideal publishing environment,” that is, each participating site is available in the Supporting Information Appendix. free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes. KEYWORDS multisite collaboration, P-hacking, publication bias, reproducibility, team science 1 | I N T RO DU CT I O N neuroscience (Button et al., 2013; Wager, Lindquist, Nichols, Kober, & Van Snellenberg, 2009). Poor reproducibility has been partly attrib- The issue of reproducibility has received considerable attention in a uted to reporting bias and problematic practices such as selective variety of fields including medicine (Prinz, Schlange, & Asadullah, reporting of outcomes (i.e., p-hacking) (Aarts et al., 2015; Baker, 2016; 2011), psychology (Aarts et al., 2015; R. A. Klein et al., 2014) and Bakker, van Dijk, & Wicherts, 2012; Ioannidis, 2005, 2008; Ioannidis, Munafo, Fusar-Poli, Nosek, & David, 2014; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011). This situation A complete list of the ENIGMA Laterality Working Group can be found in the Supporting Information Appendix. has resulted in multiple calls for more reproducible research This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2020 The Authors. Human Brain Mapping published by Wiley Periodicals LLC. 244 wileyonlinelibrary.com/journal/hbm Hum Brain Mapp. 2022;43:244–254. 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KONG ET AL. 245 (e.g., Benjamin et al., 2018; Button et al., 2013; O. Klein, Hardwicke, the 99 datasets as being from an “ideal reporting environment”, free et al., 2018; Poldrack et al., 2017; Valentin Amrhein, 2017). For from reporting bias, or other potentially problematic practices such as example, the Open Science Framework has been set up as a free and p-hacking. open source project management resource for researchers across the If we assume the meta-analytic effect sizes reported by Kong entire study cycle. In addition, the Transparency and Openness et al. (2018) to represent the “true” hemispheric asymmetry effect Promotion (TOP) Guidelines (Nosek et al., 2015) have been proposed sizes, in this way we have access to a real-world setting for examining to improve the quality and credibility of scientific literature. In neuro- reproducibility across 99 datasets in the absence of selective imaging studies, problems such as flexibility in data analysis have been reporting, which can provide a useful illustration of how consistently widely discussed, and best practices have been proposed to ensure realistic biological effects in MRI data can be detected when surveying that neuroimaging studies can produce meaningful and reliable results cohorts worldwide. If we wanted to address this question with actual (Poldrack et al., 2017). The reproducibility rate was not found to cor- papers in the literature, then an ideal publishing environment would relate with levels of experience and expertise of study authors, in a first need to be established, free from selective reporting. This seems replication study of previous findings in psychology (Aarts impossible in the current era, because many journals and scientists are et al., 2015), which suggests that some practices will not improve incentivized to report statistically significant results, while leaving merely through training, and that other factors influence reproducibil- nonsignificant findings unpublished (known as the file-drawer effect). ity within the current research convention. Specifically, Kong et al. (2018) analyzed thickness and area mea- Among these factors, low statistical power is now well under- sures for each of 34 brain regions based on the Desikan-Killiany atlas stood to contribute to the reproducibility problem (Button from FreeSurfer, as well as entire hemisphere-level average thickness et al., 2013; Ioannidis, 2005), although it was only ranked number and total area (Fischl, 2012), for a total of 70 left–right hemispheric three behind “Selective reporting” and “Pressure to publish” in a recent effects. For the present study of reproducibility, we considered each Nature survey (Baker, 2016). The positive predictive value (PPV), that of these 70 asymmetry effects as a single research question, for is, the probability that a “positive” research finding reflects a true example, does the parahippocampal gyrus show left–right asymmetri- effect, has been formulated as a function of the prior probability of cal thickness, on average, in the human brain? We further considered the effect being real (R, the prestudy odds), the statistical power of that each of these questions had been asked 99 times in separate the study (1 − β; β is the Type II error), and the level of statistical sig- datasets, with a range of different sample sizes, scanning equipment nificance required (α; α is the Type I error, for example, 0.05 or 0.01): and parameters (although image processing with Freesurfer was har- PPV = (1 − β)R/([1 − β]R + α)) (Button et al., 2013; Ioannidis, 2005). monized across datasets). Our goal was to use these data to assess For example, it is evident that a research finding is more likely true reproducibility of hemispheric asymmetry effects not only in an “ideal than false (i.e., PPV > 50%) if (1 − β)R > α. However, in many cases publishing environment”, but also in the context of extensive dataset the true effect size is unknown a priori, and/or the prestudy odds are heterogeneity, which is a feature of the real-world literature. unknown. This problem is then further complicated by potentially selective reporting or other problematic practices. The present study aimed to illustrate the reproducibility of human 2 | M A T E R I A L S A N D M ET H O D S MRI results in an unusual setting where a priori knowledge of the sta- tistical power and prestudy odds was not necessary, and in the 2.1 | Datasets absence of selective reporting. This was possible because we lever- aged summary statistics from a previous study performed via a world- We used publicly available summary statistics from the previously wide collaborative network known as the Enhancing NeuroImaging published ENIGMA cortical asymmetry project (http://conxz.net/ Genetics through Meta-Analysis (ENIGMA) consortium (Thompson neurohemi/) (Kong et al., 2018). That study used data from 17,141 et al., 2014). In that study, the ENIGMA consortium mapped left–right healthy participants from 99 separate datasets, each of which showed hemispheric asymmetry effects on 70 regional and total cortical gray different age distributions, and were from diverse ethnic backgrounds matter metrics, in over 17,000 individuals from 99 datasets (Kong (Table S1). Participants were drawn from the general population or et al., 2018). Hemispheric asymmetry is a key feature of human brain were healthy controls from clinical case–control studies (affected indi- organization, and altered brain asymmetry has been linked to various viduals were not included). In most cases each participating site contrib- cognitive and neuropsychiatric disorders (Carrion-Castillo et al., 2020; uted one dataset, but there were seven sites that contributed more Kong, Boedhoe, et al., 2020, Kong, Postema, et al., 2019; Kong than one dataset from distinct studies (Table S1), for example when dif- et al., 2020; Postema et al., 2019; Toga & Thompson, 2003; Zhen ferent scanners were used (1.5T and 3T: e.g., OCD_Cheng_1.5T and et al., 2017). For the study by Kong et al. (2018), analysis plans and OCD_Cheng_3T; OCD_VUmc 1.5T and OCD_VUmc 3T), or different scripts were prepared by a central site and then sent out to many sep- age groups were recruited (such as children and older adults: arate sites worldwide to run on their own datasets. Finally, outputs e.g., NSIOCDS_3T_Adults and NSIOCDS_3T_Child; GBB_ GRADUAL from each dataset were sent back to the central site and combined by and GBB_OLDERS). meta-analysis methodology, with no results-based selection applied to In the present study we analyzed the reproducibility of hemi- any of the datasets. Thus, we can consider summary statistics from spheric asymmetry effects on paired left–right measures of cortical 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 246 KONG ET AL. thickness and surface area, for 34 brain regions based on the Desikan- for deriving cortical thickness and surface area for each of the Killiany atlas from FreeSurfer (Fischl, 2012), as well as entire 34 regions in the Desikan-Killiany atlas. hemisphere-level average thickness and total surface area. See Kong et al. (2018) for details about the neuroimaging processing and quality control. Briefly, images were acquired using scanners of different field 2.2 | Estimation of the “true” effects strengths (1.5T and 3T; Table S1) and all images were analyzed using the automated and validated pipeline “recon-all” implemented in Given a lack of consistency of brain asymmetry findings in earlier liter- FreeSurfer (Fischl, 2012), although different software versions were ature, Kong et al. (2018) performed the largest ever study of this possible (version 5.0, 5.1, and 5.3) (Table S1). issue, based on at least an order of magnitude more participants than Table S1 gives summary information for each dataset: Briefly, the any previous study. For each regional or total thickness or surface sample size varied across datasets from 14 to 2,326 (median 72); area measure, a paired t-test (i.e., paired within subject) was used to FreeSurfer version 5.3 was used exclusively in 91 datasets, version 5.1 compare the mean left and right measure, separately within each exclusively in 6 datasets, version 5.0 in one dataset, and a mixture of dataset. These t-tests provided the hemispheric effects, and the out- 5.1 and 5.3 in one dataset; 63 datasets used a 3T scanner and puts from each of the 99 datasets were combined using inverse 29 datasets used a 1.5T scanner; the minimum age across all datasets variance-weighted random-effect meta-analysis, with the R package was 3 years, the maximum age was 90 years. All local institutional metafor, version 1.9-9. This method tests one overall effect, while review boards permitted the use of extracted measures from the weighting each dataset's contribution by the inverse of its anonymized data. corresponding sampling variance. Thus, unlike fixed-effect meta-anal- For each dataset and paired left–right measure, Kong et al. (2018) ysis, this method takes into account variability across difference stud- used paired t-tests to assess inter-hemispheric differences, and ies. In addition, test statistics in the meta-analyses were computed Cohen's d was calculated based on each paired t-test, to estimate based on a standard normal distribution (test = “z” by default). For the hemispheric effect size (i.e., the standardized difference between more details, see (Kong et al., 2018). the mean left and right measures, for a given region and dataset). In A Cohen's d effect size estimate of the population-level asymme- the procedure, analysis plans and scripts were prepared by a central try was obtained for each hemispheric effect, for each paired left– site and sent out to each dataset's own site for running the analysis, right measure, that is, the standardized difference between the mean and finally all outputs for every dataset were sent back to the central left and right measures. The Cohen's d hemispheric effects derived site for meta-analysis. (A laterality index was also used when investi- from the meta-analytic approach over 99 datasets can be taken as gating how factors such as age, sex, and handedness affected brain “true” effects representing left–right differences in the average human asymmetry [Kong et al., 2018], but this is not relevant to the present brain, as measured through this image processing and analysis pipe- study.) line. Sixty-three of the 70 hemispheric effects were significant at In addition to the per-dataset summary statistics and meta- p ≤ .05 (uncorrected for multiple testing across regions) in the meta- analysis results from Kong et al. (2018), two additional datasets with analysis over 99 datasets, while seven of the effects were not signifi- MRI data were used for the present study. These were the Human cantly different from zero (p > .05) (Kong et al., 2018). We also applied Connectome Project (HCP; https://www.humanconnectome.org/) a more stringent significance threshold of 0.05/70 to identify “true” and Brain Imaging Genetics (BIG; http://cognomics.nl/) datasets. The effects surviving multiple testing correction, which resulted in 56 sig- HCP comprises 1,113 individuals (age 22–37) scanned using the nificant effects and 14 nonsignificant effects. HCP's custom 3T Siemens Skyra, with data processing as previously Note that a meta-analytic approach was taken by Kong described (Glasser et al., 2013). For each of the 34 regions in the et al. (2018) because the central analysis team did not have individual- Desikan-Killiany atlas, cortical thickness and surface area were level data from most of the 99 datasets. The different sites ran the derived based on individual T1-weighted MRI images (HCP pipeline same script on their data and returned summary statistics to the cen- based on FreeSurfer version 5.3). The HCP data were only used to tral team for meta-analysis. Individual-level data sharing can require define a population mean surface area for each cortical region, which material transfer agreements and additional IRB approvals, so that a was used to test whether regional size is related to reproducibility of meta-analytic approach was the only way to achieve a timely study of asymmetry effects (see below). this many datasets. Nonetheless, meta-analysis and individual-level- In the BIG dataset, we included T1-weighted MRI scans of analysis were found to result in similar effect sizes in a recent, empiri- 423 participants (age 18–74) who were scanned twice on separate cal comparison using large-scale multisite neuroimaging data occasions. Their second scan was done from zero to 2,650 days after (Boedhoe et al., 2018). their first scan (median = 149; mean = 325, SD = 432). From these data we could calculate the reliability of regional measures, and then test for a relation of reliability to reproducibility (see below). BIG data 2.3 | Estimation of reproducibility were scanned using either a 1.5T Siemens Avanto or Sonata scanner, or a 3.0T Siemens TIM Trio or Skyra scanner (see Table S2 for the For the present study, the reproducibility rate for a given effect was numbers of participants by scanner). FreeSurfer version 5.3 was used calculated as the proportion of datasets in which that effect was 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KONG ET AL. 247 reproduced. Specifically, we applied a leave-one-out strategy for esti- 2.4 | Reproducibility, effect size, and sample size mation of reproducibility: each effect within each dataset was com- pared in turn to the corresponding meta-analytic effect from the The hemispheric effect size varied across different brain structural 98 other datasets, to avoid sample overlap. measures (Kong et al., 2018), from Cohen's d = 0.0015–1.76 (median For an effect that was significant at p ≤ .05 in a given meta- 0.30) (unsigned magnitudes) (Figure 1). In addition, the sample size analysis of 98 datasets (or more stringently at p ≤ .05/70 to correct varied across datasets, from 14 to 2,326 (median 72) (Figure 1a; for multiple comparisons across 70 effects), the effect in the Table S1). These variabilities allowed us to illustrate the expected rela- remaining single dataset would need to be in the same left–right tionships of reproducibility, effect size and sample size. As surface direction as in the meta-analysis, and also be nominally significant area asymmetries are generally more substantial than cortical thick- within the single dataset (p ≤ .05), in order to be counted as ness asymmetries (Figure 1b) (Kong et al., 2018), we first compared reproduced in that dataset. reproducibility rates between the hemispheric effects for these two For an effect that was nonsignificant in the meta-analysis separate types of measure. We then calculated the Spearman correla- (p > .05, or p > .05/70 to correct for multiple comparisons across tion between the “true” effect size and the reproducibility rate across 70 effects), then the effect would also need to be nonsignificant all 70 hemispheric effects, as well as across cortical thickness and sur- (p > .05) in the single dataset, in order to be counted as reproduced in face area hemispheric effects separately. that dataset. Thus reproducibility in our formulation is not simply a We also used a descriptive approach to show the relationship measure of the detection rate of significant effects (i.e., statistical between reproducibility and sample size. Specifically, to explore the power), but also incorporates the consistency of finding nonsignificant contribution of sample size of datasets to reproducibility, we calcu- effects in the individual datasets, when those effects were not signifi- lated the reproducibility rate for each effect using a range of minimum cant in meta-analysis. sample size thresholds of 15, 50, 100, 150, 200, 300, 400, and 500. F I G U R E 1 Reproducibility in the absence of selective reporting, estimated based on outputs of the ENIGMA cortical asymmetry project. (a) Sample size distribution of the 99 datasets. (b) Effect size distribution of the 70 hemispheric effects of interest. (c) Reproducibility distribution of the 70 hemispheric effects. The reproducibility was assessed by comparing each dataset in turn to the meta-analytic effect from the 98 others, to avoid overlap (see Methods). (d) Scatter plot of the correlation between the reproducibility and the effect size. (e) Relations of reproducibility, effect size and dataset sample size. Each line plots the mean and 95% confidence interval for reproducibility. We used the meta-analytic effect size over all 99 datasets for visualization purposes. The figure key shows the types of cortical measure (orange indicates surface area; green indicates cortical thickness), as well as groupings by true effect sizes 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 248 KONG ET AL. That is, in each case, the reproducibility rate was calculated based on between ICCs and reproducibility rates across the 34 regions (sepa- only the datasets with sample sizes above that threshold. This rately for thickness and surface area asymmetry effects). resulted in different subgroups of datasets with varied minimum sam- ple sizes: minimum size 15 as in the main analysis described above (97 datasets in total), and then minimum size 50 (63 datasets), mini- 2.6 | Data and code sharing mum size 100 (37 datasets), minimum size 150 (25 datasets), minimum size 200 (20 datasets), minimum size 300 (19 datasets), minimum size Data used in this study were published summary statistics from the 400 (12 datasets), and minimum size 500 (7 datasets). ENIGMA cortical asymmetry project (Kong et al., 2018). Data and In addition, we used a descriptive approach to group the scripts for all analyses are available in GitHub (https://github.com/ 70 effects based on their meta-analytic effect sizes (based on all Conxz/illusReproducibility). Additional data were from the Human 99 datasets): 0.0 ≤ d < 0.2, 0.2 ≤ d < 0.4, 0.4 ≤ d < 0.6, 0.6 ≤ d < 0.8, Connectome Project (https://www.humanconnectome.org/), and the 0.8 ≤ d < 1.8, and summarized the reproducibility rate for each effect BIG dataset (http://cognomics.nl/). size subgroup separately. The R package pwr was used for calculating statistical power to detect meta-analytic effect sizes in relation to dataset sample sizes. 3 | RE SU LT S 3.1 | Estimating reproducibility 2.5 | Reproducibility and data heterogeneity There was an overall mean reproducibility rate of 63.2%, that is, on The majority of the datasets (91 of 99) were processed using average 63.2% of the single-dataset results were consistent with the FreeSurfer version 5.3. Therefore the meta-analytic effect sizes may meta-analytic true effects (see Methods). A large variability of repro- particularly reflect this software version, to the extent that different ducibility was observed across effects (SD = 22.9%, range from 22.2 versions of FreeSurfer can yield slightly different measures to 97.0%) (Figure 1c). When using a more stringent significance (Gronenschild et al., 2012). We compared the reproducibility rates threshold of 0.05/70 for correcting multiple testing in the meta- of the 70 effects (again using the 1 dataset vs. 98 approach) analytic results, the reproducibility remained similar (mean = 64.4%, between the datasets using version 5.3 (91 datasets) versus those SD = 21.9%), which reflects that the large majority of effects were sig- using version 5.1 (6 datasets), using both the t-test and the nonpara- nificant regardless of this correction (63 significant before correction, metric Mann–Whitney test. (Note that the small number of datasets 56 significant after correction). using version 5.1 limited our ability to test whether FreeSurfer ver- For the whole hemisphere asymmetry effects (i.e., derived from sion affects reproducibility.) In the same way, we also compared the the average cortical thickness over each entire hemisphere, and the reproducibility rates of the 70 effects between the 63 datasets with total surface area of each hemisphere), the reproducibility rates across 3T scanning versus the 29 datasets with 1.5T scanning, and datasets were 36.4 and 66.7%, respectively. For regionally specific between the 18 datasets with maximum age 18 years versus the hemispheric effects, the reproducibility rate across datasets ranged 36 datasets with minimum age 19 years. We also ran meta-analyses from 27.3 to 72.7% (mean = 48.7%, SD = 12.3%), and from 22.2 to for the subgroups separately (i.e., 3T/1.5T datasets, and child/adult 97.0% (mean = 78.4%, SD = 21.7%) for cortical thickness and surface datasets) to estimate subgroup-specific “true” effects for calculating area measures, respectively (Figure 2). These findings show that reproducibility. reproducibility is far from perfect, even without any publication bias In addition, we investigated the potential relationships between or potentially problematic practices such as p-hacking. cortical regional variation in reproducibility and two regional proper- The overall reproducibility remained similar (62.7%, SD = 23.3%) ties: regional size (surface area) and measurement reliability, as a after excluding the two largest individual datasets (i.e., BIG and proxy of image quality and region-specific performance of FreeSurfer. QTIM), which might have had a disproportionate influence. In terms of variation in region size, the HCP dataset (see above) was used to estimate the population averaged surface area of homolo- gous pairs of left–right regions (i.e., the population mean [L + R]/2 3.2 | Reproducibility, effect size, and sample size per region). Spearman correlation across the 34 cortical regions was then used to relate this regional size variable to reproducibility in the As expected, regionally specific surface area measures (Figure 1b) ENIGMA results, separately for thickness and area asymmetry showed significantly higher reproducibility rates for hemispheric effects. In terms of regional measurement reliability (as a proxy for effects than cortical thickness measures (Area vs. Thickness: t variation in regional image quality and region-specific performance [33] = 6.84, p = 3.11e−09; Mann–Whitney U = 985.0, p = 6.08e−07) of FreeSurfer), we calculated the intraclass correlation coefficients (Figure 1c), as hemispheric effects on surface area are generally larger (ICC) for cortical thickness and surface area measures of each of the (Figure 2). As noted above, there were 63 significant effects and 34 regions using the test–retest dataset of 423 participants from the 7 nonsignificant effects with an uncorrected significance threshold of BIG dataset (see above), and then examined the correlations 0.05, in the meta-analysis of 99 datasets. The reproducibility rate of 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KONG ET AL. 249 F I G U R E 2 Region-wise effect sizes and reproducibility rates of hemispheric asymmetry effects. We used the meta-analytic effect sizes over all 99 datasets for visualization purposes. Effect sizes are in Cohen's d the significant effects ranged from 22.2 to 97% (mean = 63.8%, reproducibility rate of 80% achievable. In addition, the empirical find- SD = 23.7%), while the reproducibility rate of the nonsignificant ings showed that it was impossible to obtain 70% reproducibility for effects ranged from 42.4 to 84.8% (mean = 57.7%, SD = 13.7%). small effects of d < 0.2, even with a relatively large minimum sample Reproducibility showed a significant correlation with the “true” size threshold of 500. effect size for both types of measure (all measures together, rho = .84, We also examined the distributions of per-dataset effect sizes in p = 2.15e−19; thickness, rho = .52, p = .0017; area, rho = .94, relation to dataset sample sizes, and the expected power function to p = 4.28e−16) (Figure 1d). After excluding the 7 nonsignificant effects, detect each meta-analytic effect (Figure S1). There was considerable the correlation between reproducibility and effect size became more variation across datasets around the meta-analytic effect sizes, and as significant (all measures together, rho = .97, p = 5.31e−40; thickness, expected, this variation generally decreased as the sample size rho = .91, p = 7.94e−12; area, rho = .98, p = 3.72e−21). Note that increased (Figure S1). However, some of the larger datasets could also some meta-analytic effects very close to zero could show relatively yield effects that were discrepant with the corresponding meta- high reproducibility (Figure 1d), since in most individual datasets they analytic effects, which again may relate to uncontrolled heterogeneity were also found to be low and nonsignificant as in the meta-analysis affecting these datasets (see below and Discussion). (e.g., surface area asymmetry in superior parietal cortex, d = 0.002, reproducibility rate = 84.8%, and cortical thickness asymmetry in the pars opercularis, d = 0.02, reproducibility rate = 68.7%) (Figure 2). 3.3 | Reproducibility and data heterogeneity Other nonsignificant meta-analytic effects showed relatively lower reproducibility (Figure 1d), which may reflect uncontrolled sources of For subgroups of datasets processed with different versions of dataset heterogeneity affecting regional measurement (see below and FreeSurfer (version 5.3 in 91 datasets and version 5.1 in 6 datasets) Discussion). there was no significant difference in reproducibility of the 70 effects: With a corrected significance threshold of 0.05/70 for identifying mean reproducibility rate 62.8% (SD = 22.8%) for version 5.3 and the “true” effects, there were 56 significant effects, and 14 nonsignifi- 64.3% (SD = 32.8%) for version 5.1 (t = 0.31, p = 0.76; Mann–Whitney cant effects. In this case, the reproducibility rate for significant effects U = 2,693.0, p = .31). Similarly, no significant difference was found ranged from 30.3 to 97%, mean = 67.6%, SD = 22.3%, and for nonsig- between the datasets with scanner field strengths of 3T (63 datasets) nificant effects from 25.3 to 84.8%, mean = 52.0%, SD = 14.4%. versus 1.5T (29 datasets): mean reproducibility rate 59.6% (SD = 23.3%) When examining subgroups of effects and datasets according to for 3T datasets and 66.8% (SD = 25.6%) for 1.5T datasets (t = 1.73, thresholds on effect size and sample size, we found that the reproduc- p = .09; Mann–Whitney U = 2,920.0, p = .503). Furthermore, there ibility rate increased with the minimum sample size threshold, for each was no significant difference in the reproducibility rate of the specific range of effect size (Figure 1e). For example, for effects of 70 effects between the 18 datasets with maximum age 18 years d ≥ 0.6, the reproducibility rate was higher than 90% even when (mean reproducibility 59.7%, SD = 24.7%) versus the 36 datasets with including the datasets with sample sizes as low as 15, while for effects minimum age 19 years (mean reproducibility 61.9%, SD = 24.6%) of 0.4 ≤ d < 0.6, a minimum sample size of 50 was needed to obtain a (t = −0.53, p = .60, Mann–Whitney U = 2,294.5, p = .52). reproducibility rate of 90%. Moreover, for effects of 0.2 ≤ d < 0.4, a In addition, there were no significant differences of sample size minimum sample size threshold of 100 started to make a between the subgroups of datasets (version 5.3 vs. 5.1, t = −0.29, 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 250 KONG ET AL. p = .77; Mann–Whitney U = 203, p = .30:3T vs. 1.3T, t = −0.78, age range in childhood versus adulthood, cortical region size, and p = .44; Mann–Whitney U = 733.0, p = .13: Childhood/adult, regional measurement reliability contribute little to differences in the t = −1.41, p = .17; Mann–Whitney U = 300.5, p = .67), so that these reproducibility of asymmetry effects, for either cortical thickness or factors were not obviously confounded with dataset sample size. surface area. We further ran meta-analyses for the subgroups separately (i.e., 3T/1.5T datasets, and child/adult datasets) to estimate subgroup- specific “true” effects for calculating reproducibility, in case the 4 | DI SCU SSION reduced heterogeneity within subsets might boost reproducibility, although this inevitably resulted in lower numbers of datasets within In this study, we revisited the summary statistics from a worldwide subgroups, compared to the main analysis. Reproducibility of these collaborative neuroscience project that mapped cerebral cortical subgroup-specific meta-analysis effects within the subgroups asymmetry (Kong et al., 2018), to illustrate the reproducibility of real- remained similar to the main analysis (3T: 60.6%, SD = 23.0%; 1.5T: istic biological effects in the absence of p hacking or publishing bias, 69.1%, SD = 23.0%; Child: 61.7%, SD = 23.3%; Adult: 63.8%, based on heterogeneous neuroscience data and typically-used sample SD = 23.0%). sizes. Overall, reproducibility was limited, with a mean rate across all Cortical regional size was significantly correlated across the effects = 63.2%, lowest reproducibility rate = 22.2%. As expected, 34 regions with the effect size of surface area asymmetries (rho = .45, sample size and effect size were the primary drivers of reproducibility, p = .0073), but not with the effect size of thickness asymmetries, and while perhaps surprisingly, heterogeneity factors were of limited not with the reproducibility rates for either thickness or area asymme- influence. try effects (ps > .15; Figure 3a). In terms of measurement reliability Among various factors, low statistical power is now well under- (as a proxy for variation in regional image quality and region-specific stood to contribute to the reproducibility problem (Button performance of FreeSurfer) based on 423 twice-scanned participants, et al., 2013; Ioannidis, 2005). However, low power was only ranked there were no significant correlations between regional measure ICCs number three behind “Selective reporting” and “Pressure to publish” in a and either asymmetry effect sizes or reproducibility rates across the recent Nature survey on contributing factors to irreproducible 34 regions (ps > .15; Figure 3b). One may ask whether it is useful to research (Baker, 2016). In an idealized situation, where there is no apply regional reliability measures from one study to another. We selective reporting or pressure to publish, we found that the repro- investigated the correlations between our reliability measures and ducibility was still limited. As expected, the reproducibility rate those reported in a previous study using an independent scan-rescan increased with the true effect size, as well as the sample size of dataset (Iscan et al., 2015). There were high correlations between the datasets, which together contribute to statistical power. Clearly, to two studies for both thickness (r = .75, p < .0005) and area measures avoid poor reproducibility, a relatively larger sample size is necessary (r = .92, p < .0005). These high correlations indicate stable inter- than was available within many of the individual datasets of this regional differences of measurement reliability across studies. study. For example, to obtain a reproducibility rate of 80% for a true Taken together, our results indicate that the potential heteroge- effect size of around d = 0.4, the sample sizes of individual datasets neity factors of FreeSurfer version, scanner field strength, participant needed be larger than 100, that is, greater than the median sample F I G U R E 3 Scatter plots for the relationships between regional sizes, hemispheric asymmetry effect sizes, and reproducibility rates of the asymmetries. (a) Plots for region sizes. (b) Plots for measurement reliability (intraclass correlation coefficient, ICC). Effect sizes are in Cohen's d. Unit: thickness = mm, area = mm2, region size = mm2 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KONG ET AL. 251 size in this study of 99 datasets. There is therefore substantial room neural structure (LeWinn, Sheridan, Keyes, Hamilton, & to improve reproducibility by increasing sample sizes, even when McLaughlin, 2017). However, one study that investigated variation in using currently available methods. Note that the analysis of brain replicability suggested that the contribution of sample heterogeneity asymmetry involves an inherently paired sample design (i.e., paired left can also be modest (R. Klein, Vianello, et al., 2018). The datasets of and right measures within subjects), but that the overall picture and the current study differed widely in their age ranges and distributions principles illustrated here are broadly applicable. (Table S1). However, we found no significant difference of reproduc- Button et al. (2013) showed that the average statistical power of ibility rates between childhood and adult datasets (thresholded at age studies in the neurosciences is low (i.e., around 21%), which is 18 years), which suggests that this factor was of limited importance. expected to cause low reproducibility, both through false positive and In terms of technical heterogeneity too, there was variation between false negative findings. For example, many fMRI studies have tradi- datasets in terms of scanner field strength and FreeSurfer software tionally been performed using 10–20 participants (Desmond & version (see Table S1), but again we found that these factors did not Glover, 2002). Our observation that reproducibility is strongly significantly relate to reproducibility rates. One reason could be that influenced by sample size was expected, and in line with the PPV cal- these factors primarily affect bilateral measures of cortical thickness culation (Button et al., 2013) mentioned in the Introduction. However, and surface area, that is, mostly equally for the two hemispheres, such here we have demonstrated this empirically in a situation where a that hemispheric asymmetry effects are relatively robust to these fac- priori knowledge of the statistical power (1 − β) and the prestudy tors. Thus the focus on brain asymmetry may have obscured factors odds (R) was not necessary, and in a real-world setting, that is, with affecting both hemispheres equally. Note that our ability to detect a similar heterogeneity to the field in general, as regards factors such as difference in reproducibility between different versions of FreeSurfer scanner field strengths, software versions, demographic differences, was limited, as 91 out of 99 datasets had used version 5.3. and regional differences in measurement reliability. Despite all of this We acknowledge that there remains extensive heterogeneity in heterogeneity, the primary driver of reproducibility remained the sam- other factors such as scanner models and acquisition parameters, ple size in relation to the effect size, which is an important take-home which we were not able to consider in this study because variation message for the field. For example, if the expected effect size was too fragmented across the 99 datasets to do meaningful analysis (i.e., Cohen's d) in a paired-measure MRI study is below 0.2, then stud- of how they affected reproducibility. Recruitment criteria and strate- ies with 500 subjects are still not expected to achieve a reproducibility gies were heterogeneous across datasets too, for example whether rate of 80%. Consistent with this, a recent study performed an empiri- subjects were selected as healthy controls for disorder case–control cal examination of the replicability of “structural brain behavior” asso- studies, or recruited in the context of unselected population studies ciations using a permutation-based approach (again without any of (we did not use data from case participants from case–control problems of selective reporting), and concluded that it is relatively datasets for the present study). Given this heterogeneity, some “non- unlikely to find an association between behavioral traits and brain replication” of effects could be quite appropriate in certain datasets morphology with a sample size of less than 500 (replication effect that comprise specific subgroups or methodological variants, in which sizes were up to 0.4 [Pearson's r]) (Kharabian Masouleh, Eickhoff, particular effects might be of less relevance. Even some of the rela- Hoffstaedter, Genon, & Alzheimer's Disease Neuroimaging, 2019). tively larger individual datasets could sometimes show markedly dif- A reproducibility rate of 36% was reported by the Open Science ferent effects from the meta-analysis (Figure S1). This was likely Framework for 100 findings from psychological studies (Aarts caused by uncontrolled heterogeneity factors affecting those specific et al., 2015), and a reproducibility rate of 54% for 28 classic findings datasets. in psychological science was reported by a more recent Many Lab Although heterogeneity may have contributed to the overall project (R. Klein, Vianello, et al., 2018). Such poor reproducibility has 63.2% reproducibility rate in this study, we regard it as a strength been partly attributed to reporting bias and potentially problematic rather than limitation, as we wished the meta-analytic effect sizes and practices such as selective reporting of outcomes (Aarts et al., 2015; reproducibility rates to be valid in the context of the heterogeneity Baker, 2016; Bakker et al., 2012; Ioannidis, 2005, 2008; Ioannidis typical of the field. Heterogeneity was not therefore an overarching et al., 2014; John et al., 2012; Simmons et al., 2011). While we do not problem for the present study, but rather fitted its purpose. Nonethe- dispute the likely relevance of these factors, it is interesting to note less, future studies may examine how different aspects of MRI dataset that the mean reproducibility rate in the present study, where no such heterogeneity influence reproducibility, to gain further insights into factors were in play, was only 63.2%. As the true effect sizes in the the problem that the neuroscience community is facing. It is important present study ranged from zero to large (Cohen's d up to 1.8), in this to note that, while conceptually related to heterogeneity, the repro- respect they can be taken as broadly comparable to those in the ducibility rate is also influenced by sample and effect sizes human neuroscience and psychology literature, although the effect (as discussed above), and as such provides a useful way of examining size distribution within this range might not be representative of the the replicability of effects under real world conditions. literature at large. Neuroimaging studies can involve considerable flexibility regard- Varying demographic composition of datasets is another factor ing data processing and statistical analysis, while inconsistent strate- likely to influence the reproducibility of findings in human neurosci- gies can also contribute to poor reproducibility and contrasting ence, even for such fundamental processes as age-related change in conclusions (Botvinik-Nezer et al., 2020; Pauli et al., 2016). For the 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 252 KONG ET AL. present study, the pipeline for MRI quality control, processing and for cortical gray matter thickness and surface area measures, but analysis was harmonized, so that the impact of this aspect was neces- other MRI-based metrics will likely differ in the degree to which mea- sarily limited. Therefore our reproducibility rates may be somewhat surement reliability affects reproducibility. Therefore we still recom- idealized, considering how the field typically operates, that is, with dif- mend careful assessment and optimization of measurement reliability ferent researchers asking similar questions, but in different datasets in MRI studies. and using different strategies. In other words, the reproducibility would likely be worse when the processing pipelines and analysis strategies are different. 5 | CONC LU SION Another important aspect affecting reproducibility in the litera- ture may be failure to use blinded designs in primary studies, for Reproducing results is critical for accumulating knowledge in the sci- example so that researchers know the case–control status of partici- entific community. In this study, we revisited the outputs of a global pants while processing their data, and inadvertently introduce bias. collaborative project for mapping cortical brain asymmetry (Kong This is less likely to be relevant for studies based on automated et al., 2018), to empirically demonstrate reproducibility in a real-world processing of human brain MRI data, unless there would be bias dur- setting as regards dataset heterogeneity and sample sizes, but in the ing visual quality control. As regards our study specifically, the visual absence of p-hacking or reporting bias. The results indicated that quality control of FreeSurfer segmentations and parcellations was not there is substantial room to improve reproducibility using current neu- done with respect to eventual asymmetry measures, and we have no roimaging methods, even in the absence of p-hacking or reporting reason to imagine that the inspection of left and right-hemisphere bias, because dataset sample size and effect size remained the primary images was approached differently, on average. drivers of reproducibility, even in the presence of substantial hetero- The lowest reproducibility was 22.2%, for a small but significant geneity across datasets. Despite our focus on gray matter meta-analytic effect of d = 0.052 (cortical area asymmetry in the lin- asymmetries, this picture is likely to hold true to some extent across gual gyrus). As discussed above, such low reproducibility is likely due the field of brain imaging in general. Further studies will be needed to to limited power to detect such small effects, in many of the datasets. evaluate reproducibility in more contexts, in terms of imaging modal- There were also seven nonsignificant meta-analytic effects, with best ity, processing and measurement techniques, and participant demo- estimate effect sizes very close to zero, which were considered to graphics. Our findings suggest that improved reproducibility can be have been reproduced when a given dataset also showed no signifi- achieved primarily through increasing statistical power, either through cant effect. With a significance threshold of 0.05/70 for identifying increasing the sample sizes of individual datasets, or via collaborations “true” effects in the context of multiple testing of 70 cortical mea- between researchers, for example in consortia such as ENIGMA sures, there were 56 significant effects, and 14 nonsignificant effects. (Thompson et al., 2014, 2020). Some of the nonsignificant meta-analytic effects with the lowest effect sizes showed high reproducibility, as nonsignificant effects. ACKNOWLEDG MENTS However, as the reproducibility rates of some nonsignificant meta- This research was funded by the Max Planck Society (Germany). analytic effects were lower than expected at the alpha level 0.05 Funding information for each participating site is available in the (i.e., significant asymmetries were measured in some individual Supporting Information Appendix. Many thanks to an anonymous datasets even when the meta-analysis effect was close to zero and reviewer for the suggestion and a script to generate Supporting Infor- nonsignificant), then uncontrolled dataset heterogeneity, such as tech- mation Figure S1. nical variation affecting asymmetry measurement, is likely to have been involved. Our observations on the reproducibility of nonsignifi- CONFLIC T OF INT ER E STS cant effects highlight the importance of reporting negative findings in Disclosures are listed in the Supporting Information Appendix. publications. We used ICC of regional thickness and surface area measures AUTHOR CONTRIBU TIONS from 423 twice-scanned participants to understand whether inter- X.-Z. K. conceived this study and performed data analyses; X.-Z. K, regional differences in imaging quality might relate to inter-regional ELWG, and C. F. contributed data; X. Z. K. and C. F. designed the differences in reproducibility. There were no significant correlations study, interpreted the results, and wrote the article. All authors pro- for either thickness or surface area measures, which is consistent with vided feedback on the article. the relatively high overall reliability of these regional measures. There were also no significant correlations between reproducibility and DATA AVAILABILITY STAT EMEN T regional size. These observations again underline that dataset size and Data used in this study were published summary statistics from the true effect size were the main drivers of reproducibility identifiable in ENIGMA cortical asymmetry project (Kong et al., 2018). Data and the present study. Nonetheless, measurement reliability has previ- scripts for all analyses are available in GitHub (https://github.com/ ously been shown to play a role in reproducibility, given a specific, Conxz/illusReproducibility). Additional data were from the Human true effect size and sample size (Zuo, Xu, & Milham, 2019). A limita- Connectome Project (https://www.humanconnectome.org/), and the tion of the present study was the focus on cortical asymmetry effects BIG dataset (http://cognomics.nl/). 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License KONG ET AL. 253 ORCID the replicability of associations between brain structure and psycho- Xiang-Zhen Kong https://orcid.org/0000-0002-0805-1350 logical variables. eLife, 8, e43464. Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., Clyde Francks https://orcid.org/0000-0002-9098-890X … Frank, M. C. (2018). A practical guide for transparency in psycholog- ical science. Collabra: Psychology, 4(1), 20. RE FE R ENC E S Klein, R., Vianello, M., Hasselman, F., Adams, B., Adams, R.B., Alper, S., …, Aarts, A. A., Anderson, J. E., Anderson, C. J., Attridge, P. R., Attwood, A., Nosek, B. (2018). Many labs 2: Investigating variation in replicability Axt, J., … Collaboration, O.S. (2015). Estimating the reproducibility of across sample and setting. PsyArXiv. psychological science. Science, 349(6251), aac4716. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahnik, S., Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, Bernstein, M. J., … Nosek, B. A. (2014). Investigating variation in replica- 533, 452–454. bility a “many labs” replication project. Social Psychology—Germany, 45, Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game 142–152. called psychological science. Perspectives on Psychological Science, 7, Kong, X. Z., Boedhoe, P. S. W., Abe, Y., Alonso, P., Ameis, S. H., 543–554. Arnold, P. D., … Francks, C. (2020). Mapping cortical and subcortical Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., asymmetry in obsessive-compulsive disorder: Findings from the Wagenmakers, E. J., Berk, R., … Johnson, V. E. (2018). Redefine statis- ENIGMA Consortium. Biological Psychiatry, 87(12), 1022–1024. tical significance. Nature Human Behaviour, 2, 6–10. Kong, X. Z., Mathias, S. R., Guadalupe, T., Group, E. L. W., Glahn, D. C., Boedhoe, P. S. W., Heymans, M. W., Schmaal, L., Abe, Y., Alonso, P., Franke, B., … Francks, C. (2018). Mapping cortical brain asymmetry in Ameis, S. H., … Twisk, J. W. R. (2018). An empirical comparison of 17,141 healthy individuals worldwide via the ENIGMA Consortium. meta- and mega-analysis with data from the ENIGMA obsessive- Proceedings of the National Academy of Sciences of the United States of compulsive disorder working Group. Frontiers in Neuroinformatics, America, 115, E5154–E5163. 12, 102. Kong, X.-Z., Postema, M., Guadalupe, T., Kovel, C.d., Boedhoe, P.S., Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Hoogman, M., …, Francks, C. (2019). Mapping brain asymmetry in Johannesson, M., … Schonberg, T. (2020). Variability in the analysis of health and disease through the ENIGMA consortium. Human Brain a single neuroimaging dataset by many teams. Nature, 582, 84–88. Mapping, 1–15. http://dx.doi.org/10.1002/hbm.25033. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., LeWinn, K. Z., Sheridan, M. A., Keyes, K. M., Hamilton, A., & Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small McLaughlin, K. A. (2017). Sample composition alters associations sample size undermines the reliability of neuroscience. Nature Reviews between age and brain structure. Nature Communications, 8, 874. Neuroscience, 14, 365–376. Mapping brain asymmetry in health and disease through the ENIGMA Carrion-Castillo, A., Pepe, A., Kong, X.-Z., Fisher, S. E., Mazoyer, B., Tzourio- consortium. Human Brain Mapping, 2020, 1–15. http://dx.doi.org/10. Mazoyer, N., … Francks, C. (2020). Genetic effects on planum temporale 1002/hbm.25033. asymmetry and their limited relevance to neurodevelopmental disorders, Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., intelligence or educational attainment. Cortex, 124, 137–153. Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research cul- Desmond, J. E., & Glover, G. H. (2002). Estimating sample size in functional ture. Science, 348, 1422–1425. MRI (fMRI) neuroimaging studies: Statistical power analyses. Journal of Pauli, R., Bowring, A., Reynolds, R., Chen, G., Nichols, T. E., & Maumet, C. Neuroscience Methods, 118, 115–128. (2016). Exploring fMRI results space: 31 variants of an fMRI analysis in Fischl, B. (2012). FreeSurfer. NeuroImage, 62, 774–781. AFNI, FSL, and SPM. Frontiers in Neuroinformatics, 10, 1–6. Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Andersson, J. L., … Consortium, W.U.-M.H. (2013). The minimal Munafo, M. R., … Yarkoni, T. (2017). Scanning the horizon: Towards preprocessing pipelines for the human connectome project. transparent and reproducible neuroimaging research. Nature Reviews NeuroImage, 80, 105–124. Neuroscience, 18, 115–126. Gronenschild, E. H. B. M., Habets, P., Jacobs, H. I. L., Mengelers, R., Postema, M. C., van Rooij, D., Anagnostou, E., Arango, C., Auzias, G., Rozendaal, N., van Os, J., & Marcelis, M. (2012). The effects of Behrmann, M., … Francks, C. (2019). Altered structural brain asymme- FreeSurfer version, workstation type, and Macintosh operating system try in autism spectrum disorder in a study of 54 datasets. Nature version on anatomical volume and cortical thickness measurements. Communications, 10, 4958. PLoS One, 7, e38234. Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: How much Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. can we rely on published data on potential drug targets? Nature (2014). Publication and other reporting biases in cognitive sciences: Reviews. Drug Discovery, 10, 712–U81. Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive 235–241. psychology: Undisclosed flexibility in data collection and analysis allows Ioannidis, J. P. A. (2005). Why most published research findings are false. presenting anything as significant. Psychological Science, 22, 1359–1366. PLoS Medicine, 2, 696–701. Thompson, P. M., Jahanshad, N., Ching, C. R. K., Salminen, L. E., Ioannidis, J. P. A. (2008). Why most discovered true associations are Thomopoulos, S. I., Bright, J., … Consortium, E. (2020). ENIGMA and inflated. Epidemiology, 19, 640–648. global neuroscience: A decade of large-scale studies of the brain in Iscan, Z., Jin, T. B., Kendrick, A., Szeglin, B., Lu, H., Trivedi, M., … health and disease across more than 40 countries. Translational Psychiatry, DeLorenzo, C. (2015). Test–retest reliability of freesurfer measure- 10, 100. ments within and between sites: Effects of visual approval process. Thompson, P. M., Stein, J. L., Medland, S. E., Hibar, D. P., Vasquez, A. A., Human Brain Mapping, 36, 3472–3485. Renteria, M. E., … Alzheimer's Disease Neuroimaging Initiative, E.C.I.C. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence S.Y.S.G. (2014). The ENIGMA Consortium: Large-scale collaborative of questionable research practices with incentives for truth telling. analyses of neuroimaging and genetic data. Brain Imaging and Behavior, Psychological Science, 23, 524–532. 8, 153–182. Kharabian Masouleh, S., Eickhoff, S. B., Hoffstaedter, F., Genon, S., & Toga, A. W., & Thompson, P. M. (2003). Mapping brain asymmetry. Nature Alzheimer's Disease Neuroimaging, I. (2019). Empirical examination of Reviews. Neuroscience, 4, 37–48. 10970193, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hbm.25154 by Readcube (Labtiva Inc.), Wiley Online Library on [09/10/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 254 KONG ET AL. Valentin Amrhein, S. G. (2017). Remove, rather than redefine, statistical SUPPORTING INF ORMATION significance. Nature Human Behaviour, 2, 4. Additional supporting information may be found online in the Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., & Van Supporting Information section at the end of this article. Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. NeuroImage, 45, S210–S221. Zhen, Z., Kong, X. Z., Huang, L., Yang, Z., Wang, X., Hao, X., … Liu, J. How to cite this article: Kong X-Z, ENIGMA Laterality (2017). Quantifying the variability of scene-selective regions: Inter- Working Group, Francks C, Kong X-Z, et al. Reproducibility in individual, interhemispheric, and sex differences. Human Brain the absence of selective reporting: An illustration from large- Mapping, 38, 2260–2275. scale brain asymmetry research. Hum Brain Mapp. 2022;43: Zuo, X. N., Xu, T., & Milham, M. P. (2019). Harnessing reliability for neuro- science research. Nature Human Behaviour, 3, 768–771. 244–254. https://doi.org/10.1002/hbm.25154