Vilaró et al. BMC Medical Research Methodology (2019) 19:112 https://doi.org/10.1186/s12874-019-0746-4 RESEARCH ARTICLE Open Access Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review Marta Vilaró1,12* , Jordi Cortés1, Albert Selva-O’Callaghan2,3,4, Agustín Urrutia2,3,5, Josep-Maria Ribera2,3,6, Francesc Cardellach2,7, Xavier Basagaña8,9,10, Matthew Elmore1, Miquel Vilardell2,3,4, Douglas Altman11, José-Antonio González1 and Erik Cobo1,2 Abstract Background: From 2005 to 2010, we conducted 2 randomized studies on a journal (Medicina Clínica), where we took manuscripts received for publication and randomly assigned them to either the standard editorial process or to additional processes. Both studies were based on the use of methodological reviewers and reporting guidelines (RG). Those interventions slightly improved the items reported on the Manuscript Quality Assessment Instrument (MQAI), which assesses the quality of the research report. However, masked evaluators were able to guess the allocated group in 62% (56/90) of the papers, thus presenting a risk of detection bias. In this post-hoc study, we analyse whether those interventions that were originally designed for improving the completeness of manuscript reporting may have had an effect on the number of citations, which is the measured outcome that we used. Methods: Masked to the intervention group, one of us used the Web of Science (WoS) to quantify the number of citations that the participating manuscripts received up December 2016. We calculated the mean citation ratio between intervention arms and then quantified the uncertainty of it by means of the Jackknife method, which avoids assumptions about the distribution shape. Results: Our study included 191 articles (99 and 92, respectively) from the two previous studies, which all together received 1336 citations. In both studies, the groups subjected to additional processes showed higher averages, standard deviations and annual rates. The intervention effect was similar in both studies, with a combined estimate of a 43% (95% CI: 3 to 98%) increase in the number of citations. Conclusions: We interpret that those effects are driven mainly by introducing into the editorial process a senior methodologist to find missing RG items. Those results are promising, but not definitive due to the exploratory nature of the study and some important caveats such as: the limitations of using the number of citations as a measure of scientific impact; and the fact that our study is based on a single journal. We invite journals to perform their own studies to ascertain whether or not scientific repercussion is increased by adhering to reporting guidelines and further involving statisticians in the editorial process. Keywords: Reporting guidelines, Peer-review, Reproducibility, Transparency, Number of citations * Correspondence: marta.vilaro@upc.edu 1 Universitat Politècnica Catalunya, Barcelona, Spain 12 Statistical Researcher, Statistics and Operational Research, Barcelona Tech, C/Jordi Girona, 1-3. Edifici C5, planta 2, Campus Nord, 08034 Barcelona, Spain Full list of author information is available at the end of the article © The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 2 of 7 Background (Fig. 1b). In both the IQ and ET studies, the main out- The full progress of science relies on peer review, yet come was an assessed rather than measured endpoint. many have called into question the benefits of peer review As masked evaluators were able to guess the interven- [1–7]. In essence, critics assert that “studies have shown tion arm more often than could be ascribed to chance, that peer reviewers were not able to appropriately detect partially unblinded evaluators could have introduced errors, improve the completeness of reporting, or decrease detection bias in both studies [8]. the distortion of the study results” [8]. Nevertheless, the Due to these limitations, and in order to assess the purposes for which Reporting Guidelines (RG) have been long-term impact of those interventions, we adopted a developed over the past two decades are to help authors, new main outcome: the number of citations that each editors and peer reviewers check and improve the trans- paper received on the WoS from publication up to parency of research studies while ensuring that papers December 312,016, with our hypothesis being that are both accurate and complete [9–14]. According to greater transparency and more comprehensible report- the systematic review published by Bruce et al. (2016) [8], ing may facilitate an increase in citations. which we expand on in Section 7 of the Additional file 1, The IQ study divided the papers into 4 groups as a at least 23 randomized trials have studied some aspects of result of combining the two interventions into a 2 × 2 the peer review process, with the majority of them focus- factorial design: a suggestion to the reviewers to employ ing on the quality of peer review as a surrogate outcome an abridged checklist for the evaluation of basic biomed- while only 3 [15–17] analysed the completeness of report- ical research papers (C) [24]; and adding a statistician ing as an outcome. Of these 3 trials that we previously (S) from the reviewer panel list. Consequently, the 4 conducted, only 2 [15, 16] found positive results regarding groups were defined as: papers which received a stand- completeness of reporting — although only one of these ard review process (reference), papers which received a reached statistical significance. Those studies were based review process using a local checklist (C), papers which on a partly subjective outcome, the Manuscript Quality received a standard review process and a revision from a Assessment Instrument (MQAI) [18], and there is evi- statistician (S) and papers which received a standard dence that evaluators could have successfully guessed review process and a revision from a statistician using a which were in the intervention group [8]. Consequently, local checklist (SC). The reference intervention followed raters could have favoured the group receiving an the usual journal process based on 1–3 reviewers. In additional intervention, thus raising the risk of detection order to combine those results with those of the ET bias. Therefore, we follow up on those studies here by study, only the 24 papers allocated to the group with taking advantage of the Web of Science [19] (WoS) to both interventions (C and S) and the 27 allocated to the reassess those 2 trials by using the number of citations reference group (neither C nor S) were now included in later received by those papers. We consider such a mea- the main analysis. sured outcome to be impartial and fair, as it is naturally The ET study modified this design in 3 ways: first, by free from the risk of evaluation bias. The relationship relying on just one senior methodological expert rather between the completeness of a report and the number of than choosing a statistical reviewer from an expert list; citations has been previously studied, with promising second, by combining both interventions, with the se- though not statistically significant results having been nior methodological reviewer proposing specific changes found [20, 21]. We also previously explored this relation- based on relevant international reporting guidelines; ship with a shorter follow-up (SM, Sections 5 and 6). and, third, it avoided attrition by delaying the interven- tion until the decision had been made on whether or not Methods to publish. We conducted two previous trials [15, 16], in which we Masked to the intervention group, one of us (MV) found partially positive results from adding statistical collected from WoS the number of citations that the ET reviewers and RGs to the peer review process. The first and IQ articles received. A search was made using the one was conducted in 2007 and called the “Improve website’s search tab and including 3 references: (1) the Quality” (IQ) study [15], in which we randomly allocated publication name, “Medicina Clinica (Barcelona)”; (2) 129 suitable manuscripts into 4 intervention groups the publication year (either 2004 to 2005 or 2009 to (Fig. 1a). Unfortunately, after peer review, 16 manu- 2010); and, (3) either the article’s title or by searching scripts were rejected and 14 were lost to follow-up. for the topic in order to consider posterior changes to Those losses introduced unpredictable (attrition) bias the title (between the submitted and finally published [22, 23] and may have affected the estimates. version). Baseline MQAI and study group were obtained The second trial was the 2011 “Enhance Transparency” from the data of the ET and IQ studies. (ET) study [16], in which we randomized 92 manuscripts We aim to estimate the ratio of the average either to both a statistical review and RGs or to neither citation-per-year between intervention arms (which we Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 3 of 7 Fig. 1 Scheme of the allocation of interventions of IQ and ET studies. Groups not included in the main analysis are in a shaded style. R = reference; C=Checklist; S=Statistician; SC = both Checklist and Statistician refer to in this paper as “mean citation ratio”). As the also had a slightly increased number of articles with 0 data did not fit to the distributional assumptions of the citations (Table 1 and Fig. 2). previously masked specified Poisson model, our main Figure 3 shows positive intervention effects that are analysis relies on the more robust Jackknife method, similar in both studies. Combining both estimates, the which provides wider and more conservative intervals. intervention increased the citation rate by 43% (95%CI: 3 As sensitivity analyses, we also report alternative ana- to 98%). This effect is due mainly to the ET study, which lyses such as the previously mentioned Poisson model has higher weight (85.9) in the meta-analysis due to a (Sections 2 to 4 of SM). more precise estimate. The weight of the studies within Additional collected variables are described in Section the meta-analysis has been calculated from the inverse 1 of SM. Section 6 of SM and the master’s thesis of the of the variances of mean ratio estimates, thereby obtain- first author [25] show the results of other exploratory ing 31.58 and 5.17 for ET and IQ, respectively. data analyses that were previously performed with All analyses show some intervention effect (Fig. 3), shorter follow-up. which is slightly larger in the ET study while there is Analyses were performed using R software version greater uncertainty from random error in the IQ study. 3.2.1. In order to check the robustness of the results, we ran sensitivity analyses: a pre-specified Poisson model (which provided shorter and consequently more-precise confi- Availability of data and materials dence intervals); and alternative statistical models that The dataset supporting the conclusions of this article were suitable for count data (Sections 2 to 4 in SM). All is available at https://www-eio.upc.edu/redir/Number together, these provided consistent results. Citations, where researchers can: (1) reproduce the results of our analysis; (2) check our data at the Web Discussion of Science [19] as of December 2016; and (3) update If we consider both studies together, we find that includ- the number of citations in order to replicate our re- ing a methodological reviewer (for example, a senior sults with a longer follow-up. The critical scientist statistician) who is dedicated to looking for missing RG can try to reproduce both our outcome measurements items increases the number of citations by 43% (95%CI: 3 and analyses. to 98%), a result that — if this finding is sustained — might justify the cost and time expenditure by the jour- Results nal [26, 27]. The number of papers with zero-citations Of the 129 randomized papers, 99 IQ articles were pub- was also higher in the intervention groups of both stud- lished between 4 February 2005 and 12 May 2006, with ies, which raises the possibility that greater transparency a mean (standard deviation (SD)) follow-up period of deters citations for some kinds of papers. This unex- 11.35 (0.31) years. Those publications received a total of pected result warrants confirmation in future studies. 927 citations (mean 9.36, SD 14.87). ET included 92 To the best of our knowledge, this is the first study randomized papers that were published between 24 June showing that the completeness of reporting is potentially 2009 and 3 April 2010, with a mean (SD) follow-up associated with higher citation counts as a result of a period of 7.29 (0.31) years. They received a total of 409 specific intervention, namely: adding to the peer review citations (mean 4.44, SD 4.08). In both studies, the process a methodological expert who ensures that the group with both interventions had larger means, stand- reporting guidelines are adhered to. Although the num- ard deviations and annual rates. All intervention groups ber of citations is considered by some authors to be an Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 4 of 7 Table 1 Number of citations by study and intervention group Number of citations Annual rate Articles with 0 citations N1 N2 Mean (SD) Median (Max) Mean (SD) IQ study Standard review process (reference) 37 27 8.4 (12.2) 4 (45) 0.7 (1.1) 1 (3.7%) Statistician 31 26 8.4 (13.7) 4.5 (67) 0.7 (1.2) 4 (15.4%) Checklist 32 22 10.3 (18.8) 4.5 (89) 0.9 (1.6) 3 (13.6%) Statistician + Checklist 29 24 10.7 (15.5) 6.5 (60) 0.9 (1.3) 3 (12.5%) ET study Standard review process (reference) 41 41 3.6 (2.5) 3 (10) 0.5 (0.3) 2 (4.9%) Statistician + Checklist 51 51 5.1 (4.9) 3 (19) 0.7 (0.7) 7 (13.7%) N1 = number of randomized manuscripts; N2 = number of analysed manuscripts indicator of a paper’s influence [28–32], some have of RGs at the end of the editorial phase, after peer argued that “citation counts are an indicator more of review, leads to an increase in the papers’ potential impact than of quality or importance” [28]; thus, we scientific value. This interpretation assumes that all should not conflate the number of citations with re- added citations are “positive” in the sense that they search quality [21, 33]. Due to the high uncertainty contribute to the body of knowledge. behind the IQ study (including the risk of bias due to In interpreting this effect size, we should keep in mind attrition) and the weight of the ET study when esti- the uncertainty reflected by the confidence intervals. mating the combined effects, our interpretation Our next important limitation pertains to the fact that mainly follows the ET results in that the formal use our results rely on just one journal that is not Fig. 2 Number of citations by study and intervention group. Groups not included in the main analysis are in a shaded style Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 5 of 7 Fig. 3 Citations-per-year mean ratio. Point effect estimates are obtained through (1) resampling methods with relaxed distribution assumptions; and generalized linear (GLM) Poisson Models using either (2) non-adjusted or (3) adjusted by follow-up methods. All 95%CI estimates came from the Jackknife method top-quality and they therefore cannot be transported to Conclusions top-tier journals where those interventions have Our findings indicate that the citation counts increased probably already been implemented. According to the by 43% (95% CI from: 3 to 98%) after including in the Scimago Journal Country Rank website, journals with editorial process a methodologist who ensures the Impact Factor ≥ 10 account for just 1% (15,259 out of proper reporting of checklist items. As our original stud- 1,528,749 articles published in 2016) of biomedical ies were originally designed to test those hypotheses for scientific production; thus, our focus is not on the a different outcome, this present study was not powered top-quality journals but on second-tier journals who to test this post-hoc analysis; therefore, our results could benefit from the intervention. should not be interpreted as definitive and they need to It is essential that our results be interpreted accord- be confirmed in properly powered designs. We invite ing to the exploratory nature of this extended journals to perform their own studies to ascertain follow-up study. First, we did not have enough ad- whether or not scientific impact is increased, first, by vance information to know the fit between our data adhering to reporting guidelines, and second, by further and the statistical models. Second, and more import- involving statisticians or methodological experts in the antly, we had neither previous studies to sustain the editorial process. hypothesis nor a sample size rationale to guarantee any desired power for testing this hypothesis. There- Additional file fore, in keeping with the American Statistical Associ- ation (ASA) statement on p-value [34], we should not Additional file 1: Adherence to reporting guidelines increases the interpret the results of any hypothesis test. Accord- number of citations: the argument for including a methodologist in the editorial process and peer-review. Supplementary material. (DOCX 687 kb) ingly, we should also not be concerned about whether or not the 95% confidence intervals (CI) include the Abbrevations neutral value of 1, because there is no such previous ASA: American Statistician Association; C: Intervention group with suggestion hypothesis. However, as we stated prior to data to the reviewers to employ an abridged checklist for the evaluation of basic collection that our objective is “to estimate the effects biomedical research papers; CI: Confidence Interval; ET: “Enhance Transparency” study. Cobo E, Selva-O’Callaghan A, Ribera JM, Cardellach F, of those interventions on the number of citations”, Dominguez R, Vilardell M. Statistical Reviewers Improve Reporting in Biomed- selective outcome reporting is of no concern. ical Articles: A Randomized Trial. Plos One. 2007; 2 (3): e332; IQ: “Improve Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 6 of 7 Quality” study. Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting Received: 22 October 2018 Accepted: 29 April 2019 guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomized trial. BMJ. 2011; 343: d6783; MQAI: Manuscript Quality Assessment Instrument; RG: Reporting Guidelines; S: Intervention group adding a Statistician from the reviewer panel list References statistician; SD: Standard Deviation; SM: Supplementary Material; WoS: Web 1. Smith R. Peer review: a flawed process at the heart of science and journals. of Science J R Soc Med. 2006;99(4):178–82. 2. Baxt WG, Waeckerle JF, Berlin JA, et al. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer Acknowledgements performance. Ann Emerg Med. 1998;32(3 Pt 1):310–7. Not applicable. 3. Shashok K. Content and communication: how can peer review provide helpful feedback about the writing. BMC Med Res Methodol. 2008;8:3. Funding 4. Kravitz RL, Franks P, Feldman MD, et al. Editorial peer reviewers’ DGA, JAG and EC belong to MiRoR (Methods in Research on Research), recommendations at a general medical journal: are they reliable and do which has received funding from the European Union’s Horizon 2020 editors care? PLoS One. 2010;5(4):e10072. research and innovation programme under the Marie Skłodowska-Curie 5. Henderson M. Problems with peer review. BMJ. 2010;340:c1409. grant agreement No. 676207. EC and JC are partially supported by the grants 6. Yaffe MB. Re-reviewing peer review. Scielo Signal. 2009;2(85):eg11. MTM2015–64465-C2–1-R (MINECO/FEDER, UE) from the Secretaría de Estado 7. Stahel PF, Moore EE. Peer review for biomedical publications: we can de Investigación, Desarrollo e Innovación del Ministerio de Economía y improve the system. BMC Med. 2014;12(1):179. Competitividad (Spain) and through grant 2017 SGR 622 from the 8. Bruce R, Chauvin A, Trinquart L, Ravaud P, Boutron I. Impact of interventions Departament d’Economia i Coneixement de la Generalitat de Catalunya to improve the quality of peer review of biomedical journals: a systematic (Spain). The funding bodies had no role in the study design, data collection review and meta-analysis. BMC Med. 2016;14:85. and analysis, results interpretation, or in writing the manuscript. 9. Wang X, Chen Y, Yang N, et al. Methodology and reporting quality of reporting guidelines: systematic review. BMC Med Res Methodol. 2015;15:74. Availability of data and materials 10. Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and The dataset supporting the conclusions of this article is available at https:// accurate reporting increases reliability, utility, and impact of your research: www-eio.upc.edu/redir/NumberCitations, where researchers can: (1) reporting guidelines and the EQUATOR network. BMC Med. 2010;8:24. reproduce the results of our analysis; (2) check our data at the Web of 11. McIntosh L, Juehne A, Vitale C, et al. Repeat: a framework to assess Science [20] as of December 2016; and (3) update the number of citations in empirical reproducibility in biomedical research. BMC Med Res Methodol. order to replicate our results with a longer follow-up. The critical scientist 2017;17:143. can try to reproduce both our outcome measurements and analyses. 12. Park I, Peacey MW, Munafo MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. Authors’ contributions 2014;506(7486):93–6. All authors contributed to and approved the paper, drafted by MV, EC, DGA, 13. Triaridis S, Kyrgidis A. Peer review and journal impact factor: the two pillars JAG, ME, JC and XB. MV collected and analysed the citation data. JAG and EC of contemporary medical publishing. Hippokratia. 2010;14(Suppl 1):5–12. had the original idea. JC analysed the previous ET study. DGA co-designed 14. Shamseer L, Stevens A, Skidmore B, et al. Does journal endorsement of the ET study; JMR, FC, AS, AU and MV designed and conducted both original reporting guidelines influence the completeness of reporting of health studies and contributed to the new idea. XB validated the statistical code research? A systematic review protocol. Syst Rev. 2012;1:24. and results. 15. Cobo E, Selva-O’Callaghan A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS One. 2007;2(3):e332. Ethics approval and consent to participate 16. Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting guidelines during Not applicable. peer review on quality of final manuscripts submitted to a biomedical journal: masked randomized trial. BMJ. 2011;343:d6783. Consent for publication 17. Arnau C, Cobo E, Ribera JM, et al. Effect of statistical review on manuscript Not applicable. quality in Medicina Clinica (Barcelona): a randomized study. Med Clin (Barc). 2003;121(18):690–4. https://doi.org/10.1016/S0025-7753(03)74064-0. Competing interests 18. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before All authors declare: no support from any organisation for the submitted and after peer review and editing at annals of internal medicine. Ann Intern work; no financial relationships with any organisations that might have an Med. 1994;121(1):11–21. interest in the submitted work in the previous three years; no other 19. Web of Science. https://webofknowledge.com. Accessed 1 Jan 2017. relationships or activities that could appear to have influenced the 20. Van der Pol CB, McInnes MDF, Petrcich W, Tunis AS, Hanna R. Is quality and submitted work. completeness of reporting of systematic reviews and meta-analyses published in high impact radiology journals associated with citation rates? PLoS One. 2015;10(3):e0119892. https://doi.org/10.1371/journal.pone. Publisher’s Note 0119892. Springer Nature remains neutral with regard to jurisdictional claims in 21. Nieminem P, Carpenter J, Rucker G, Shumacher M. The relationship published maps and institutional affiliations. between quality of research and citation frequency. BMC Med Res Methodol. 2006;6:42. https://doi.org/10.1186/1471-2288-6-42. Author details 22. Porta N, Bonet C, Cobo E. Discordance between reported intention-to-treat 1 Universitat Politècnica Catalunya, Barcelona, Spain. 2Medicina Clínica, and per protocol analyses. J Clin Epidemiol. 2007;60(7):663–9. Elsevier-Barcelona, Barcelona, Spain. 3Universitat Autònoma de Barcelona, 23. Hernan M, Alonso A, Logan R, et al. Observational studies analysed like Barcelona, Spain. 4Vall D’Hebron Hospital, Barcelona, Spain. 5Hospital randomized experiments: an application to postmenopausal hormone Germans Trias I Pujol, Badalona, Spain. 6José Carreras Leukaemia Research therapy and coronary heart disease. Epidemiology. 2008;19(6):766–79. Institute, Catalan Institute of Oncology, Badalona, Spain. 7Universitat de 24. Bosch F, Guardiola E. Abridged checklist for the evaluation of basic Barcelona and Hospital Clínic, Barcelona, Spain. 8ISGlobal, Centre for Research biomedical research papers. Med Clin. 2003;121(6):228–30. in Environmental Epidemiology (CREAL), Barcelona, Spain. 9Universitat 25. Vilaró M. Master thesis: improving the impact of biomedical research. 2015. Pompeu Fabra, Barcelona, Spain. 10Ciber Epidemiología y Salud Pública https://www.hdl.handle.net/2099.1/26123. (CIBERESP), Barcelona, Spain. 11Centre for Statistics in Medicine, University of 26. Chan AW, et al. Increasing value and reducing waste: addressing Oxford, Botnar Research Centre, Oxford, UK. 12Statistical Researcher, Statistics inaccessible research. The Lancet. 2014;383(9913):257–66. and Operational Research, Barcelona Tech, C/Jordi Girona, 1-3. Edifici C5, 27. Glasziou P, et al. Reducing waste from incomplete or unusable reports of planta 2, Campus Nord, 08034 Barcelona, Spain. biomedical research. Lancet. 2014;383(9913):267–76. Vilaró et al. BMC Medical Research Methodology (2019) 19:112 Page 7 of 7 28. MacRoberts MH, MacRoberts BR. Problems of citation analysis: a critical review. J Am Soc Inf Sci. 1989;40:342–9. 29. Subelj L, Fiala D, Bajec M. Network-based statistical comparison of citation topology of bibliographic databases. Nature. 2014;4:6496. 30. Yao L, Wei T, Zeng A, Fan Y, Di Z. Ranking scientific publications: the effect of nonlinearity. Nature. 2014;4:6663. 31. Zhang Z, Van Poucke S. Citations for randomized controlled trials in sepsis literature: the halo effect caused by journal impact factor. PLoS One. 2017;12(1):e0169398. 32. Martin BR. The use of multiple indicators in the assessment of basic research. Scientometrics. 1996;36(3):343–62. 33. Seglen PO. The skewness of science. J Am Soc Inf Sci. 1992;43(9):628–38. 34. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–33.