Superchi et al. BMC Medical Research Methodology (2019) 19:48 https://doi.org/10.1186/s12874-019-0688-x RESEARCH ARTICLE Open Access Tools used to assess the quality of peer review reports: a methodological systematic review Cecilia Superchi1,2,3* , José Antonio González1, Ivan Solà4,5, Erik Cobo1, Darko Hren6 and Isabelle Boutron7 Abstract Background: A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research. Methods: We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis. Results: We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of ‘quality’. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer’s comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18). Conclusion: Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research. Keywords: Peer review, Quality control, Methods, Report, Systematic review * Correspondence: cecilia.superchi@upc.edu 1 Department of Statistics and Operations Research, Barcelona-Tech, UPC, c/ Jordi Girona 1-3, 08034 Barcelona, Spain 2 INSERM, U1153 Epidemiology and Biostatistics Sorbonne Paris Cité Research Center (CRESS), Methods of therapeutic evaluation of chronic diseases Team (METHODS), F-75014 Paris, France Full list of author information is available at the end of the article © The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 2 of 14 Background as Google® (search date: October 20, 2017) for all reports The use of editorial peer review originates in the eighteenth describing a tool to assess the quality of a peer review century [1]. It is a longstanding and established process that report in biomedical research. Search strategies were generally aims to provide a fair decision-making mecha- refined in collaboration with an expert methodologist nism and improve the quality of a submitted manuscript (IS) and are presented in the Additional file 1. We [2]. Despite the long history and application of the peer hand-searched the citation lists of included papers and review system, its efficacy is still a matter of controversy consulted a senior editor with expertise in editorial [3–7]. About 30 years after the first international Peer Re- policies and peer review processes to further identify view Congress, there are still ‘scarcely any bars to eventual relevant reports. publication. There seems to be no study too fragmented, no hypothesis too trivial [...] for a paper to end up in print’ Eligibility criteria (Drummond Rennie, chair of the advisory board) [8]. We included all reports describing a tool to assess the Recent evidence suggests that many current editors quality of a peer review report. Sanderson and colleagues and peer reviewers in biomedical journals still lack the defined a tool as ‘any structured instrument aimed at appropriate competencies [9]. In particular, it has been aiding the user to assess the quality [...]’ [18]. Building shown that peer reviewers rarely receive formal training on this definition, we defined a quality tool as any struc- [3]. Moreover, their capacity to detect errors [10, 11], tured or unstructured instrument assisting the user to identify deficiencies in reporting [12] and spin [13] has assess the quality of peer review report (for definitions been found lacking. see Table 1). We restricted inclusion to the English Some systematic reviews have been performed to esti- language. mate the effect of interventions aimed at improving the peer review process [2, 14, 15]. These studies showed Study selection that there is still a lack of evidence supporting the use of We exported the references retrieved from the search interventions to improve the quality of the peer review into the reference manager Endnote X7 (Clarivate Ana- process. Furthermore, Bruce and colleagues highlighted lytics, Philadelphia, United States), which was subse- the urgent need to clarify outcomes, such as peer review quently used to remove duplicates. We reviewed all report quality, that should be used in randomized con- records manually to verify and remove duplicates that trolled trials evaluating these interventions [15]. had not been previously detected. A reviewer (CS) A validated tool that clearly defines peer review report screened all titles and abstracts of the retrieved citations. quality in biomedical research is greatly needed. This A second reviewer (JAG) carried out quality control on will allow researchers to have a structured instrument to a 25% random sample obtained using the statistical soft- evaluate the impact of interventions aimed at improving ware R 3.3.3 [19]. We obtained and independently exam- the peer review process in well-performed trials. Such a ined the full-text copies of potentially eligible reports for tool could also be regularly used by editors to evaluate further assessment. In the case of disagreement, consen- the work of reviewers. sus was determined by a discussion or by involving a Herein, as starting point for the development of a new third reviewer (DH). We reported the result of this tool, we identify and describe existing tools that assess the quality of peer review reports in biomedical research. Table 1 Definition of terms used in the present study Structured quality tool: scale or checklist including more than one item aimed at guiding the user to assess the overall quality of a peer review Methods report. Study design Unstructured quality tool: scale or checklist including only one item We conducted a methodological systematic review and inquiring the overall quality of a peer review report. followed the standard Preferred Reporting Items for Sys- Items: elements of a scale or checklist representing a component of tematic Review and Meta-Analysis (PRISMA) guidelines peer review report quality. Items in a scale could or could not have an [16]. The quality of peer review reports is an outcome attached numerical score. If there is no attached score, these items provide the evaluator with a guidance to assess the overall quality of a that in the long term is related to clinical relevance and peer review report. patient care. However, the protocol was not registered in Overall quality score in a scale is measured as: PROSPERO, as this review does not contain direct • Sum of scores: score obtained by summing all scores for each item health-related outcomes [17]. present in a scale. • Mean of scores: score obtained by dividing the sum of scores for each item with the total number of items included in the tool. Information sources and search strategy • Single score: score obtained in those scales based on a single item. We searched PubMed, EMBASE (via Ovid) and The • Summary score: score obtained in those scales with more than one Cochrane Methodology Register (via The Cochrane item deriving from a question inquiring the overall quality of peer review report. Library) from their inception to October 27, 2017 as well Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 3 of 14 process through a PRISMA flowchart [16]. When several reviewer (DH). Authors of the reports were contacted in tools were reported in the same article, they were included cases where we needed further clarification of the tool. as separate tools. When a tool was reported in more than one article, we extracted data from all related reports. Quality components of the peer review report considered in the tools We followed the systematic multi-step approach recently Data extraction described by Gentles [20], which is based on a constant General characteristics of tools comparative method of analysis developed within the We designed a data extraction form using Google® Docs Grounded Theory approach [21]. Initially, a researcher and extracted the general characteristics of the tools. We (CS) extracted all items included in the tools and for determined whether the tool was scale or checklist. We each item identified a ‘key concept’ representing a qua- defined a tool as a scale when it included a numeric or lity component of peer review reports. Next, two nominal overall quality score while we considered it as a researchers (CS and DH) organized the key concepts checklist when an overall quality score was not present. into a domain-specific matrix (analogous to the We recorded the total number of items (for definitions topic-specific matrices described by Gentles). Initially, see Table 1). For scales with more than 1 item we the matrix consisted of domains for peer review report extracted how items were weighted, how the overall quality, followed by items representative of each domain score was calculated, and the scoring range. Moreover, and references to literature sources that items were ex- we checked whether the scoring instructions were tracted from. As the analysis progressed, subdomains adequately defined, partially defined, or not defined were created and the final version of the matrix included according to the subjective judgement of two reviewers domains, subdomains, items and references. (CS and JAG) (an example of the definition for scoring Furthermore, we calculated the proportions of instructions is shown in Table 2). Finally, we extracted domains based on the number of items included in each all information related to the development, validation, domain for each tool. According to the proportions and assessment of the tool’s reliability and if the concept obtained, we created a domain profile for each tool. of quality was defined. Then, we calculated the matrix of Euclidean distances Two reviewers (CS and JAG) piloted and refined the between the domain profiles. These distances were used data extraction form on a random 5% sample of to perform the hierarchical, complete-linkage clustering extracted articles. Full data extraction was conducted by analysis, which provided us with a tree structure that we two reviewers (CS and JAG) working independently for represent in a chart. Through this graphical summary, all included articles. In the case of disagreement, consen- we were able to identify domain similarities among sus was obtained by discussion or by involving a third the different tools, which helped us draw our Table 2 Examples of definition of scoring system instructions Scoring system instructions Defined Partially defined Not defined 5 (Exceptional) = The rare outstanding critique that is 1 (Poor) = Does not follow reviewer guideline structure or 1 = poor; comprehensive, objective, and insightful. Evaluates purpose of preferred formatting in providing comments; unfavourable 2 = fair; the study, study design, scientific validity, and conclusions by timeliness. 3 = good; numbering questions and constructive suggestions to be 2 (Acceptable) = Comments are somewhat helpful; review meets 4 = excellent addressed by the author. Includes comments to the editor about timeline. (Friedam1995) whether this is something new and important and useful to our 3 (Reliable) = Thorough and helpful comments; timely [22] readers. submission. 4 (Very good) = Excellent review indicating that the paper was 4 (Excellent) = Very strong and detailed comments; review was carefully evaluated. Helpful comments to the author and editor submitted early or on time; comments enhance the manuscript’s with well-documented reasons for decision. merit and relevance in the field. 3 (Good) = Useful type of very satisfactory review. Analysis not as (Rajesh 2013) [32] well organized, documented, or as complete as above but is reasonable, with adequate comments for the authors. 2 (Below average) = Very brief, superficial evaluation. Reasons for the decision not explained and comments to authors not helpful. 1 (Unacceptable) = Such a poor review that consideration should be given to not sending further papers to this reviewer. Reasons could include evidence of bias, unfair, faulty reasoning, or evaluation (totally disagrees with the opinion of other reviewers and editor) and comments to author either absent, inappropriate, or inadequate to explain how the paper was rated. (Landkroon 2006) [42] Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 4 of 14 analytical conclusions. The calculations and graphical Six scales consisted of a single item enquiring into the representations were obtained using the statistical overall quality of the peer review report, all of them software R 3.3.3 [19]. based on directly asking users to score the overall quality [22, 25, 29–32]. These tools assessed the quality of a peer review report by using: 1) a 4 or 5 Likert point scale Results (n = 4); 2) as ‘good’, ‘fair’ and ‘poor’ (n = 1); and 3) a re- Study selection and general characteristics of reports stricted scale from 80 to 100 (n = 1). Seventeen scales The screening process is summarized in a flow diagram and one checklist had several items ranging in number (Fig.1). Of the 4312 records retrieved, we finally included from 4 to 26. Of these, 10 used the same weight for each 46 reports: 39 research articles; 3 editorials; 2 infor- item [23, 24, 27, 28, 33–38]. The overall quality score mation guides; 1 was a letter to the editor and 1 study was the sum of the score for each item (n = 3); the mean was available only as an abstract (excluded studies are of the score of the items (n = 6); or the summary score listed in Additional file 2; included studies are listed in (n = 11) (for definitions see Table 1). Three scales re- Additional file 3). ported more than one way to assess the overall quality [23, 24, 36]. The scoring system instructions were not General characteristics of the tools defined in 67% of the tools. In the 46 reports, we identified 24 tools, including 23 None of the tools reported the definition of peer scales and 1 checklist. The tools were developed from review report quality, and only one described the tool 1985 to 2017. Four tools had from 2 to 4 versions [22–25]. development [39]. The first version of this tool was Five tools were used as an outcome in a randomized con- designed by a development group composed of four trolled trial [23, 25–28]. Table 3 lists the general character- researchers and three editors. It was based on a tool istics of the identified tools. Table 4 presents a more used in an earlier study and that had been developed by complete descriptive summary of the tools’ characteristics, reviewing the literature and interviewing editors. Succes- including types and measures of validity and reliability. sively, the tool was modified by rewording some Fig. 1 Study selection flow diagram Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 5 of 14 Table 3 Main characteristics of the included tools 9 tools [24–27, 31, 36, 39, 41, 42]; all reported Characteristics of tools N (%) inter-rater reliability and 2 also reported test-retest reli- Type of tool: ability. One tool reported the internal consistency mea- Scale 23 (96%) sured with the Cronbach’s alpha [39]. Checklist 1 (4%) Quality components of the peer review reports Number of items: considered in the tools with more than one item 1 6 (25%) We extracted 132 items included in the 18 tools. One >1 18 (75%) item asking for the percentage of co-reviews the re- Weight of items a: viewer had graded was not included in the classification Same weight 10 (42%) because it represented a method of measuring reviewer’s performance and not a component of peer Different weight 2 (8%) review report quality. User defined weight 1 (4%) We organized the key concepts from each item into Not applicable 11 (46%)a ‘topic-specific matrices’ (Additional file 4), identifying Score System Instruction: nine main domains and 11 subdomains: 1) relevance of Defined 5 (21%) study (n = 9); 2) originality of the study (n = 5); 3) inter- Partially defined 3 (12%) pretation of study results (n = 6); 4) strengths and weak- nesses of the study (n = 12) (general, methods and Not defined 16 (67%) statistical methods); 5) presentation and organization of Tool development: the manuscript (n = 8); 6) structure of the reviewer’s Reported 1 (4%) comments (n = 4); 7) characteristics of reviewer’s com- Not reported 23 (96%) ments (n = 14) (clarity, constructiveness, detail/tho- Overall quality assessment b roughness, fairness, knowledgeability, tone); 8) timeliness Single score 6 (22%) of the review report (n = 7); and 9) usefulness of the review report (n = 10) (decision making and manuscript Summary score 11 (41%) improvement). The total number of tools corresponding Mean score 6 (22%) to each domain and subdomain is shown in Fig. 2. An Sum score 3 (11%) explanation and example of all domains and subdomains Not reported 1 (4%) is provided in Table 5. Some domains and subdomains a Item weight is not applicable for scale with a single item (n = 6), checklist (n were considered in most tools, such as whether the = 1) and for scale including more than one item without a numerical score reviewers’ comments were detailed/thorough (n = 11) and attached but presenting only a summary score (n = 4) b The total number is different because three tools presented more than one constructive (n = 9), whether the reviewers’ comments way to assess the overall quality and the checklist did not provide an were on the relevance of the study (n = 9) and if the peer overall score review report was useful for manuscript improvement (n = 9). However, other items were rarely considered, questions after some group discussions and a guideline such as whether the reviewer made comments on the for using the tool was drawn up. statistical methods (n = 1). Only 3 tools assessed and reported a validation process [39–41]. The assessed types of validity included Clustering analysis among tools face validity, content validity, construct validity, and We created a domain profile for each tool. For example, preliminary criterion validity. Face and content validity the tool developed by Justice et al. consisted of 5 items could involve either a sole editor and author or a group [35]. We classified three items under the domain ‘Cha- of researchers and editors. Construct validity was racteristics of the reviewer’s comments’, one under ‘Time- assessed with multiple regression analysis using dis- liness of the review report’ and one under ‘Usefulness of criminant criteria (reviewer characteristics such as age, the review report’. According to the aforementioned clas- sex, and country of residence) and convergent criteria sification, the domain profile (represented by propor- (training in epidemiology and/or statistics); or the over- tions of domains) for this tool was 0.6:0.2:0.2 for the all assessment of the peer review report by authors and incorporating domains and 0 for the remaining ones. an assessment of (n = 4–8) specific components of the The hierarchical clustering used the matrix of Euclidean peer review report by editors or authors. Preliminary distances among domain profiles, which led to five main criterion was assessed by comparing grades obtained by clusters (Fig. 3). an editor to those obtained by an editor-in-chief using The first cluster consisted of 5 tools developed from an earlier version of the tool. Reliability was assessed in 1990 to 2016. All tools included at least one item in the Table 4 Descriptive characteristics of tools used to assess the quality of a peer review report a g h Journal or Company Name First Author, Format Quality Overall Items Items Scoring Scoring Scale/ Validity Reliability Internal c RCTs b c i Year defined quality (n) weights range d system Checklist onsistency e f assessment instruction Development Advances in Nursing Science; Issues in Shattell 2010 Scale N Summary 6 S 1–10 N NR NR NR NR 0 Mental Health Nursing; The Journal of [33] Score Holistic Nursing American Journal of Roentgenology Friedman 1995 Scale N Single 1 NA 1–4 N NR NR NR NR 0 [22] Score American Journal of Roentgenology Kliewer 2005 Scale N Summary 4 NA 1–4 N NR NR NR NR 0 [49] Score American Journal of Roentgenology Rajesh 2013 Scale N Single 1 NA 1–4 P NR NR NR NR 0 [32] Score American Journal of Roentgenology Berquist 2017 Scale N Summary 4 NA 0–4 Y NR NR NR NR 0 [50] Score Superchi et al. BMC Medical Research Methodology Annals of Emergency Medicine Callaham 1998 Scale N Single 1 NA 1–5 N NR NR Inter-Rater NR 2m [25] Score (ICC = 0.44, 0.24, 0.12) l Annals of Emergency Medicine Callaham 2002 Scale N Summary 6 NA 1–5 N NR NR Inter-Rater NR 1 [26, 51] Score (ICC = 0.44, 0.24, 0.12) l (2019) 19:48 Annals of Emergency Medicine; Annals Justice 1998 Scale N Summary 4 S 1–5 N NR NR NR NR 0 of Internal Medicine; JAMA; Obstetrics [35] Score & Gynecology and Ophthalmology British Journal of General Practice Moore 2014 Scale N Single 1 NA A-E Y NR NR NR 0 [29] Score British Medical Journal Black 1998 (RQI Scale N Summary 7 S 1–5 N Y Face (N = 20) Test-Retest Internal 5 3.2) [23, 39] Score (Kw = 1.00) Consistency (Cronbach’s Mean Content (N = Inter-Rater alpha = 0.84) 20) (Kw = 0.83) Construct n British Medical Journal Van Rooyen Scale N Mean 8 S 1–5 N NR NR Inter-Rater 2 1999 (RQI 4) (Kw = [27] 0.38–0.67) o Chinese Journal of Tuberculosis Yang 2009 [52] Checklist N NA 5 NA NA N NR NR NR 0 and Respiratory Diseases Journal of Clinical Investigation Stossel Scale N Single 1 NA Good- Y NR NR NR 0 1985 [30] Score Fair- Poor Page 6 of 14 Table 4 Descriptive characteristics of tools used to assess the quality of a peer review report (Continued) a g h Journal or Company Name First Author, Format Quality Overall Items Items Scoring Scoring Scale/ Validity Reliability Internal c RCTs b c i Year defined quality (n) weights range d system Checklist onsistency e f assessment instruction Development Journal of General Internal Medicine McNutt 1990 Scale N Summary 9 S 1–5 N NR Construct NR 1 [28, 40] Score Journal of Vascular Feurer 1994 Scale N Sum 7 D 0–14 N NR Content (N = 2) Inter-Rater 0 Interventional Radiology [41] Preliminary (ICC = 0.84) Criterion (N = 2) (Kendall = 0.94) NA Review quality Scale N Mean 4 User- 0–100 N NR NR NR 0 collector (RQC) defined 2012 [53] weights Nursing Research Henly 2009 [24] Scale N Mean (CAS, 15 S 1–5 P NR NR Inter-Rater 0 GAS scale) (ICC = 0.79) p Superchi et al. BMC Medical Research Methodology Summary 1–5 Score (OAS scale) Summary 0–100 Score (GRQ scale) (2019) 19:48 Nursing Research Henly 2010 [36] Scale N Mean (CAS, 26 S 1–5 P NR NR Inter-Rater 0 GAR, (ICC = SARNR scale) 0.75)p Summary 0–100 Score (GRQ scale) Obstetrics & Gynecology, Landkroon Scale N Summary 5 NA 1–5 Y NR NR Test-Retest 0 Dutch Journal of Medicine 2006 [42] Score (ICC =0.66– 0.88) Inter-Rater (ICC = 0.62) q Pakistan Journal of Jawaid 2006 [34] Scale N NR 5 S 1–5 N NR NR NR 0 Medical Sciences Peerage of science Peerage Essay Scale N Mean 3 S 1–5 N NR NR NR 0 Quality (PEQ) 2011 [37] Publons Academy Review Rating Scale N Sum 4 S 0–3 N NR NR NR 0 and Feedback (Full Form 2016 [38] score: 0–12) Page 7 of 14 Table 4 Descriptive characteristics of tools used to assess the quality of a peer review report (Continued) a g h Journal or Company Name First Author, Format Quality Overall Items Items Scoring Scoring Scale/ Validity Reliability Internal c RCTs b c i Year defined quality (n) weights range d system Checklist onsistency e f assessment instruction Development The Journal of Bone and Thompson Scale N Single Score 1 NA 80–100 Y NR NR Inter-Rater 0 Joint Surgery 2016 [31] (ICC = -4.5 to 0.99) r The National Medical Das Sinha Scale N Sum 5 D 0–100 N NR NR NR 0 Journal of India 1999 [54] a Name of journal or company/organization where the tool was used to assess the quality of their peer review reports b The quality of a peer review report is not clearly defined in any reports c NA Not applicable, S Same weight for each item, D Different weight for each item d NA Not applicable e Y Yes defined, P Partially defined, N Not defined f, g, h NR Not reported i Number of randomized controlled trials where the tool was used as outcome criteria l The ICC was 0.44 for reviewers, 0.24 for editors, and 0.12 for manuscripts m Superchi et al. BMC Medical Research Methodology One article consists of two studies. First study is not a RCT while the second one is a RCT [55] n The overall quality is based on the mean of the first seven items (the item about the tone of the review was not included) o The inter-rater reliability was measured with weighted K for item from 1 to 7 for two editors’ independent assessments p The tool includes more than one scale. We reported inter-rater reliability only for General Review Quality (GRQ) scale q Not reported. Although the authors reported that the reviewers were rated as excellent, good and average based on the quality of the reviews, it is not reported how they assessed the overall quality of peer review reports r ICC range for 11 manuscripts. There was one outlier manuscript that if removed brought the range to 0.87–0.99 (2019) 19:48 Page 8 of 14 Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 9 of 14 Fig. 2 Frequency of quality domains and subdomains characteristics of the reviewer’s comments domain, repre- Discussion senting at least 50% of each domain profile. In the To the best of our knowledge, this is the first compre- second cluster, there were 3 tools developed from 1994 hensive review that has systematically identified tools to 2006. These tools were characterized to incorporate used in biomedical research for assessing the quality of at least one item in the usefulness and timeliness do- peer review reports. We have identified 24 tools from mains. The third cluster included 6 tools that had been both the medical literature and an internet search: 23 developed from 1998 to 2010 and exhibited the most scales and 1 checklist. One out of four tools consisted of heterogeneous mix of domains. These tools were distinct a single item that simply asked the evaluator for a direct from the rest because they encompassed items related to assessment of the peer review report’s ‘overall quality’. interpretation of the study results and originality of the The remaining tools had between 4 to 26 items in which study. Moreover, the third cluster included two tools the overall quality was assessed as the sum of all items, with different versions and variations. The first, second, their mean, or as a summary score. and third cluster were linked together in the hierarchical Since a definition of overall quality was not provided, tree that presented tools with at least one quality com- these tools consisted exclusively of a subjective quality ponent grouped in the domain characteristics of the assessment by the evaluators. Moreover, we found that reviewer’s comments. In the fourth cluster, there are 2 only one study reported a rigorous development process tools developed from 2011 to 2017 that consist of at of the tool, although it included a very limited number least one component in the strengths and weaknesses of people. This is of concern because it means that the domain. Finally, the fifth cluster included 2 tools deve- identified tools were, in fact, not suitable to assess the loped from 2009 to 2012 and which consisted of the quality of a peer review report, particularly because they same 2 domains. The fourth and fifth clusters were sepa- lack a focused theoretical basis. We found 10 tools that rated from the rest in the hierarchical tree that were evaluated for validity and reliability; in particular, presented tools with only a few domains. criterion validity was not assessed for any tool. Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 10 of 14 Table 5 Explanations and Examples of quality domains and subdomains N Domains Subdomains Explanations and Examples 1 Relevance of the study Explanation: Items inquiring if the reviewer has discussed in the peer review report the importance of the research question and usefulness of the study. Example: ‘Did the reviewer give appropriate attention to the importance of the question?’ [28] 2 Originality of the study Explanation: Items inquiring if the reviewer has commented in the peer review report on the originality of the manuscript. Example: ‘Did the reviewer discuss the originality of the paper?’ [23, 27] 3 Interpretation of the Explanation: Items inquiring if the reviewer has commented in the peer review report on how study results authors interpreted and discussed the results of the study. Example: ‘The reviewer commented accurately and productively on the quality of the author’s interpretation of the data, including acknowledgment of the data’s limitations.’ [26] 4 Strengths and weaknesses General Explanation: Items inquiring if the reviewer has identified and commented in the peer review of the study report on the general strong and weak points of the study. Example: ‘How well it identified the study’s strengths and weaknesses?’ [35] Methods Explanation: Items inquiring if the reviewer has identified and commented in the peer review report on the strong and weak points specifically related to study’s methods Example: ‘Did the reviewer clearly identify strengths and weaknesses in the study’s methods?’ [28] Statistical Explanation: Items inquiring if the reviewer has identified and commented in the peer review methods report on the strong and weak points specifically related to study’s statistical methods Example: ‘Confidence intervals/p-values/overall fit’ [36] 5 Presentation and organization Explanation: Items inquiring if the reviewer has made comments in the peer review report on of the manuscript the data presentation such as tables and figures and on the organization of the manuscript such as writing communication. Example: ‘Are there any constructive suggestions on improvement of a. writing; b. data presentation and c. interpretation’ [54] 6 Structure of reviewer’s Explanation: Items inquiring if the reviewer has made in the peer review report organized and comments structured comments. Example: ‘Concise well-organized comments to the editor’ [50] 7 Characteristics of reviewer’s Clarity Explanation: Items inquiring if the reviewer has provided in the peer review report clear and comments easily to read comments. Example: ‘How clear was this review? The review was easily read and interpreted by the editor and authors.’ [38] Constructiveness Explanation: Items inquiring if the reviewer has provided in the peer review report helpful, relevant and realistic comments. Example: ‘Were the reviewer’s comments constructive?’ [23, 27] Detail/ Explanation: Items inquiring if the reviewer has provided in the peer review report detailed and Thoroughness thorough comments supplying appropriate evidence. Example: ‘Detail of commentary’ [33] Fairness Explanation: Items inquiring if the reviewer has provided in the peer review report balanced and objective comments. Example: ‘Balanced/fair’ [24, 36] Knowledgeability Explanation: Items inquiring if the reviewer has showed in the peer review report to know and understand correctly the content of the manuscript. Example: ‘Knowledge of the manuscript’s content area.’ [28] Tone Explanation: Items inquiring if the reviewer has used a courteous tone in the peer review report. Example: ‘Overall tone of the reviewers was also assessed as harsh or courteous.’ [34] 8 Timeliness of the r Explanation: Items inquiring if the reviewer has completed the peer review report on time. eview report Example: ‘Punctuality of the review’ [49] 9 Usefulness of the Decision making Explanation: Items inquiring if the reviewer has provided a peer review report useful to make a review report decision about the acceptance, revision or rejection of a manuscript Example: ‘The reviewer provided the editor with the proper context and perspective to make a decision about acceptance or revision of the manuscript.’ [26] Manuscript Explanation: Items inquiring if the reviewer has provided useful suggestions in the peer review improvement report to improve the manuscript. Example: ‘This aspect is solely interested in how well the review aids the authors for improving their work and/or writing. Whether the review makes a good judgment regarding acceptance of the submission plays no role here whatsoever.’ [53] Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 11 of 14 Fig. 3 Hierarchical clustering of tools based on the nine quality domains. The figure shows which quality domains are present in each tool. A slice of the chart represents a tool, and each slice is divided into sectors, indicating quality domains (in different colours). The area of each sector corresponds to the proportion of each domain within the tool. For instance, the “Review Rating” tool consists of two domains: Timeliness, meaning that 25% of all its items are encompassed in this domain, and Characteristics of reviewer’s comments occupying the remaining 75%. The blue lines starting from the centre of the chart define how the tools are divided into the five clusters. Clusters #1, #2 and #3 are sub-nodes of a major node grouping all three, meaning that the tools in these clusters have a similar domain profile compared to the tools in clusters #4 and #5 Most of the scales with more than one item resulted It is necessary to clearly define what the tool mea- in a summary score. These scales did not consider how sures. For example, the Risk of Bias (RoB) tool [45] items could be weighted differently. Although commonly has a clear aim (to assess trial conduct and not used, scales are controversial tools in assessing quality reporting), and it provides a detailed definition of primarily because using a score ‘in summarization each domain in the tool, including support for judg- weights’ would cause a biased estimation of the mea- ment. Furthermore, it was developed with transparent sured object [43]. It is not clear how weights should be procedures, including wide consultation and review of assigned to each item of the scale [18]. Thus different the empirical evidence. Bias and uncertainty can arise weightings would produce different scales, which could when using tools that are not evidence-based, rigor- provide varying quality assessments of an individual ously developed, validated and reliable; and this is study [44]. particularly true for tools that are used for evaluating n our methodological systematic review, we found only interventions aimed at improving the peer review one checklist. However, it was neither rigorously devel- process in RCTs, thus affecting how trial results are oped nor validated and therefore we could not consider it interpreted. adequate for assessing peer review report quality. We be- We found that most of the items included in the lieve that checklists may be a more appropriate means for different tools did not cover the scientific aspects of a assessing quality because they do not present an overall peer review report nor were constrained to biomedical score, meaning they do not require a weight for the items. research. Surprisingly, few tools included an item related Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 12 of 14 to the methods used in the study, and only one inquired used to evaluate interventions aimed at improving the about the statistical methods. peer review process in RCTs. Furthermore, it would help In line with a previous study published in 1990 [28], editors: 1) evaluate the work of reviewers; 2) provide we believe that the quality components found across all specific feedback to reviewers; and 3) identify reviewers tools could be further organized according to the who provide outstanding review reports. Finally, it might perspective of either an editor or author, specifically by be further used to score the quality of peer review taking into account the different yet complementary uses reports in developing programs to train new reviewers. of a peer review report. For instance, reviewer’s com- ments on the relevance of the study and interpretation of Additional files the study’s results could assist editors in making an editorial decision, clarity and detail/thoroughness of Additional file 1: Search strategies. (PDF 182 kb) reviewer’s comments are important attributes which help Additional file 2: Excluded studies. (PDF 332 kb) authors improve manuscript quality. We plan to further Additional file 3: Included studies. (PDF 244 kb) investigate the perspectives of biomedical editors and Additional file 4: Classification of peer review report quality components. (PDF 2660 kb) authors towards the quality of peer review reports by conducting an international online survey. We will also Abbreviations include patient editors as survey’s participants as their PRISMA: Preferred Reporting Items for Systematic Reviews; RCT: Randomized involvement in the peer review process can further controlled trials; RoB: Risk of Bias ensure that research manuscripts are relevant and Acknowledgments appropriate to end-users [46]. The authors would like to thank the MiRoR consortium for their support, The present study has strengths but also some limita- Elizabeth Moylan for helping to identify further relevant reports and Melissa tions. Although we implemented a comprehensive Sharp for providing advice during the writing of this article. search strategy for reports by following the guidance for Funding conducting methodological reviews [20], we cannot ex- This project was supported by the European Union’s Horizon 2020 research clude a possibility that some tools were not identified. and innovation programme under the Marie Sklodowska-Curie grant agree- ment no 676207. The funders had no role in the study design, data collec- Moreover, we limited the eligibility criteria to reports tion and analysis, decision to publish, or preparation of the manuscript. published only in English. Finally, although the number of eligible records we identified through Google® was Availability of data and materials The datasets supporting the conclusions of the present study will be very limited, it is possible that we introduced selection available in the Zenodo repository in the Methods in Research on Research bias due to a (re)search bubble effect [47]. (MiRoR) community [https://zenodo.org/communities/miror/ Due to the lack of a standard definition of quality, a ?page=1&size=20]. variety of tools exist for assessing the quality of a peer Authors’ contributions review report. Overall, we were able to establish 9 qua- All authors provided intellectual contributions to the development of this lity domains. Between two to seven domains were used study. CS, EC and IB had the initial idea and with JAG and DH, designed the study. CS designed the search in collaboration with IS. CS conducted the among each of the 18 tools. The variety of items and screening and JAG carried out a quality control of a 25% random sample. CS item combinations amongst tools raises concern about and JAG conducted the data extraction. CS conducted the analysis and with variations in the quality of publications across bio- JAG designed the figures. CS led the writing of the manuscript. IB led the supervision of the manuscript preparation. All authors provided detailed medical journals. Low-quality biomedical research im- comments on earlier drafts and approved the final manuscript. plies a tremendous waste of resources [48] and explicitly affects patients’ lives. We strongly believe that a vali- Ethics approval and consent to participate dated tool is necessary for providing a clear definition of Not required. peer review report quality in order to evaluate interven- Consent for publication tions aimed at improving the peer review process in Not applicable. well-performed trials. Competing interests All authors have completed the ICMJE uniform disclosure form at http:// Conclusions www.icmje.org/coi_disclosure.pdf (available on request from the The findings from this methodological systematic review corresponding author) and declare that (1) no authors have support from any company for the submitted work; (2) IB is the deputy director of French show that the tools for assessing the quality of a peer re- EQUATOR that might have an interest in the work submitted; (3) no author’s view report have various components, which have been spouse, partner, or children have any financial relationships that could be grouped into 9 domains. We plan to survey a sample of relevant to the submitted work; and (4) none of the authors has any non- financial interests that could be relevant to the submitted work. editors and authors in order to refine our preliminary classifications. The results from further investigations Publisher’s Note will allow us to develop a new tool for assessing the Springer Nature remains neutral with regard to jurisdictional claims in quality of peer review reports. This in turn could be published maps and institutional affiliations. Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 13 of 14 Author details 23. Black N, Van Rooyen S, Godlee F, Smith R, Evans S. What makes a good 1 Department of Statistics and Operations Research, Barcelona-Tech, UPC, c/ reviewer and a good review for a general medical journal? JAMA. 1998; Jordi Girona 1-3, 08034 Barcelona, Spain. 2INSERM, U1153 Epidemiology and 280(3):231–3. Biostatistics Sorbonne Paris Cité Research Center (CRESS), Methods of 24. Henly SJ, Dougherty MC. Quality of manuscript reviews in nursing research. therapeutic evaluation of chronic diseases Team (METHODS), F-75014 Paris, Nurs Outlook. 2009;57(1):18–26. France. 3Paris Descartes University, Sorbonne Paris Cité, Paris, France. 25. Callaham ML, Baxt WG, Waeckerle JF, Wears RL. Reliability of editors’ 4 Iberoamerican Cochrane Centre, Hospital de la Santa Creu i Sant Pau, C/ subjective quality ratings of peer reviews of manuscripts. JAMA. Sant Antoni Maria Claret 167, Pavelló 18 - planta 0, 08025 Barcelona, Spain. 1998;280(3):229–31. 5 CIBER de Epidemiología y Salud Pública (CIBERESP), Madrid, Spain. 26. Callaham ML, Knopp RK, Gallagher EJ. Effect of written feedback by 6 Department of Psychology, Faculty of Humanities and Social Sciences, editors on quality of reviews: two randomized trials. JAMA. 2002; University of Split, Split, Croatia. 7Centre d’épidémiologie Clinique, Hôpital 287(21):2781–3. Hôtel-Dieu, 1 place du Paris Notre-Dame, 75004 Paris, France. 27. Van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers ’ recommendations : a Received: 11 July 2018 Accepted: 20 February 2019 randomised trial. BMJ. 1999;318(7175):23–7. 28. Mcnutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA. 1990;263(10):1371–6. 29. Moore A, Jones R. Supporting and enhancing peer review in the BJGP. Br J References Gen Pract. 2014;64(624):e459–61. 1. Kronick DA. Peer review in 18th-century scientific journalism. JAMA. 1990; 30. Stossel TP. Reviewer status and review quality. N Engl J Med. 1985; 263(10):1321–2. 312(10):658–9. 2. Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review. 31. Thompson SR, Agel J, Losina E. The JBJS peer-review scoring scale: a valid, JAMA. 2002;287(21):2784–6. reliable instrument for measuring the quality of peer review reports. Learn 3. Smith R. Peer review: a flawed process at the heart of science and journals. Publ. 2016;29:23–5. J R Soc Med. 2006;99:178–82. 32. Rajesh A, Cloud G, Harisinghani MG. Improving the quality of manuscript 4. Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? reviews : impact of introducing a structured electronic template to submit Feasibility of using a fictitious manuscript to evaluate peer reviewer reviews. AJR. 2013;200:20–3. performance. Ann Emerg Med. 1998;32(3):310–7. 33. Shattell MM, Chinn P, Thomas SP, Cowling WR. Authors’ and editors’ 5. Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, William M. Editorial perspectives on peer review quality in three scholarly nursing journals. J peer reviewers’ recommendations at a general medical journal : are they Nurs Scholarsh. 2010;42(1):58–65. reliable and do editors care? PLoS One. 2010;5(4):2–6. 34. Jawaid SA, Jawaid M, Jafary MH. Characteristics of reviewers and quality of 6. Yaffe MB. Re-reviewing peer review. Sci Signal. 2009;2(85):1–3. reviews: a retrospective study of reviewers at Pakistan journal of medical 7. Stahel PF, Moore EE. Peer review for biomedical publications : we can sciences. Pakistan J Med Sci. 2006;22(2):101–6. improve the system. BMC Med. 2014;12(179):1–4. 35. Justice AC, Cho MK, Winker MA, Berlin JA. Does masking author identity 8. Rennie D. Make peer review scientific. Nature. 2016;535:31–3. improve peer review quality ? A randomized controlled trial. JAMA. 9. Moher D. Custodians of high-quality science: are editors and peer reviewers 1998;280(3):240–3. good enough? https://www.youtube.com/watch?v=RV2tknDtyDs&t=454s. 36. Henly SJ, Bennett JA, Dougherty MC. Scientific and statistical reviews of Accessed 16 Oct 2017. manuscripts submitted to nursing research: comparison of completeness, 10. Ghimire S, Kyung E, Kang W, Kim E. Assessment of adherence to the quality, and usefulness. Nurs Outlook. 2010;58(4):188–99. CONSORT statement for quality of reports on randomized controlled trial 37. Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser abstracts from four high-impact general medical journals. Trials. 2012;13:77. KE, et al. Peerage of science: will it work? Trends Ecol Evol. 2012; 11. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of 27(4):189–90. randomized controlled trials with statistically nonsignificant results. JAMA. 38. Publons. Publons for editors: overview. https://static1.squarespace.com/ 2010;303(20):2058–64. static/576fcda2e4fcb5ab5152b4d8/t/58e21609d482e9ebf98163be/ 12. Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of 1491211787054/Publons_for_Editors_Overview.pdf. Accessed 20 Oct 2017. peer review on reports of randomised trials published in open peer review 39. Van Rooyen S, Black N, Godlee F. Development of the review quality journals: retrospective before and after study. BMJ. 2014;349:g4145. instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 13. Lazarus C, Haneef R, Ravaud P, Boutron I. Classification and prevalence of 1999;52(7):625–9. spin in abstracts of non-randomized studies evaluating an intervention. 40. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of BMC Med Res Methodol. 2015;15:85. peer reviewers who produce good-quality reviews. J Gen Intern Med. 14. Jefferson T, Rudin M, Brodney Folse S, et al. Editorial peer review for 1993;8(8):422–8. improving the quality of reports of biomedical studies. Cochrane Database 41. Feurer I, Becker G, Picus D, Ramirez E, Darcy M, Hicks M. Evaluating peer Syst Rev. 2007;2:MR000016. reviews: pilot testing of a grading instrument. JAMA. 1994;272(2):98–100. 15. Bruce R, Chauvin A, Trinquart L, Ravaud P, Boutron I. Impact of interventions 42. Landkroon AP, Euser AM, Veeken H. Quality assessment of reviewers’ reports to improve the quality of peer review of biomedical journals: a systematic using a simple instrument. Obstet Gynecol. 2006;108(4):979–85. review and meta-analysis. BMC Med. 2016;14:85. 43. Greenland S, O’Rourke K. On the bias produced by quality scores in 16. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred reporting meta-analysis, and a hierarchical view of proposed solutions. items for systematic reviews and meta-analyses : the PRISMA statement. Biostatistics. 2001;2(4):463–71. PLoS Med. 2009;6(7):e1000097. 44. Jüni P, Witschi A, Bloch R. The hazards of scoring the quality of clinical trials 17. NHS. PROSPERO International prospective register of systematic reviews. for meta-analysis. JAMA. 1999;282(11):1054–60. https://www.crd.york.ac.uk/prospero/. Accessed 6 Nov 2017. 45. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. 18. Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and The Cochrane Collaboration’s tool for assessing risk of bias in randomised susceptibility to bias in observational studies in epidemiology: a systematic trials. BMJ. 2011;343:d5928. review and annotated bibliography. Intern J Epidemiol. 2007;36:666–76. 46. Schroter S, Price A, Flemyng E, et al. Perspectives on involvement in the 19. R Core Team. R: a language and environment for statistical computing. peer-review process: surveys of patient and public reviewers at two journals. http://www.r-project.org/. Accessed 4 Dec 2017. BMJ Open. 2018;8:e023357. 20. Gentles SJ, Charles C, Nicholas DB, Ploeg J, McKibbon KA. Reviewing the 47. Ćurković M, Košec A. Bubble effect: including internet search engines in research methods literature: principles and strategies illustrated by a systematic systematic reviews introduces selection bias and impedes scientific overview of sampling in qualitative research. Syst Rev. 2016;5:172. reproducibility. BMC Med Res Methodol. 2018;18(1):130. 21. Glaser B, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967. 48. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, 22. Friedman DP. Manuscript peer review at the AJR: facts, figures, and quality et al. How to increase value and reduce waste when research priorities are assessment. Am J Roentgenol. 1995;164(4):1007–9. set. Lancet. 2014;383(9912):156–65. Superchi et al. BMC Medical Research Methodology (2019) 19:48 Page 14 of 14 49. Kliewer MA, Freed KS, DeLong DM, Pickhardt PJ, Provenzale JM. Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American journal of roentgenology. AJR. 2005;184(6):1731–5. 50. Berquist T. Improving your reviewer score: it’s not that difficult. AJR. 2017;209:711–2. 51. Callaham ML, Mcculloch C. Longitudinal trends in the performance of scientific peer reviewers. Ann Emerg Med. 2011;57(2):141–8. 52. Yang Y. Effects of training reviewers on quality of peer review: a before-and- after study (Abstract). https://peerreviewcongress.org/abstracts_2009.html. Accessed 7 Nov 2017. 53. Prechelt L. Review quality collector. https://reviewqualitycollector.org/static/ pdf/rqdef-example.pdf. Accessed 20 Oct 2017. 54. Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews? Natl Med J India. 1999;12(5):210–3. 55. Callaham ML, Schriger DL. Effect of structured workshop training on subsequent performance of journal peer reviewers. Ann Emerg Med. 2002;40(3):323–8.