F1000Research 2020, 9:82 Last updated: 07 OCT 2020 RESEARCH ARTICLE Toolkit of methodological resources to conduct systematic reviews [version 2; peer review: 2 approved] Marta Roqué 1,2, Laura Martínez-García 1,2, Ivan Solà1,2, Pablo Alonso-Coello1,2, Xavier Bonfill1-3, Javier Zamora2,4 1Iberoamerican Cochrane Centre - Sant Pau Biomedical Research Institute (IIB-Sant Pau), Barcelona, Spain 2CIBER of Epidemiology and Public Health (CIBERESP), Madrid, Spain 3Autonomous University of Barcelona, Bellaterra, Spain 4Clinical Biostatistics Unit, Ramón y Cajal Health Research Institute, Madrid, Spain v2 First published: 04 Feb 2020, 9:82 Open Peer Review https://doi.org/10.12688/f1000research.22032.1 Latest published: 11 Aug 2020, 9:82 https://doi.org/10.12688/f1000research.22032.2 Reviewer Status Abstract Invited Reviewers Background: Systematic reviews (SR) can be classified by type depending on the research question they are based on. This work 1 2 identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer health care questions version 2 regarding prevalence, prognosis, diagnostic accuracy and effects of (revision) report report interventions. 11 Aug 2020 Methods: Methodological resources have been identified from literature searches and consulting guidelines from institutions that version 1 develop SRs. The selected resources are organized by type of SR, and report report 04 Feb 2020 stage of development of the review (formulation of the research question, development of the protocol, literature search, risk of bias assessment, synthesis of findings, assessment of the quality of 1. Miranda Cumpston , Monash University, evidence, and report of SR results and conclusions). Melbourne, Australia Results: Although the different types of SRs are developed following University of Newcastle, Newcastle, Australia the same steps, each SR type requires specific methods, differing in characteristics and complexity. The extent of methodological 2. Edward Purssell , City, University of development varies by type of SR, with more solid guidelines available for diagnostic accuracy and effects of interventions SRs. London, London, UK This methodological toolkit describes the most up-to-date risk of bias Any reports and responses or comments on the instruments: Quality in Prognostic Studies (QUIPS) tool and Prediction model study Risk Of Bias Assessment Tool (PROBAST) for prognostic article can be found at the end of the article. SRs, Quality assessment of diagnostic accuracy studies tool (QUADAS- 2) for diagnostic accuracy SRs, Cochrane risk of bias tool (ROB-2) and Risk of bias in non-randomised studies of interventions studies tool (ROBINS-I) for effects of interventions SRs, as well as the latest developments on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Conclusions: This structured compilation of the best methodological resources for each type of SR may prove to be a very useful tool for those researchers that wish to develop SRs or conduct methodological research works on SRs Page 1 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Keywords Systematic reviews, prevalence, prognostic, diagnostic accuracy, efficacy of interventions Corresponding author: Marta Roqué (mroque@santpau.cat) Author roles: Roqué M: Conceptualization, Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing; Martínez-García L: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing; Solà I: Writing – Original Draft Preparation; Alonso-Coello P: Writing – Original Draft Preparation; Bonfill X: Writing – Original Draft Preparation; Zamora J: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing Competing interests: The authors are members of CIBERESP (Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública - Biomedical Research Center Network of Epidemiology and Public Health), and hold active roles within Cochrane and the GRADE Working Group. Grant information: Roqué M is supported by the Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública (CIBERESP) as part of a Training Programme call for “Internal mobility: Internships in CIBERESP groups”, within the framework of the subprogramme 7.4 “Methodology, clinical records and scientific dissemination.” Martínez-García L has a Miguel Servet contract from the Institute of Health Carlos III [CP18/00007]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Copyright: © 2020 Roqué M et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite this article: Roqué M, Martínez-García L, Solà I et al. Toolkit of methodological resources to conduct systematic reviews [version 2; peer review: 2 approved] F1000Research 2020, 9:82 https://doi.org/10.12688/f1000research.22032.2 First published: 04 Feb 2020, 9:82 https://doi.org/10.12688/f1000research.22032.1 Page 2 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 tests that allow us to diagnose the medical condition, and effects REVISED Amendments from Version 1 of interventions to treat the medical condition. SRs can be This new version incorporates a number of clarificaitons and classified by the type of research question they answer, as shown modifications suggested by the peer reviewers, importantly a in Table 1. table organizing the resources identified in this project, classified into guidelines to conduct systematic reviews, reporting guidelines, primary methods papers and references to examples The stages to develop an SR are common to all the types of SRs: used in the manuscript. 1) Formulating the research question, 2) development of the pro- Any further responses from the reviewers can be found at the tocol that explicitly describes the methods to carry out each end of the article step of the SR, 3) literature search, 4) risk of bias assessment, 5) synthesis of findings, 6) assessment of the quality of evi- dence, and 7) report of SR results and conclusions1. Although the different types of SRs share the same structure and follow a Introduction similar development process, their methods can be different and Systematic reviews (SR) are studies that use a systematic and more or less complex depending on the type of SR. explicit method to identify, analyse and synthesize empiri- cal evidence, and to answer a specific research question1. Nowadays there are numerous methodological resources to Therefore, SRs are key tools to make informed health choices2,3. conduct reviews, especially for intervention SRs and diagnos- tic SRs. However, the scattering of these resources and the All SRs are based on a specific research question. Classic epide- lack of widely established manuals or recommendations are, miological research questions relate to the prevalence of a medical in many situations, an obstacle to access them, especially for condition, the associated prognosis of the medical condition prevalence SRs and prognostic SRs. Therefore, the objective (including incidence or global prognosis, prognostic factors of this review is to identify and describe the methodological associated to the condition’s incidence or outcome, and risk resources available to develop prevalence SRs, prognostic SRs, profiles defined by prognostic models4), diagnostic accuracy of diagnostic accuracy SRs and effects of interventions SRs. Table 1. Research question by type of systematic review. Type of systematic Acronym for the research Example of research question review question Prevalence review CoCoPop-S What is the prevalence of frailty and prefrailty (condition) in community- (condition, context, population and dwelling older adults (population) living in low- and middle-income countries study design) (context)?5 What is the worldwide (population) prevalence of insufficient physical activity (condition)6 Prognostic review CoCoPop-S What is the incidence of dementia (condition) in individuals of at least 60 - global prognosis (condition, context, population and years of age (population) living in high-income countries (context)?7 study design) Prognostic review- PICOT-S Is protease activity (prognostic factor) an independent prognostic factor for prognostic factors (population, intervention or factor, wound healing (outcome) at 24 weeks (timeframe) in people with venous leg comparison, outcome, time and ulcers (population)?8 study design) PFO-S (population, factor or model, outcome and study design) Prognostic review- PICOT-S What is best prognostic model to predict overall or progression- free survival prognostic models (population, intervention or factor, (outcome) in patients with chronic lymphocytic leukaemia (condition)?9 comparison, outcome, time and study design) Diagnostic PIRD-S Do self-reported frailty to predict survival in adults with bacterial meningitis accuracy review (population, index test, reference screening instruments (index test) accurately identify older people test, diagnosis of interest and study (population) at risk of frailty and prefrailty (condition of interest)?10 design) Is PET 18F florbetapen (index test) useful in early diagnosing dementia (condition) in patients with mild cognitive impairment (population)?11 Effects of PICO-S What is the effect of ribavirin (intervention) in patients with Crimean Congo intervention review (population, intervention, haemorrhagic fever to prevent death (outcome)?12 comparison, outcome of interest Does comprehensive geriatric assessment (intervention) in older adults and study design) (population) reduce mortality (outcome)?13 Page 3 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Methods results pedagogically with the aim to highlight key differences Information sources and search strategy between review types, present the key characteristics of each We consulted the guidelines from the main organizations that method, and be a comprehensive tool that contains the most establish methods to conduct SRs (Cochrane, Joanna Briggs relevant advice based on the authors judgement. Institute, European Network for Health Technology Assessment (EUNETHTA), Enhancing the Quality and Transparency Results of Health Research (EQUATOR) network, Grading of Recom- We identified guidance handbooks, primary studies and report- mendations Assessment, Development and Evaluation (GRADE)) ing guidelines as a result of the bibliographic searches. The in order to identify their proposed resources. resources selected are presented in Table 2. Additionally, we performed a literature search in MEDLINE We have identified methodological guidelines dedicated to (accessed through PubMed) in November 2019 using the the development of prevalence SRs14, global prognosis15, and following search syntax: ((“Review Literature as Topic”[Mesh] prognostic factor SRs16–18. OR systematic review*[tiab] ) AND (handbook*[ti] OR methodolog*[ti] OR manual[ti] OR guide[ti]). During the performed search, we identified methodologi- cal manuals to develop prognostic model SRs in the series of We also performed ad hoc scientific literature searches to find publications from the PROGRESS project19, and in the resource other resources for each type of SR in relation to the research compilation from Cochrane’s Prognosis Methods Group. question structure, the literature search strategy, the risk of bias assessment and the statistical analysis. For diagnostic accuracy SRs and effects of interventions SRs, we have identified the methodological manuals developed by Eligibility criteria Cochrane Collaboration are available1,20. The recommendations We included the resources available to design prevalence drawn from the guidance handbooks identified are SRs, prognostic SRs, diagnostic SRs and intervention SRs. complemented whenever necessary with specific primary method studies identified in our search. We excluded the methodological resources to develop other types of SRs (methodological, economic evaluation and qualitative Formulating the research question research SRs, or overviews). The type of SR is determined by the research question, which must be formulated in a structured manner as shown in Table 1. Data selection and extraction Careful development of the research question is vital, since The authors are members of CIBERESP (Centro de Investigación the SR inclusion criteria will stem from it. Biomédica en Red de Epidemiología y Salud Pública - Biomedical Research Center Network of Epidemiology and Prevalence review. Prevalence SRs aim to answer the ques- Public Health), hold active roles within Cochrane and the GRADE tion “How common is a health problem in a specific popula- Working Group, and are experts in different fields of knowl- tion?” Prevalence SRs focus on existing cases at a given time, edge (statistics, development of Cochrane reviews, research measure the global burden of a health problem, and describe the methodology, information retrieval, development of clinical guide- characteristics of the affected population, the geographical dis- lines). They evaluated the search results, selected the most rel- tribution of that problem and its variation among subgroups. The evant and accurate resources, and summarized the most relevant structure of the research question must include the elements of information by development stage and type of SR. condition, context, population and study design (CoCoPop-S)21, as shown in Table 1. The most adequate study designs to The resources were selected based on the authors expert judge- estimate the prevalence would be population registers or cross- ment, prioritising those resources which were endorsed or part of sectional studies that include population-representative samples. a guideline from the organisations cited above, and those which For instance, Guthold et al. (2018) considers studies based on were more recent. The resources were organised in 7 sections, population surveys as a reliable source of information to obtain following the development stages of an SR: 1) Formulating the global prevalence estimators of insufficient physical activity6. research question, 2) development of the protocol and review registration, 3) search strategy, 4) risk of bias assessment, Prognostic review. SRs of prognosis are mainly based on three 5) statistical synthesis of findings, 6) quality of evidence assess- types of research questions: 1) “What is the risk of an specific ment, and 7) results report and presentation. The resources are population to have a health problem?”, descriptive question presented by type of SR in each section, and an example of (review of global prognosis) that focuses in new cases their use is included5–13. occurring within a period of time (incidence), 2) “what factors are associated with or determine a specific outcome?”, an explana- For each pre-defined section, the authors selected and summa- tory question (review of prognostic factors), and 3) “are there rized the methods that were considered to be more rigorous and risk profiles that have higher probability of presenting specific widely accepted, prioritizing major methods applicable to all outcomes?”, a result prediction question (review of prognos- reviews over more controversial methods, or methods which tic models or risk prediction). We have excluded from the required highly specialized knowledge. The text organises the aim of this project a 4th type of prognostic question, known as Page 4 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Table 2. Organisation of resources by type. Best practice manuals and Primary methods Reporting chapters of manuals guidelines JBI manual Hemingway 20134 Moher 200972 Aromataris 201714 Munn 201821 Moher 201573 (Munn 201715 Iorio 201522 Beller 201374 Moola 201716 Bossuyt 200623 Zorzela 201675 Campbell 201731) Lijmer 199925 McInnes 201877 Cochrane DTA manual Strauss 201026 Moher 200778 Deeks 201020 Ge 201827 Page 201679 (Bossuyt 200824 Page 201828 Salameh 201980 deVet 200832 Atkinson 201530 Macaskill 201060) Lefebvre 201333 Cochrane intervention manual Glanville 200634 Higgins 20191 Wilczynski 200435 (Lefebvre 201929 Beynon. 201336 Higgins 201955 Sampson 201137 Deeks 201958 Bramer 201738 Chaimani 201963 Hartling 2016 39 Schünemann 201968) Glanville 201440 GRADE Working group manual Isojarvi 201841 Schünemann 201364 Horsley 201142 (Schünemann 202065 Gentles 201643 Schünemann 202066 Hartling 201744 Santesso 201967) Booth 201645 PROGRESS project Rethlefsen 201446 Riley 201917 Rethlefsen 201547 Debray 201719 Spencer 201848 Dekkers 201918 manual Hoy 201249 Hayden 201350 Morgan 201851 Morgan 201952 Wolff 201953 Whiting 201154 Sterne 201656 Lau 199757 Popay 200659 Rutter 200161 Rücker 200862 Murad 201769 Harder 201770 Huguet 201371 Campbell 202076 stratified medicine, and that alludes to the use of prognostic designs that provide more reliable incidence estimates are information to individualise therapeutic choices in a group of prospective cohort studies with representative samples15,22. people with similar characteristics4. Structured questions regarding either prognostic factors or models must include population; exposure in terms of the prognostic Structured questions about global prognosis must specify factor or model of interest, including how it is measured, the population, outcome, condition to be predicted, context and intensity and the exposure time; outcome, condition to be pre- time frame to determine the incidence (CoCoPop-S). The study dicted; follow-up time; and context (PICOT-S or PFO-S)19,21. The Page 5 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 best study designs to evaluate prognostic factors or models are Similarly to the requirement of clinical trial registration, the SR also prospective cohort studies. For instance, Westby et al. (2018) should also be registered in order to avoid redundancies and, published a prognostic factor SR that gives priority to the more importantly, to avoid reporting bias, therefore guaran- inclusion of cohort studies and, if none is found, it resorts to teeing transparency and rigor during the development of the including case-control studies, which also explore the association SR26. Prospective registration of an SR protocol is recommended of prognostic factors with the outcome of interest, although by the PRISMA guidelines and is associated with higher SR with less reliability8. methodological quality27. The largest and most well-known SR register is PROSPERO, produced by the Centre for Reviews Diagnostic accuracy review. Diagnostic SRs aim to answer and Dissemination in York. With PROSPERO, it is possible to the question “How good is a test to identify or dismiss the pres- prospectively register any type of review, provided that its aim is ence of a condition or health problem in a particular population, a health-related outcome. It contains more than 30,000 entries28. in comparison with a reference test?” The research question All Cochrane SR protocols are published in Cochrane Library can be posed with the elements of population, index test, and automatically registered in PROSPERO. reference test, diagnosis of interest and study design (PIRD-S)21. The SR approach will depend on the role of the index test in Search strategy the clinical diagnostic pathway: if it replaces another test, if Designing a comprehensive research study for an SR is vital in it will be used in addition to another test to refine the diagnosis, order to reduce bias when identifying studies, and it is impor- or if it is a triage test previous to other tests23,24. tant to describe it in the relevant section within the protocol in a transparent and thorough manner to facilitate its evaluation Diagnostic SRs preferentially include cr oss-sectional stud- by third parties and its reproducibility. ies, where the participants are evaluated using the index test and/or the reference test to determine if they have the condi- Methodological reference standards to design comprehensive tion of interest. Case-control designs are subject to risk of bias searches have been published29,30. In addition, methodological and their inclusion in diagnostic SRs is not recommended25. manuals to develop SRs provide guidelines for diagnostic and For instance, Ambagtsheer et al. (2017) include in their SR effects of interventions SRs31–33. cross-sectional studies where one or more self-reported frailty screening scales have been compared with one of three reference The design of the search strategies does not differ by type of standards: frailty phenotype, frailty index or comprehensive SR, but rather their differences are due to the elements of the geriatric assessment10. research question and the design of studies to be identified. In general terms, electronic searches are designed to identify Effects of interventions review Interventions SRs aim to answer bibliographic references that use a language similar to the the question “What effect does a specific intervention have elements of the review’s clinical question. To this effect, the on the relevant outcomes in people with a particular health strategies are built based on the elements of the structured clini- problem, in comparison with a reference intervention?” The research cal question. Search algorithms use a combination of natural question is posed with the elements of population, intervention, language and the appropriate controlled vocabulary for each comparator, outcomes of interest and study design (PICO-S)1. bibliographic database. Validated filters can be applied to these strategies to determine specific study designs that can be useful to identify, among others, clinical trials32–34, or prognostic studies35. The randomised clinical trial (RCT) is the most appropriate However, the use of filters is controversial in diagnostic accuracy study design to evaluate the effects of an intervention, as it is studies32,36. the design with less risk of bias and that best helps to establish causality. In cases where it is not possible to conduct randomised Search performance will vary depending on the type of stud- trials for ethical or organizational reasons, non-randomised ies that are included in the SR. Thus, in intervention SRs, the trials, before-after studies, time series, cohort studies or case- search results for RCTs are more precise (they have a higher control studies can be considered for their inclusion in the SR1. proportion of relevant references among all the references that the For instance, the SR by Johnson et al. (2018) regarding riba- search has identified), due to better indexation of this type of virin for treating Crimean Congo haemorrhagic fever included studies in bibliographic databases. On the contrary, in SRs that both RCTs and non-randomised trials to use the available include observational studies, like prognostic SRs, identifying data, given the previous lack of preparedness for experimental studies is more complex given the variability of designs to be research therapeutics in outbreak situations, but concludes that included and its poorer indexation in databases, which results estimates of effect based on the existing literature are highly in less specific literature searches that lead to a longer and more uncertain due to confounding in non-randomised studies12. complex study selection process17. Development of the protocol and review registration Searches must be designed to optimise their sensitivity (the ability Writing the SR protocol is a fundamental step that must be to retrieve as many relevant study references as possible), which done before designing an SR. Herein, the stages and meth- is a feature that tends to be a detriment to precision, which in ods to be applied during the development of the SR can be SRs ranges on an average of 3%37. To obtain an efficient search pre-specified. The identified guidelines can be used to identify with adequate sensitivity, performing searches in MEDLINE the methods that need to be stated in the protocol, and some have and EMBASE may be sufficient, particularly in intervention specific chapters on protocol development1,14,20. reviews, as they are the two most frequently used bibliographic Page 6 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 databases38, and they are enough to identify most relevant stud- of incidence studies considering the duration of the incidence ies for a specific SR39. These searches can be complemented with period7. additional searches in other databases such as PEDro, which provide specific information for certain topics. For the prognostic factor studies, the tools QUIPS and “RoB instrument for NRS of exposures” were identified50–52. The Searching in bibliographic databases can be completed with QUIPS tool helps assess the risk of bias using 31 questions additional strategies, such as checking public trial registers40,41, divided in 6 domains. For each domain, a judgement of high, low searching in the reference list of relevant studies42, or cross- or unclear risk of bias is made. Before using the tool, one must searching citations43. Searching grey literature, understood as any carefully consider the potential confounders that can lead to document that is not published in biomedical or scientific jour- bias. Clinical experts in the specific topic of the SR should nals, has a limited impact in effects of interventions SRs44, but participate. The tool “RoB instrument for NRS of exposures” offers good results in other types of SRs, such as qualitative evaluates the risk of bias using 32 questions divided in 7 domains, evaluation SRs45. including a key domain regarding confounders and a domain regarding departures from intended exposures. For each domain, If we take into consideration the methodological and technical a judgement of critical, serious, moderate or low risk of bias challenges that the design and implementation of search strat- is made. An example of the use of the QUIPS scale can be egies pose, involving a medical librarian can be desirable to seen in the review by Westby et al. (2018). The authors defined improve the search quality46–48. a priori two key confounders (age and infection), which the experts and the literature described as prognostic factors for their Risk of bias assessment condition of interest (venous leg ulcers), and which were simul- Assessing the risk of bias of the included studies is a key taneously associated with the prognostic factor of interest in element in any SR. It helps evaluate and interpret the included the SR (protease activity biomarker). These two confounders studies results, and it is a determinant of the evidence quality were included in the QUIPS scale in the section of control by of the SR results. The current tools to assess risk of bias are confounders8. organised by domains, which roughly correspond to the classic epidemiological biases related to each type of research question. We identified the Prediction model Risk Of Bias ASsessment The identified tools to assess risk of bias are presented in Table 3, Tool (PROBAST) for the prognostic model SRs53. This tool organised by type of SR and by domain of epidemiological assesses the risk of bias using 20 questions divided in 4 domains bias assessed. (participants, predictors, outcome and analysis). For each domain, a judgement of high, low or unclear risk of bias is made. The Each of the domains of these tools includes a number of index questions vary according to the aim of the study (development, questions related to specific aspects of study design or devel- validation, or development and validation of the prognostic opment that can lead to a bias in that domain. The tools can be model). adapted a priori to each review, modifying or deleting ques- tions, or adding new questions specific to the considered research Diagnostic accuracy review The tool QUADAS-2, which evalu- question. The process to assess risk of bias is similar in all the ates 11 questions divided in 4 domains, is available to assess the current scales. Firstly, they identify the risk of bias in each risk of bias in diagnostic accuracy studies54. For each domain, domain based on the answers to the questions, and secondly, they a judgement of high, low or unclear risk of bias is made. In addi- integrate these risks in a risk of bias assessment for each health tion, the external validity or study applicability in relation to the problem, prognostic factor, diagnosed condition or outcome SR is assessed in each domain. of interest assessed, depending on the type of SR. Diagnostic SRs mainly include observational studies, which Prevalence review. The tool to assess risk of bias by Hoy et al. are more subject to risk of bias, and therefore adapting the (2012) is available for prevalence SRs. It assesses internal and QUADAS-2 tool, modifying or adding specific questions to the external validity aspects in the prevalence study49. The tool SR topic, is virtually a requirement during the protocol stage. comprises 10 questions where a judgement of high or low risk For instance, the SR by Martínez et al. (2017) studied the diag- of bias is made. Based on the answers, the researcher makes a nostic accuracy of an imaging test (amyloid PET) that requires subjective assessment of the study’s overall risk of bias as low, complex visual interpretation. For this reason, a question was moderate or high49 . included in the QUADAS scale to assess whether the test interpretation was performed by trained readers11. Prognostic review. There is no scale available to assess the risk of bias in global prognostic studies, although a series of Effects of interventions review. For intervention SRs, the criteria has been proposed to assess risk of bias. These are Risk of Bias (RoB) 2.0 tool is available to assess the poten- classified in 1) definition and representativeness of the population, tial bias in randomised clinical trials, and the Risk Of Bias In 2) completeness of follow-up, and 3) objective and unbiased Non-randomised Studies - of Interventions (RoBiNS-I) tool measurement of outcome of interest22. However, some authors in non-randomised clinical trials55,56. The RoB 2.0 tool includes like Roerh et al. (2018) use a version of the scale to assess risk 16 questions divided in 5 domains, including a specific domain of bias designed by Hoy et al. (2012), adapted to the assessment for randomisation and a domain for deviations from intended Page 7 of 28 Table 3. Tools to assess risk of bias by type of systematic review. Selective outcome Exposure and Confounder Other biases Scale Selection bias Outcome detection Attrition bias(number reporting performance bias bias (number (number of (n items) (number of items) bias (number of items) of items) bias (number of items) of items) items) (number of items) Prevalence Hoy 2012 -  Representativeness of (0) -  Data collection (2) -  Impact of missing (0) (0) -  Appropriate review (10)49 population sample (1) -  Case definition data (1) computation -  Sample and recruitment (2) and timeframe for of prevalence prevalence (2) estimator (1) -  Reliability of measuring instrument (1) Prognostic QUIPS (31)50 -  Study participation (3) -  Prognostic factors -  Outcome definition -  Description and -  Statistical -  Selective -  Statistical review- -  Sample and recruitment (3) definition and and measurement (3) impact of attrition (6) analysis of reporting analysis (3) prognostic measurement (6) confounding of results factors -  Confounders factors (2) (1) definition and measurement (4) RoB for -  Selection of participants (5) -  Exposure -  Outcome definition -  Description and -  Statistical -  Selective (0) NRS definition and and measurement (5) impact of attrition (5) analysis of reporting - exposures measurement (5) confounding of results (32)52 -  Deviations factors (6) (3) from intended exposure (4) Prognostic PROBAST -  Design of study and -  Prognostic factors -  Outcome definition -  Inclusion of (0) -  Selective -  Statistical review- (20)53 selection of participants (2) definition and and measurement (6) participants in the reporting analysis (6) prognostic measurement (3) analysis (2) of results models (1) Diagnostic QUADAS-2 -  Selection of participants (3) -  Index test -  Adequacy and -  Inclusion of (0) (0) (0) accuracy (11)54 interpretation (1) interpretation of participants in the review -  Threshold reference test(2) analysis (1) specification for -  Time interval between index test (1) tests, and coverage of reference test (3) Effects of ROB-2 (16)55 -  Selection of participants -  Blinding of -  Blinding of outcome -  Impact of attrition (3) (0) -  Selective -  Analysis of intervention (randomisation, concealment, participants and detection (2) reporting participants review and basal imbalances) (3) personnel (2) of results in the -  Deviations (2) allocated from intended intervention intervention (2) arm (2) ROBINS-I -  Selection of participants (6) -  Classification of -  Outcome -  Description and -  Confounders -  Selective (0) (35)56 intervention (3) measurement (4) impact of attrition (5) (8) reporting -  Deviations of results from intended (3) intervention (6) No risk of bias tool has been identified for global prognosis systematic reviews. The number of items in the risk of bias tools may vary depending on the effect of interest and the included study designs, as well as the addition or suppression of index questions by the researchers to tailor the tool to the SR. Page 8 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 interventions55. The number of questions may vary, depending parameter of interest is obtained from the estimators of the on the effect of interest and the design of the study assessed. individual studies. Table 4 shows a non-exhaustive compilation For each domain, a judgement is made: high or low risk of bias, or of the main characteristics of the meta-analysis methods and the some concerns. For instance, in their SR, Ellis et al. (2017) main software commands for each type of SR. assessed the risk of bias in the evaluation of results separately for the objective outcomes (such as living at home or death) and A necessary previous step to any meta-analysis is the evalua- for the subjective outcomes, showing a lower risk of bias in the tion of the existing clinical and statistical heterogeneity in the set evaluation of the objective outcomes13. of studies, which will inform us 1) if it is reasonable to perform a quantitative synthesis of findings, 2) what meta-analysis The RoBiNS-I tool assesses the biases that the non-randomised model we should apply, and 3) if additional investigation of study has when compared with an ideal, pragmatic, unbi- the causes of heterogeneity is required, for example, subgroup ased randomised trial, which answers the clinical question of and sensitivity analyses, or meta-regressions57,58. In those cases interest (even if this ideal trial may not be feasible or ethical)56. when a quantitative synthesis is precluded, the SR will be RoBiNS-I has 34 questions divided in 7 domains, including a restricted to a narrative synthesis. A narrative synthesis should key domain regarding confounders and a domain for deviations not simply summarize the findings from the included studies in from intended interventions. As in the case of prognostic order to draw conclusions about the body of evidence, but instead SRs, there should be an a priori careful consideration of the should be a more formal process which includes a formulation potential confounders that must be included in the tool to assess of the theory of how the intervention works, why and for whom, individual studies. A judgement of critical, serious, moderate or the exploration of the relationships in the data, and the assessment low risk of bias is made for each domain. A low risk of bias implies of the robustness of the synthesis59. that the non-randomised study is comparable to a well-performed randomised trial. For instance, Johnson et al. (2018) excluded When it is reasonable to perform a statistical synthesis, there from their analyses the non-randomised studies that showed a are two main models to conduct a meta-analysis: fixed effects critical risk of bias according to RoBiNS-I, rejecting 18 out model and random effects model. For practical purposes, of the 22 included studies12. the chosen model determines how the studies included in the meta-analysis will be numerically weighed. Both models are Statistical synthesis of findings based on different assumptions regarding distribution of effects SRs may include a section with a quantitative statistical and heterogeneity in the set of studies, and they differ in their synthesis or meta-analysis, where a combined estimator of the application and interpretation58. Table 4. Methodological characteristics of meta-analysis by type of systematic review. Assessment of Command Measures to combine Model Method heterogeneity (package) Prevalence review -  Proportion (prevalence) -  Qualitative -  Fixed/Random -  Inverse-variance -  Metaprop (Stata) effects methoda Prognostic review -  Cumulative incidence -  Meta-regression -  Fixed/Random -  Inverse-variance -  Metan (Stata) - global prognosis -  Incidence rate effects methodb -  Metaprop (Stata) -  Review Manager Prognostic review- -  Hazard Ratio -  Meta-regression -  Random effects -  Inverse-variance -  Metafor (R) prognostic factors -  Odds Ratio method Prognostic review- -  Calibration -  Meta-regression -  Random effects -  Multivariate -  Metamisc (R) prognostic models -  Discrimination methods Diagnostic -  Sensitivity -  Meta-regression -  Random effects -  HSROC methodc -  Metadas (SAS) accuracy review -  Specificity -  Bivariate model -  Metandi (Stata) Effects of -  Mean difference -  I2 -  Fixed/Random -  Mantel-Haenzsel -  Metafor (R) intervention review -  Risk difference -  Meta-regression effects method -  Metan (Stata) -  Standardised mean -  Multivariate -  Review Manager difference methods -  Hazard Ratio -  Incidence rate ratio -  Odds Ratio -  Risk ratio a Tukey-Freeman or logit transformation. bTransformation for the cumulative incidence. c Hierarchical summary receiver operating characteristic (HSROC) method allows estimation of a Receiver operating characteristic (ROC) curve or sensitivity and specificity indexes. Page 9 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Finally, there is a variety of resources to conduct meta-analyses, heterogeneous results, and it is recommended to use the random from specific programs to perform meta-analyses (free or effects model by default and perform a comprehensive exami- paid) to user-defined routines using general statistics packages nation of the sources of heterogeneity using meta-regression60. (SAS, Stata, SPSS), as well as Excel utilities or R libraries. For instance, the protocol of the SR by Ambagtsheer et al. An archive with software and utilities is available from SR (2017) expects to estimate an average sensitivity and specificity Tool Box. for the frailty scales, when the included studies have applied the same explicit cut-off points to the considered scales. However, Due to the complexity of the statistical techniques to synthe- given that they are subjective, self-reported scales, the studies sise results, and the difficulty to standardise methods and deci- could share the same explicit cut-off point, and yet that cut-off sions to be made during the analysis, it is vital to involve a point could correspond to different levels of frailty in the stud- statistician in the planning and conduct stages of the meta-analysis, ies (implicit thresholds), which will advise against calculating especially for prognostic and diagnostic SRs. pooled estimates of diagnostic accuracy10. Prevalence review. In prevalence SRs, the meta-analysis Effects of interventions review. In intervention SRs, the combines ratios, which are transformed to be meta-analysed meta-analysis combines different measures, depending on the using the inverse-variance method58. Siriwardhana et al. (2018) type of outcome: odds ratio or risk ratio for binary outcomes, calculated combined frailty prevalence estimates using a random mean difference or standardised mean difference for continuous effects model. The authors assessed that there was high clinical outcomes, hazard ratio for time-to-event outcomes, and incidence heterogeneity between the studies in terms of actual frailty rate ratios for outcomes that count number of events. prevalence, geographic setting, frailty assessment method, cut-off points applied and sample age, although this heterogeneity In intervention SRs, the I2 estimator has been proposed to assess did not rule out performing a meta-analysis5. statistical heterogeneity as a supplement to the assessment of clinical and methodological heterogeneity. This indicator is Prognostic review. In global prognostic SRs, the meta-analysis defined as the percentage of the overall variability that cannot be combines cumulative incidence ratios or incidence rates, while explained by chance, and has values ranging from 0% to 100%; in prognostic factor SRs, the meta-analysis combines odds with higher values indicating higher statistical heterogeneity59. ratios or hazard ratios, which can be presented in individual For instance, in the SR by Ellis et al. (2017), the authors estab- studies as raw estimates or as covariate-adjusted estimations lished a 70% heterogeneity limit for I2, beyond which a derived from logistic or Cox regression models. If combining meta-analysis combining the results would not be performed13. adjusted estimates, all of them should be adjusted by a mini- Despite its popularity and ease of interpretation, the use of this mum set of common factors17. In prognostic model SRs, the indicator is not exempt of controversy due to its dependence meta-analysis combines estimates of model discrimination and on the number of studies and sample size; thus, a small statisti- calibration. These indicators can be synthesised separately or cal heterogeneity could seem substantial only by the effect of a jointly using multivariate models19. large sample size of the included studies62. Prognostic studies usually show significant variability in terms In intervention SRs, pairwise metanalysis has been extended to of design, sample case-mix, measurement instruments, analy- network metanalysis, which allows the simultaneous comparison sis methods and presentation of results17. Therefore, in prog- of three or more interventions, combining direct and indirect nostic factor and model SRs, it is recommended to perform the evidence from a network of studies63. meta-analysis using the random effects model, and even to use multivariate meta-analysis methods adjusting for relevant Quality of evidence factors17. For instance, the SR by Westby et al. (2018) describes The quality (also confidence or certainty) of evidence in an how the authors dismissed performing a meta-analysis due to SR is the degree of confidence that is held against the fact that the high risk of bias and the extreme heterogeneity across the an estimate of effect or association is close to the actual value included studies in terms of population, measurement of the of interest1. Certainty of evidence is best evaluated with the prognostic factor (cut-off points and analytical methods) and GRADE system. Certainty in the obtained estimates for each one outcome measurement8. of the key SR outcomes or factors is classified as high, moderate, low or very low. A level of certainty of evidence is first established Diagnostic accuracy review. In diagnostic SRs, the meta-analysis from the design of the studies that form the evidence body, combines estimates of sensitivity and specificity of the index which might or might not have an optimal design for the type of test. The meta-analysis in diagnostic SRs shows a higher degree considered question. This initial confidence in the evidence body of complexity because the studies may have used different can then decrease in one or two levels if the following is detected: thresholds, both implicit and explicit, to define a positive 1) design or execution limitations, 2) inconsistency between result in the evaluated test. This leads to a correlation between estimates, 3) indirect evidence, 4) imprecision in estimates, the sensitivity and specificity indexes, which must be modelled or 5) publication bias64. jointly using multivariate methods60. The most common avail- able statistical methods are the bivariate hierarchical model and The certainty of evidence is a key element to interpret and the HSROC model (Hierarchical summary receiver-operating communicate results, and as such, it should be included in the characteristic)61. Diagnostic SRs tend to combine studies with very sections of results, discussion, conclusion and abstract, using Page 10 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 semi-standardised statements65. Additionally, it can be included inconsistency, because heterogeneity is common and hard to in a Summary of Findings table, where for each comparison, the quantify in diagnostic SRs, and it often cannot be explained even key information regarding relative effect and absolute effect if multivariate models are adjusted. Judgments on extent of magnitude, quantity of available evidence and its certainty is heterogeneity should be based on similarity of the point presented66. Certainty of evidence can be assessed too when estimates, overlap of confidence intervals, and the exploration of no quantitative synthesis is possible67. possible explanations for the inconsistency from subgroup or sensitivity analyses70. We will now highlight the specific aspects in which the GRADE system adapts to each type of SR. Imprecision judgments should be based on both the width of the confidence or credible intervals for sensitivity and Prevalence review. There are no formal adaptations of the GRADE specificity, but also on the implications for patient management in system for prevalence SRs, but there is a proposal to assess terms of true and false positives, and true and false negatives. the quality of the evidence based on this system68. High initial When the estimated intervals include values that may lead to certainty is awarded to survey or cross-sectional study designs different conclusions of the test’s value, the certainty of the with population representativeness that have been properly evidence may be lowered71. designed and conducted, while studies with no population repre- sentativeness will have lower initial quality. With regard to the criteria to increase the level of evidence, it is unclear whether they should be applied at all and how to do it Prognostic review. There is a GRADE proposal for global in diagnostic SRs71. The uncertainty surrounding the process prognostic SRs22 and an adaptation for prognostic factor SR69. of assessing the quality of evidence in diagnostic SRs explains Guidelines for prognostic model SR are still under development. why it is not a requirement in Cochrane SRs at the moment. For instance, the SR by Martínez et al. (2017) only included In global prognostic SRs, the study designs that have high a Summary of Findings table with numerical results and an initial certainty are longitudinal cohort studies and pragmatic estimation of the absolute effect that the test would have on a randomised controlled trials with representative samples22. Other hypothetical cohort of individuals11. observational designs would offer low initial certainty. In prognostic factor SRs, explanatory and confirmatory longitudinal Effects of interventions review. The GRADE system for assess- designs offer high initial certainty, while exploratory studies are ing the quality of evidence was initially developed for interven- considered to be of moderate quality69. tion SRs, and it is the indication for which clearer and widely agreed guidelines are available64. In terms of study design, In prognostic SRs, the assessment of the limitations follows RCTs are initially classified as having high certainty, while the general procedure already described, with two particu- all non-randomised or observational studies are classified as larities: 1) qualitative assessment of inconsistency, because of having low certainty. This proposal for pairwise metanalysis can low reliability of I2 estimator in the prognostic field22,69, and be extended to network metanalysis63. 2) possibility of increased certainty in the studies that do not show limitations in the quality of evidence, if (i) the estimated effect The assessment of the certainty limitations is well-defined in magnitude is substantial, or (ii) there is an exposure-response intervention SRs. Inconsistency can be assessed using the I2 gradient67. For instance, the prognostic factor SR by Westby estimator64. Imprecision is assessed taking into account whether et al. (2018) considered the possibility of increasing the certainty the review meets the optimal information size, and whether of evidence in the studies presenting no limitations. Due to the the confidence interval of the effect estimate allows reaching a exploratory nature of the included studies and their high risk conclusion, because either it only includes values consistent with of bias, the certainty was not increased in any case and the a relevant intervention effect, or it completely dismisses it63. In evidence obtained in the review was of very low quality8. observational studies that do not have limitations in the quality of evidence, three criteria are considered to increase certainty: 1) the Diagnostic accuracy review. There is a GRADE proposal estimated effect magnitude is important or very important, for assessing the certainty of evidence for test accuracy70,71. 2) there is an exposure-response gradient, and 3) all possible The study designs that start with the highest degree of evidence biases that could reduce the observed effect confirm the obtained are cohort or cross-sectional studies where the index test and an conclusions. appropriate reference standard have been directly compared in patients with diagnostic uncertainty70. If the SR included For instance, the SR by Ellis et al. (2017) applied the GRADE case-control studies, these would offer low-quality initial system to the included randomised trials, and it concluded evidence25. that there was high certainty of the effect of the compre- hensive geriatric assessment on the effects outcomes based Indirectness of evidence would be assessed through any appli- on a high number of studies and participants, with a globally cability concerns of the patient sample, the intervention and low risk of bias, and results consistent among studies. However, the comparator with respect to the clinical pathway where the the certainty of evidence obtained in cost-effectiveness was low, test is to be applied. There is uncertainty regarding how to assess due to imprecision and inconsistency of results13. Page 11 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Results report new formats for synthesis and presentation of SR results It is vital to inform about the methods, results and conclusions are being explored nowadays to help their dissemination and of the SRs in a transparent and thorough manner so that their the adoption of their conclusions in clinical practice and users can interpret, evaluate and apply them. The EQUA- healthcare decision-making. For instance, new formats for result TOR initiative has developed, and keeps up-to-date, a library presentation and Summary of Findings tables are being proposed, with guidelines to communicate the different types of research adapted to the profile of their potential users86,87. studies. The PRISMA statement (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) has been proposed in Limitations and strengths the SR field72. This statement consists of a checklist comprised An inherent limitation of this project is its methodology based of 27 items and a flow diagram to present the number of studies on a selection of resources and summary of guidance informed considered in the SR. In addition, several extensions focusing on by expert opinion, which may be susceptible to implicit reporting specific aspects of SRs have been developed, such as selection biases or lack of comprehensiveness. PRISMA-P for reporting SR protocols73, PRISMA- Abstracts for reporting abstracts74, and PRISMA- Harms for reporting The four types of SRs considered in this paper are fundamen- harms outcomes in SRs75. Additionally, the SWiM guideline tal to define preventive activities and public health policies, as is available for reporting intervention SRs where the effects of well as to make health decisions. The selection of resources interventions are synthethised narratively without metanalysis, done is not dependent on whether the reviewer explores focusing on the key features of narrative information synthe- questions on efficacy or effectiveness, often described as explan- sis (grouping of studies, presentation of data and summary atory or pragmatic questions, and will be useful to the research- text, and appropriate discussion of limitations of this type of ers regardless of their intended purpose. However, we have synthesis)76. not considered the resources to conduct in-depth exploration of effectiveness issues such as reviews of complex interven- tions or implementation reviews. Additionally, this research Although the PRISMA statement and the cited extensions are has not considered other types of SRs, such as methodological, focused on intervention SRs, a specific PRISMA extension economic evaluation and qualitative research SRs, for which it has also been developed for diagnostic SRs77. On the contrary, would be convenient to perform similar methodological no tools have been identified to communicate prevalence or prog- compilations. Reviews of reviews (or overviews) were also not nostic SRs. In recent years, clarity and transparency in study considered, and as such, there are a number of review-level communications has improved thanks to the development of resources which have not been discussed, for example the risk of checklists for scientific paper publication, although there is bias assessment tool ROBIS or the methodological assessment tool still room for improvement78–80. AMSTAR88,89. Another limitation of this research is the need to keep it up to date, given the speed at which the methods Discussion and methodological resources to develop SRs are updated. Key results This review identifies and describes the most relevant methodo- On the other hand, the main strengths of this paper are its trans- logical resources to conduct prevalence, prognostic, diagnostic versal approach for the different types of reviews, and the accuracy and effects of interventions SRs. This review offers identification of resources for all the stages in the develop- a general and comparative perspective of the methodological ment of an SR. There are few previous publications that offer a resources by SR stage, highlighting the differential elements transversal perspective of the different types of systematic of each type of SR. This project does not aim to be a stan- reviews, and these are focused on a specific stage of the review dalone tool for a researcher to find complete guidance on how to or on a particular topic. For instance, the work carried out by conduct and report a review, but rather it aims to be a sign- Munn et al. (2018) defined a typology for SRs, characterised from post pointing out to the resources where researchers may find in 10 different types of research questions, and delving into the for- depth guidance to develop their reviews. mat of each type of question21. Pollock et al. (2017) review the steps of an SR for 5 types of question, specifically focusing on Current context the particularities of the reviews on stroke rehabilitation90. Muka This paper corroborates that developing a rigorous SR is a com- et al. (2019) offer a structured compilation of resources for plex and resource-intensive task81,82. In order to tackle the increas- each SR stage, but without delving into the specificities of the ing complexity of SRs and ensure the adoption of rigorous different types of SRs91. Finally, organising the resources to assess methodology, it is necessary that the reviews are made by multi- the risk of bias by type of review is a strength and a novelty disciplinary work groups with knowledge and experience in compared with previous works, which compile the quality methodology (such as statistical analysis and information assessing tools by type of study design but without linking them retrieval)83,84. In addition, it is important to consider the increas- to the aim of the study nor the type of systematic review92,93. ing availability of artificial-intelligence-based technological tools, which make it possible to semi-automate the different steps Conclusions of the SR development, and thus reduce the time and SRs are a key research tool to make decisions in healthcare, human resources required to conduct the review85. public health and medical research. There are methods and resources to develop high-quality reviews to answer most types of Once the rigorous SR has been developed, ensuring the con- clinical questions. This review offers a complete resource guide veyance of the generated knowledge is essential. In this sense, for prevalence, prognostic, diagnostic and intervention reviews, Page 12 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 and is a very useful tool for those researchers that wish to Acknowledgements develop SRs or conduct methodological research works in that Marta Roqué i Figuls is presently working on her PhD with field. the PhD Programme in Biomedical Research Methodology and Public Health from the Autonomous University of Barcelona. Data availability Underlying data The authors are indebted to the helpful peer reviewer com- All data underlying the results are available as part of the article ments of Dr Cumpston and Dr Purssell, which have helped and no additional source data are required. improve the manuscript. References 1. Higgins JPT, Thomas J, Chandler J, et al.: (editors). Cochrane Handbook Manual. The Joanna Briggs Institute. 2017. [Accessed on 10/12/2018]. for Systematic Reviews of Interventions version 6.0 (updated July 2019). Reference Source Cochrane. 2019. [Accessed on 29/11/2019]. 17. Riley RD, Moons KGM, Snell KIE, et al.: A guide to systematic review and meta- Reference Source analysis of prognostic factor studies. BMJ. 2019; 364: k4597. 2. Urrútia G, Bonfill X: Revisiones sistemáticas, una herramienta clave para la PubMed Abstract | Publisher Full Text toma de decisiones clínicas y sanitarias. Rev Esp Salud Pública. 2014; 88(1): 18. Dekkers OM, Vandenbroucke JP, Cevallos M, et al.: COSMOS-E: Guidance on 1–3. conducting systematic reviews and meta-analyses of observational studies of Publisher Full Text etiology. PLoS Med. 2019; 16(2): e1002742. 3. Ferreira González I, Urrútia G, Alonso-Coello P: Systematic reviews and meta- PubMed Abstract | Publisher Full Text | Free Full Text analysis: scientific rationale and interpretation. Rev Esp Cardiol. 2011; 64(8): 19. Debray TP, Damen JA, Snell KI, et al.: A guide to systematic review and meta- 688–96. analysis of prediction model performance. BMJ. 2017; 356: i6460. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text 4. Hemingway H, Croft P, Perel P, et al.: Prognosis research strategy (PROGRESS) 20. Deeks JJ, Bossuyt PM, Gatsonis C, (editors): Cochrane Handbook for Systematic 1: a framework for researching clinical outcomes. BMJ. 2013; 346: e5595. Reviews of Diagnostic Test Accuracy Version 1.0. The Cochrane Collaboration. PubMed Abstract | Publisher Full Text | Free Full Text 2010; [Accessed on 21/12/2018]. 5. Siriwardhana DD, Hardoon S, Rait G, et al.: Prevalence of Frailty and Prefrailty Reference Source Among Community-Dwelling Older Adults in Low-Income and Middle-Income 21. Munn Z, Stern C, Aromataris E, et al.: What kind of systematic review should I Countries: A Systematic Review and Meta-Analysis. BMJ Open. 2018; 8(3): conduct? A proposed typology and guidance for systematic reviewers in the e018195. medical and health sciences. BMC Med Res Methodol. 2018; 18(1): 5. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 6. Guthold R, Stevens GA, Riley LM, et al.: Worldwide trends in insufficient 22. Iorio A, Spencer FA, Falavigna M, et al.: Use of GRADE for assessment of physical activity from 2001 to 2016: a pooled analysis of 358 population-based evidence about prognosis: rating confidence in estimates of event rates in surveys with 1·9 million participants. Lancet Glob Health. 2018; 6(10): e1077–86. broad categories of patients. BMJ. 2015; 350: h870. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text 7. Roehr S, Pabst A, Luck T, et al.: Is dementia incidence declining in high-income 23. Bossuyt PM, Irwig L, Craig J, et al.: Comparative accuracy: assessing new tests countries? A systematic review and meta-analysis. Clin Epidemiol. 2018; 10: against existing diagnostic pathways. BMJ. 2006; 332(7549): 1089–92. Erratum 1233–1247. in:BMJ.2006 Jun 10;332(7554):1368. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 8. Westby MJ, Dumville JC, Stubbs N, et al.: Protease activity as a prognostic factor for wound healing in venous leg ulcers. Cochrane Database Syst Rev. 24. Bossuyt PM, Leeflang MM: Chapter 6: Developing Criteria for Including Studies. 2018; 9(9): CD012841. In: Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version PubMed Abstract | Publisher Full Text | Free Full Text 0.4 [updated September 2008]. The Cochrane Collaboration. 2008. Reference Source 9. Skoetz N, Trivella M, Kreuzer KA, et al.: Prognostic models for chronic lymphocytic leukaemia: an exemplar systematic review and meta-analysis. 25. Lijmer JG, Mol BW, Heisterkamp S, et al.: Empirical evidence of design-related Cochrane Database of Syst Rev. 2016; 1: CD012022. bias in studies of diagnostic tests. JAMA. 1999; 282(11): 1061–6. Publisher Full Text PubMed Abstract | Publisher Full Text 10. Ambagtsheer RC, Thompson MQ, Archibald MM, et al.: Diagnostic test accuracy 26. Straus S, Moher D: Registering systematic reviews. CMAJ. 2010; 182(1): 13–14. of self-reported frailty screening instruments in identifying community- PubMed Abstract | Publisher Full Text | Free Full Text dwelling older people at risk of frailty and pre-frailty: a systematic review 27. Ge L, Tian JH, Li YN, et al.: Association between prospective registration and protocol. JBI Database System Rev Implement Rep. THE JOANNA BRIGGS overall reporting and methodological quality of systematic reviews: a INSTITUTE, 2017; 15(10): 2464–2468. meta-epidemiological study. J Clin Epidemiol. 2018; 93: 45–55. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text 11. Martínez G, Vernooij RW, Fuentes Padilla P, et al.: 18F PET with florbetaben for 28. Page MJ, Shamseer L, Tricco AC: Registration of systematic reviews in the early diagnosis of Alzheimer’s disease dementia and other dementias in PROSPERO: 30,000 records and counting. Syst Rev. 2018; 7(1): 32. people with mild cognitive impairment (MCI). Cochrane Database Syst Rev. PubMed Abstract | Publisher Full Text | Free Full Text 2017; 11(11): CD012883. 29. Lefebvre C, Glanville J, Briscoe S, et al.: Chapter 4: Searching for and selecting PubMed Abstract | Publisher Full Text | Free Full Text studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, 12. Johnson S, Henschke N, Maayan N, et al.: Ribavirin for treating Crimean Congo Welch VA, (editors). Cochrane Handbook for Systematic Reviews of Interventions haemorrhagic fever. Cochrane Database Syst Rev. 2018; 6(6): CD012713. version 6.0. (updated July 2019). Cochrane, 2019; [Accessed on 29/11/2018]. PubMed Abstract | Publisher Full Text | Free Full Text Publisher Full Text 13. Ellis G, Gardner M, Tsiachristas A, et al.: Comprehensive geriatric assessment 30. Atkinson KM, Koenka AC, Sanchez CE, et al.: Reporting standards for literature for older adults admitted to hospital. Cochrane Database Syst Rev. 2017; 9(1): searches and report inclusion criteria: making research syntheses more CD006211. transparent and easy to replicate. Res Synth Methods. 2015; 6(1): 87–95. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text 14. Aromataris E, Munn Z, (Editors): Joanna Briggs Institute Reviewer’s Manual. The 31. Campbell JM, Kulgar M, Ding S, et al.: Chapter 9: Diagnostic test accuracy Joanna Briggs Institute, 2017. [Accessed on 10/12/2018]. systematic reviews. In: Aromataris E, Munn Z, (Editors). Joanna Briggs Institute 15. Munn Z, Moola S, Lisy K, et al.: Chapter 5: Systematic reviews of prevalence and Reviewer’s Manual. The Joanna Briggs Institute, 2017; [Accessed on 10/12/2018]. incidence. In: Aromataris E, Munn Z, (Editors). Joanna Briggs Institute Reviewer’s Reference Source Manual. The Joanna Briggs Institute. 2017. [Accessed on 10/12/2018]. 32. de Vet HCW, Eisinga A, Riphagen II, et al.: Chapter 7: Searching for Studies. In: Reference Source Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 16. Moola S, Munn Z, Tufanaru C, et al.: Chapter 7: Systematic reviews of etiology 0.4 [updated September 2008]. The Cochrane Collaboration, 2008. and risk. In: Aromataris E, Munn Z, (Editors). Joanna Briggs Institute Reviewer’s Reference Source Page 13 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 33. Lefebvre C, Glanville J, Wieland LS, et al.: Methodological developments in 155(8): 529–36. searching for studies for systematic reviews: past, present and future? Syst PubMed Abstract | Publisher Full Text Rev. 2013; 2: 78. 55. Higgins JPT, Savović J, Page MJ, et al.: Chapter 8: Assessing risk of bias in PubMed Abstract | Publisher Full Text | Free Full Text a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, 34. Glanville JM, Lefebvre C, Miles JN, et al.: How to identify randomized controlled Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of trials in MEDLINE: ten years on. J Med Libr Assoc. 2006; 94(2): 130–136. Interventions. version 6.0 (updated July 2019). Cochrane, 2019. PubMed Abstract | Free Full Text Publisher Full Text 35. Wilczynski NL, Haynes RB; Hedges Team: Developing optimal search strategies 56. Sterne JA, Hernán MA, Reeves BC, et al.: ROBINS-I: a tool for assessing risk for detecting clinically sound prognostic studies in MEDLINE: an analytic of bias in non-randomised studies of interventions. BMJ. 2016; 355: i4919. survey. BMC Med. 2004; 2(1): 23. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 57. Lau J, Ioannidis JP, Schmid CH: Quantitative synthesis in systematic reviews. 36. Beynon R, Leeflang MM, McDonald S, et al.: Search strategies to identify Ann Intern Med. 1997; 127(9): 820–826. diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database PubMed Abstract | Publisher Full Text Syst Rev. 2013; (9): MR000022. 58. Deeks JJ, Higgins JPT, Altman DG: (editors). Chapter 10: Analysing data and PubMed Abstract | Publisher Full Text undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, 37. Sampson M, Tetzlaff J, Urquhart C: Precision of healthcare systematic review Li T, Page MJ, Welch VA, (editors). Cochrane Handbook for Systematic Reviews of searches in a cross-sectional sample. Res Synth Methods. 2011; 2(2): 119–25. Interventions version 6.0 (updated July 2019). Cochrane. 2019. PubMed Abstract | Publisher Full Text Reference Source 38. Bramer WM, Rethlefsen ML, Kleijnen J, et al.: Optimal database combinations 59. Popay J, Roberts H, Sowden A, et al.: Guidance on the conduct of narrative for literature searches in systematic reviews: a prospective exploratory study. synthesis in systematic reviews: A product from the ESRC Methods Syst Rev. 2017; 6(1): 245. Programme. 2006. PubMed Abstract | Publisher Full Text | Free Full Text Publisher Full Text 39. Hartling L, Featherstone R, Nuspl M, et al.: The contribution of databases to the 60. Macaskill P, Gatsonis C, Deeks JJ, et al.: Chapter 10: Analysing and Presenting results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. Results. In: Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook 2016; 16(1): 127. for Systematic Reviews of Diagnostic Test Accuracy Version 1.0. The Cochrane PubMed Abstract | Publisher Full Text | Free Full Text Collaboration. 2010. 40. Glanville JM, Duffy S, McCool R, et al.: Searching ClinicalTrials.gov and the Reference Source International Clinical Trials Registry Platform to inform systematic reviews: 61. Rutter CM, Gatsonis CA: A Hierarchical Regression Approach to Meta-Analysis what are the optimal search approaches? J Med Libr Assoc. 2014; 102(3): of Diagnostic Test Accuracy Evaluations. Stat Med. 2001; 20(19): 2865–84. 177–83. MANUAL GRADE. Spanish version. [Accessed May 2019]. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text | Free Full Text 41. Isojarvi J, Wood H, Lefebvre C, et al.: Challenges of identifying unpublished data 62. Rücker G, Schwarzer G, Carpenter JR, et al.: Undue Reliance on I2 in Assessing from clinical trials: Getting the best out of clinical trials registers and other Heterogeneity May Mislead. BMC Med Res Methodol 2008; 8(1): 79. novel sources. Res Synth Methods. 2018; 9(4): 561–578. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text 63. Chaimani A, Caldwell DM, Higgins T, et al.: Chapter 11: Undertaking network 42. Horsley T, Dingwall O, Sampson M: Checking reference lists to find additional meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, studies for systematic reviews. Cochrane Database Syst Rev. 2011; (8): Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of MR000026. Interventions. version 6.0 (updated July 2019). Cochrane, 2019. PubMed Abstract | Publisher Full Text Publisher Full Text 43. Gentles SJ, Charles C, Nicholas DB, et al.: Reviewing the research methods 64. Schünemann H, Brożek J, Guyatt G, et al.: (editors). Handbook for grading the literature: principles and strategies illustrated by a systematic overview of quality of evidence and the strength of recommendations using the GRADE sampling in qualitative research. Syst Rev. 2016; 5(1): 172. approach. (updated October 2013). [Accessed June 2020]. PubMed Abstract | Publisher Full Text | Free Full Text Reference Source 44. Hartling L, Featherstone R, Nuspl M, et al.: Grey literature in systematic 65. Santesso N, Glenton C, Dahm P, et al.: GRADE Guidelines 26: Informative reviews: a cross-sectional study of the contribution of non-English reports, Statements to Communicate the Findings of Systematic Reviews of unpublished studies and dissertations to the results of meta-analyses in child- Interventions. J Clin Epidemiol. 2020; 119: 126–135. relevant reviews. BMC Med Res Methodol. 2017; 17(1): 64. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text | Free Full Text 66. Schünemann HJ, Higgins JPT, Vist GE, et al.: Chapter 14: Completing ‘Summary 45. Booth A: Searching for qualitative research for inclusion in systematic reviews: of findings’ tables and grading the certainty of the evidence. In: Higgins JPT, a structured methodological review. Syst Rev. 2016; 5: 74. Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane PubMed Abstract | Publisher Full Text | Free Full Text Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). 46. Rethlefsen ML, Murad MH, Livingston EH: Engaging medical librarians to Cochrane. 2019. improve the quality of review articles. JAMA. 2014; 312(10): 999–1000. Reference Source PubMed Abstract | Publisher Full Text 67. Murad MH, Mustafa RA, Schünemann HJ, et al.: Rating the certainty in evidence 47. Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, et al.: Librarian co-authors in the absence of a single estimate of effect. Evid Based Med. 2017; 22(3): correlated with higher quality reported search strategies in general internal 85–87. medicine systematic reviews. J Clin Epidemiol. 2015; 68(6): 617–26. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text 68. Harder T, Takla A, Eckmanns T, et al.: PRECEPT: An Evidence Assessment 48. Spencer AJ, Eldredge JD: Roles for librarians in systematic reviews: a scoping Framework for Infectious Disease Epidemiology, Prevention and Control. Euro review. J Med Libr Assoc. 2018; 106(1): 46–56. Surveill. 2017; 22(40): 16-00620. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 49. Hoy D, Brooks P, Woolf A, et al.: Assessing risk of bias in prevalence studies: 69. Huguet A, Hayden JA, Stinson J, et al.: Judging the Quality of Evidence in modification of an existing tool and evidence of interrater agreement. J Clin Reviews of Prognostic Factor Research: Adapting the GRADE Framework. Epidemiol. 2012; 65(9): 934–9. Syst Rev. 2013; 2: 71. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text | Free Full Text 50. Hayden JA, van der Windt DA, Cartwright JL, et al.: Assessing bias in studies of 70. Schünemann HJ, Mustafa RA, Brozek J, et al.: GRADE guidelines: 21 part 1. prognostic factors. Ann Intern Med. 2013; 158(4): 280–6. Study design, risk of bias, and indirectness in rating the certainty across PubMed Abstract | Publisher Full Text a body of evidence for test accuracy. J Clin Epidemiol. 2020; 122: 129–141. 51. Morgan RL, Thayer KA, Santesso N, et al.: Evaluation of the risk of bias in non- PubMed Abstract | Publisher Full Text randomized studies of interventions (ROBINS-I) and the ‘target experiment’ concept in studies of exposures: Rationale and preliminary instrument 71. Schünemann HJ, Mustafa RA, Brozek J, et al.: GRADE guidelines: 21 part 2. Test development. Environ Int. 2018; 120: 382–387. accuracy: inconsistency, imprecision, publication bias, and other domains PubMed Abstract | Publisher Full Text for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol. 2020; 122: 142–152. 52. Morgan RL, Thayer KA, Santesso N, et al.: A risk of bias instrument for non- PubMed Abstract | Publisher Full Text randomized studies of exposures: A users’ guide to its application in the context of GRADE. Environ Int. 2019; 122: 168–184. 72. Moher D, Liberati A, Tetzlaff J, et al.: Preferred Reporting Items for Systematic PubMed Abstract | Publisher Full Text Reviews and Meta-Analyses: The PRISMA Statement. BMJ. 2009; 339: b2535. PubMed Abstract | Publisher Full Text | Free Full Text 53. Wolff RF, Moons KGM, Riley RD, et al.: PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019; 73. Moher D, Shamseer L, Clarke M, et al.: Preferred Reporting Items for Systematic 170(1): 51–58. Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Syst Rev. PubMed Abstract | Publisher Full Text 2015; 4(1): 1. 54. Whiting PF, Rutjes AW, Westwood ME, et al.: QUADAS-2: a revised tool for the PubMed Abstract | Publisher Full Text | Free Full Text quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011; 74. Beller EM, Glasziou PP, Altman DG, et al.: PRISMA for Abstracts: Reporting Page 14 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Systematic Reviews in Journal and Conference Abstracts. PLoS Med. 2013; in Health Care: Standards for Systematic Reviews. Washington (DC): National 10(4): e1001419. Academies Press (US); 2011.2, Standards for Initiating a Systematic Review. 2011. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text 75. Zorzela L, Loke YK, Ioannidis JP, et al.: PRISMA Harms Checklist: Improving 85. Marshall IJ, Wallace BC: Toward systematic review automation: a practical Harms Reporting in Systematic Reviews. BMJ. 2016; 352: i157. guide to using machine learning tools in research synthesis. Syst Rev. 2019; PubMed Abstract | Publisher Full Text 8(1): 163. 76. Campbell M, McKenzie JE, Sowden A, et al.: Synthesis without meta-analysis PubMed Abstract | Publisher Full Text | Free Full Text (SWiM) in systematic reviews: reporting guideline. BMJ. 2020; 368: l6890. 86. Carrasco-Labra A, Brignardello-Petersen R, Santesso N, et al.: Improving GRADE PubMed Abstract | Publisher Full Text | Free Full Text evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format. J Clin Epidemiol. 77. McInnes MDF, Moher D, Thombs BD, et al.: Preferred Reporting Items for a 2016; 74: 7–18. Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: PubMed Abstract | Publisher Full Text The PRISMA-DTA Statement. JAMA. 2018; 319(4): 388–396. PubMed Abstract | Publisher Full Text 87. Marquez C, Johnson AM, Jassemi S, et al.: Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and 78. Moher D, Tetzlaff J, Tricco AC, et al.: Epidemiology and Reporting policy-makers? A mixed-methods study. Implement Sci. 2018; 13(1): 84. Characteristics of Systematic Reviews. PLoS Med. 2007; 4(3): e78. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 88. Whiting P, Savović J, Higgins JP, et al.: ROBIS: A new tool to assess risk of bias 79. Page MJ, Shamseer L, Altman DG, et al.: Epidemiology and Reporting in systematic reviews was developed. J Clin Epidemiol. 2016; 69: 225–234. Characteristics of Systematic Reviews of Biomedical Research: A Cross- PubMed Abstract | Publisher Full Text | Free Full Text Sectional Study. PLoS Med. 2016; 13(5): e1002028. PubMed Abstract | Publisher Full Text | Free Full Text 89. Shea BJ, Reeves BC, Wells G, et al.: AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of 80. Salameh JP, McInnes MDF, Moher D, et al.: Completeness of Reporting of healthcare interventions, or both. BMJ. 2017; 358: j4008. Systematic Reviews of Diagnostic Test Accuracy Based on the PRISMA-DTA PubMed Abstract | Publisher Full Text | Free Full Text Reporting Guideline. Clin Chem. 2019; 65(2): 291–301. PubMed Abstract | Publisher Full Text 90. Pollock A, Berge E: How to do a systematic review. Int J Stroke. 2018; 13(2): 138–56. 81. Turner T, Green S, Tovey D, et al.: Producing Cochrane systematic reviews-a PubMed Abstract | Publisher Full Text qualitative study of current approaches and opportunities for innovation and improvement. Syst Rev. 2017; 6(1): 147. 91. Muka T, Glisic M, Milic J, et al.: A 24-step guide on how to design, conduct, PubMed Abstract | Publisher Full Text | Free Full Text and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol. 2020; 35(1): 49–60. 82. Borah R, Brown AW, Capers PL, et al.: Analysis of the time and workers needed PubMed Abstract | Publisher Full Text to conduct systematic reviews of medical interventions using data from the 92. Harrison JK, Reid J, Quinn TJ, et al.: Using Quality Assessment Tools to PROSPERO registry. BMJ Open. 2017; 7(2): e012545. Critically Appraise Ageing Research: A Guide for Clinicians. Age Ageing. 2017; PubMed Abstract | Publisher Full Text | Free Full Text 46(3): 359–65. 83. Ioannidis JP, Greenland S, Hlatky MA, et al.: Increasing Value and Reducing PubMed Abstract | Publisher Full Text | Free Full Text Waste in Research Design, Conduct, and Analysis. Lancet. 2014; 383(9912): 93. Zeng X, Zhang Y, Kwong JS, et al.: The Methodological Quality Assessment 166–75. Tools for Preclinical and Clinical Studies, Systematic Review and Meta- PubMed Abstract | Publisher Full Text | Free Full Text Analysis, and Clinical Practice Guideline: A Systematic Review. J Evid Based 84. Institute of Medicine (US) Committee on Standards for Systematic Reviews of Med. 2015; 8(1): 2–10. Comparative Effectiveness Research, Eden J, Levit L, et al.: Finding What Works PubMed Abstract | Publisher Full Text Page 15 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Open Peer Review Current Peer Review Status: Version 2 Reviewer Report 07 October 2020 https://doi.org/10.5256/f1000research.28216.r69215 © 2020 Purssell E. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Edward Purssell School of Health Sciences, City, University of London, London, UK I would just push point 5 regarding heterogeneity. Cochrane is quite clear about this - stating "Thus, the test for heterogeneity is irrelevant to the choice of analysis; heterogeneity will always exist whether or not we happen to be able to detect it using a statistical test"1. I think that this just needs a bit more explanation. References 1. Higgins J, Thomas J, et al.: Cochrane Handbook for Systematic Reviews of Interventions (Version 6.1). 2020. Reference Source Competing Interests: No competing interests were disclosed. Reviewer Expertise: Paediatrics, infection control, systematic review. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Reviewer Report 01 September 2020 https://doi.org/10.5256/f1000research.28216.r69216 © 2020 Cumpston M. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Miranda Cumpston 1 Monash University, Melbourne, Australia 2 University of Newcastle, Newcastle, Australia Page 16 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Thanks to the authors for their work in revising this paper and responding effectively to the comments raised in peer review. I have no further comments to raise. Thanks also for including me in the Acknowledgements - please note that my correct title is 'Ms'. Competing Interests: I am an Associate Editor of the Cochrane Handbook for Systematic Reviews, and an Editor with Cochrane Public Health. Reviewer Expertise: In presenting my comments, I acknowledge that my own experience focuses on methods around systematic reviews of the effects of interventions, and so I cannot comment with expertise on the details of methods described for other review types. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Version 1 Reviewer Report 28 February 2020 https://doi.org/10.5256/f1000research.24297.r59540 © 2020 Purssell E. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Edward Purssell School of Health Sciences, City, University of London, London, UK I have a few suggestions which may be helpful to the authors: 1. When discussing research methods, it may be worth just mentioning the efficacy/effectiveness issue. 2. I think that this "To obtain an efficient search with adequate sensitivity, performing searches in MEDLINE and EMBASE is sufficient, as they are the two most frequently used bibliographic databases39, and they are enough to identify most relevant studies for a specific SR40" might be a bit of a sweeping statement - are the two databases always sufficient? I think it would be helpful to mention some others as well - unless the authors really do believe that these are enough. 3. "involving a medical librarian can be convenient to improve the search quality47–49". I am not sure that convenient is quite the right word. 4. Be clear to differentiate risk of bias at the study level from RoB at the review level. For example RoB2 is study-level, ROBIS is review level. Page 17 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 5. "For instance, in the SR by Ellis et al. (2017), the authors established a 70% heterogeneity limit for I2, beyond which a meta-analysis combining the results would not be performed13" - why 70%? This does not sound like a sensible decision making process anyway. Either the authors thought it was worth doing a MA in which case the heterogeneity form parts of the results, or it is not worth doing in which case this would not be a consideration. Also not doing a MA may have the effect of hiding this heterogeneity. There are also ways of investigating heterogeneity that can be illuminating. 6. Mention ROBIS and AMSAR2 as review-level tools. Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Partly If applicable, is the statistical analysis and its interpretation appropriate? Not applicable Are all the source data underlying the results available to ensure full reproducibility? No Are the conclusions drawn adequately supported by the results? Yes Competing Interests: No competing interests were disclosed. Reviewer Expertise: Paediatrics, infection control, systematic review. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Author Response 19 Jul 2020 Marta Roqué, Iberoamerican Cochrane Centre - Sant Pau Biomedical Research Institute (IIB-Sant Pau), Barcelona, Spain ○ When discussing research methods, it may be worth just mentioning the efficacy/effectiveness issue. ANSWER: We agree that this is an important issue, which merits a full discussion that unfortunately falls outside the scope of this project. The selection of resources done is not dependent on whether the reviewer explores efficacy or effectiveness, and will be useful to the researchers regardless of their intended purpose. However, we have clarified this issue Page 18 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 throughout the text by referring to 'effects of interventions' (rather than efficacy of interventions) and we have explicitly commented on it in the discussion.We have substituted 'efficacy of interventions' by 'effects of interventions'. We have added the following limitation "The selection of resources done is not dependent on whether the reviewer explores questions on efficacy or effectiveness, which are often described as explanatory or pragmatic questions, and will be useful to the researchers regardless of their intended purpose. However, we have not considered the resources to conduct in-depth exploration of effectiveness issues such as reviews of complex interventions or implementation reviews. " ○ I think that this "To obtain an efficient search with adequate sensitivity, performing searches in MEDLINE and EMBASE is sufficient, as they are the two most frequently used bibliographic databases39, and they are enough to identify most relevant studies for a specific SR40" might be a bit of a sweeping statement - are the two databases always sufficient? I think it would be helpful to mention some others as well - unless the authors really do believe that these are enough. ANSWER: Certainly, while MEDLINE and EMBASE are most used databases, there are other databases which can provide complementary information. However, the bibliographic references provided in the text supports the notion that these two databases are enough for an efficient search regardless of topic, and the role of these other databases is mostly complementary, but with little added benefit. To avoid the implicit suggestion that specialized databases cannot generate any added value, a sentence has been added mentioning them. Two sentences have been added "To obtain an efficient search with adequate sensitivity, performing searches in MEDLINE and EMBASE is may be sufficient especially in intervention reviews". " These searches can be complemented with additional searches in other databases such as PEDro (Hyperlink: https://www.pedro.org.au/), which may provide specific information for certain topics. " ○ "involving a medical librarian can be convenient to improve the search quality47–49". I am not sure that convenient is quite the right word. ANSWER: Thanks, we've substituted the term 'convenient' for 'desirable' ○ Be clear to differentiate risk of bias at the study level from RoB at the review level. For example RoB2 is study-level, ROBIS is review level. ANSWER: Thanks for pointing out this issue, we have clarified in the risk of bias section that this project focuses in presenting study-level resources, and that tools for reviews are not included in the manuscript. As we expand later in comment 6, we have been more explicit about the exclusion of overviews in the eligibility criteria section, and we have added a mention to this fact in the discussion section, as a limitation of the review, with an explicit mention of these tools. A sentence has been added "Assessing the risk of bias of the included studies is a key element in any SR". We have modified the eligibility criteria ("We excluded the methodological resources to develop other types of SRs (methodological, economic evaluation and qualitative research SRs, or overviews)" and the discussion ("Reviews of reviews (or overviews) were also not considered, and as such, there are a number of review-level resources which have not been discussed, for example the risk of bias assessment tool ROBIS or the methodological tool AMSTAR2") ○ "For instance, in the SR by Ellis et al. (2017), the authors established a 70% heterogeneity limit for I2, beyond which a meta-analysis combining the results would not be performed13" - why 70%? This does not sound like a sensible decision making process anyway. Either the authors thought it was worth doing a MA in which case Page 19 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 the heterogeneity form parts of the results, or it is not worth doing in which case this would not be a consideration. Also not doing a MA may have the effect of hiding this heterogeneity. There are also ways of investigating heterogeneity that can be illuminating. ANSWER: We agree with the peer reviewer that the analysis choices of the examples shown may not always be universally shared. We chose the examples based on several considerations, as they had to be useful for the different parts of the manuscript, but they are not necessarily a perfect role model in all their decisions. Readers will need to take into account that these examples are only illustrative, as we are presenting these choices descontextualized, without a full vision of the authors decision making process. ○ Mention ROBIS and AMSTAR2 as review-level tools. ANSWER: Thanks for this suggestion. We restricted our focus to a limited list of SR types, not including overviews (or reviews of reviews). For this reason, only study-level resources for assessing risk of bias have been described, and review-level resources (such as ROBIS or AMSTAR2) are not mentioned. We realize that this issue may not be clear enough, and consequently we have been more explicit about the exclusion of overviews in the eligibility criteria section, and we have added a mention to this fact in the discussion section, as a limitation of the review, with an explicit mention of these tools. We have modified the eligibility criteria ("We excluded the methodological resources to develop other types of SRs (methodological, economic evaluation and qualitative research SRs, or overviews)" and the discussion ("Reviews of reviews (or overviews) were also not considered, and as such, there are a number of review-level resources which have not been discussed, for example the risk of bias assessment tool ROBIS or the methodological tool AMSTAR2") Competing Interests: No competing interests were disclosed. Reviewer Report 14 February 2020 https://doi.org/10.5256/f1000research.24297.r59538 © 2020 Cumpston M. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Miranda Cumpston 1 Monash University, Melbourne, Australia 2 University of Newcastle, Newcastle, Australia The authors have summarised the current literature around methods for conducting systematic reviews, encompassing reviews of prevalence, prognosis, diagnosis and intervention effectiveness. In a field of methodology where some guidance is collated in well-known handbooks, but other areas are scattered among the methods literature, and where development of new methods is relatively fast-moving, this map of available guidance and methods studies is likely to be very useful to researchers new to systematic reviews or branching out into different review types. The authors are experienced experts in this field, and the review is well-written and Page 20 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 clear. In my understanding, this paper aims to fulfil two roles: (1) a toolkit that enables authors of systematic reviews to navigate out to key methods resources, and (2) a brief summary of methods guidance for each review type. It may be helpful throughout the paper, including in the abstract and conclusions, to separate out these two roles more clearly. I have some suggestions to improve the transparency and usability of the paper from each of these perspectives. Major recommendations As a review and toolkit of methodological resources, I would make the following suggestions: ○ Methods: The authors could provide more transparent detail on the selection process for included documents, including specifying which organisational websites were searched, how many resources were identified, and if any were excluded. Although this is intended as a mapping, rather than systematic review, some additional detail would help readers to understand how the guidance documents were selected, and why perhaps some resources they may have used before are not listed. ○ Importantly, more detail could presented on how the authors selected the most “relevant” or “best” resources, especially where there may have been multiple candidate documents. For example, were they most recent, most comprehensive, those endorsed by a credible organisation, most introductory, most rigorous methods? The role of the authors’ expert judgement in these decisions should be stated explicitly. Note that I am not suggesting that a different process should have been used, just suggesting a more detailed description of the judgement process. ○ Results: As a navigation guide, I found that citing the resources within the text did not tell me everything I wanted to know. Also, the key resources are mixed in the reference list with exemplar reviews and other citations. A table outlining the recommended resources by category with hyperlinks to each resource would be helpful. ○ As a reader, I would find it helpful for the authors to draw the distinction between the different types of resources cited. For example, some were synthesised, ‘best practice’ guidance intended for use by authors (such as the Cochrane or JBI Handbooks). Some were reporting guidelines (which often list good practice but are not intended to provide detailed guidance on the conduct of reviews). Some were primary methods studies (e.g. measuring the prevalence or impact of a method, which may describe or evaluate possible methods options but are not intended as guidance on the ‘best’ available methods for authors. All may be useful in different ways. As a brief summary of methods guidance, I would make the following suggestions: ○ Methods: It would be helpful to include a brief description of your role in selecting and writing this summary of guidance, for example, stating briefly that key areas of methods were summarised, and how the methods to be highlighted were selected, especially where multiple and potentially conflicting sources were available (e.g. major methods applicable to all reviews, key differences between review types, expert opinion about interesting or important advice). ○ It might be helpful to note that the summary of guidance presented is just that, a brief summary, and that authors who are new to systematic reviews should consult the more Page 21 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 detailed documents for complete guidance. ○Results: It might be particularly helpful to note where there is disagreement in the literature in relation to particular areas of guidance. For example, comments that searching the grey literature is not useful for effectiveness reviews, or that risk of bias tools should be adapted for each review, which may be in disagreement with some of the major guidance handbooks and reporting guidelines. I’m not arguing about these specific items, just noting that some methods choices have been made by the authors, and it may be helpful to make this transparent. Minor recommendations ○Title and abstract: It would be helpful to briefly define the scope of this review, especially in the context of a journal such as F1000Research, which publishes in a wide range of scientific fields. For example, using a term such as ‘systematic reviews in health care’, and noting that the review looks at prevalence, prognosis, diagnosis and health interventions (and not other areas relevant to health such as environmental exposure). ○ Competing interests: The authors could note any roles with Cochrane and the GRADE Working Group. These are not necessarily financial conflicts, but may be relevant to the authors’ decision to recommend guidance from these sources (which of course I agree with!). ○ Results: In the first two paragraphs, you discuss some specific points in relation to resources available for diagnostic and prognosis reviews. As all review types appeared to cite both guidance handbooks and additional primary methods studies, I wasn’t clear on the point you were trying to make in this section. ○ There were some key guidance documents that I would have expected to see, although there may be good reasons not to include them, such as: ○ General guidance on systematic reviews from the US Institute of Medicine and the Centre for Reviews and Dissemination at the University of York (both are older documents, so this may be the reason). ○ Tools to assess the risk of bias in reviews, such as ROBIS or AMSTAR, which may enable reflection by authors on their choice of methods, in a similar manner to reporting guidelines. ○ Resources on the development of protocols (as distinct from study registration), such as Chapter 1 of the Cochrane Handbook. ○ Guidance on synthesis in the absence of meta-analysis, such as Chapter 12 in the Cochrane Handbook and papers by (Campbell, McKenzie et al. (2020)1) and (Popay, Roberts et al. (2006)2). This may be relevant to synthesis, assessment of heterogeneity and GRADE. ○ Guidance on network meta-analysis, such as Chapter 11 of the Cochrane Handbook and multiple journal articles, as well GRADE guidance and CINeMA. ○ Two papers from the GRADE Working Group (published this week!) on the use of GRADE for diagnostic test accuracy studies published this week. ○ In the text on risk of bias and Table 2, is it worth noting that the number of items in RoB 2 varies depending on the effect of interest and the included study designs? Page 22 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 ○ In Table 3, as there are more variations on the measures to combine (e.g. Ratio of Means), and statistical methods for meta-analysis of efficacy of interventions (including the inverse- variance method, which is mentioned for other types), could it be helpful to note in the table somewhere that these are common characteristics, not an exhaustive list? ○ When discussing fixed effect and random effects models, it may be helpful to note that they differ in relation to assumptions and heterogeneity, as you mention something briefly about analysis using random-effects models later in the paper that relies on understanding this. ○ On page 9, under “Efficacy of intervention reviews”, I think hazard ratios are listed in error against binary outcomes as well as time-to-event outcomes. Perhaps you meant risk ratios? ○ In the section on assessing certainty of the evidence, it would be helpful to describe the GRADE approach in the first paragraph (it is currently named without description in the third paragraph). ○ Discussion: I would also acknowledge under limitations of this review that the selection of resources and summary of guidance were informed by expert opinion, and that others may have selected different resources or made different recommendations. ○ References: 1: I think there may be an error – should this be a reference to the 2019 edition of the Cochrane Handbook, rather than an older paper by Higgins in Cochrane Methods on RoB 2? I assume this based on its use in the Introduction as a general reference about review methods, and also on the availability of general Handbooks in para 2 of the Results. ○ 20: could be based on the references to individual chapters, i.e. Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. ○ 63: It’s great to see the Spanish Manual GRADE cited. If I am correct, this is a translation of the 2013 GRADE Handbook (apologies if I am wrong). Would it be helpful to cite some of the more recent GRADE papers as well, as they reflect the most up to date guidance? I note you also cite the paper on using standard language to express GRADE, as well as the recent Cochrane Handbook Chapter, so perhaps this is sufficient. References 1. Campbell M, McKenzie J, Sowden A, Katikireddi S, et al.: Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020. Publisher Full Text 2. Popay J, Roberts H, et al.: Guidance on the conduct of narrative synthesis in systematic reviews. ESRC methods programme. 2006. 47-71 Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Page 23 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 Yes Are sufficient details of methods and analysis provided to allow replication by others? No If applicable, is the statistical analysis and its interpretation appropriate? Not applicable Are all the source data underlying the results available to ensure full reproducibility? No Are the conclusions drawn adequately supported by the results? Yes Competing Interests: I am an Associate Editor of the Cochrane Handbook for Systematic Reviews, and an Editor with Cochrane Public Health. Reviewer Expertise: In presenting my comments, I acknowledge that my own experience focuses on methods around systematic reviews of the effects of interventions, and so I cannot comment with expertise on the details of methods described for other review types. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Author Response 19 Jul 2020 Marta Roqué, Iberoamerican Cochrane Centre - Sant Pau Biomedical Research Institute (IIB-Sant Pau), Barcelona, Spain ○ Methods: The authors could provide more transparent detail on the selection process for included documents, including specifying which organisational websites were searched, how many resources were identified, and if any were excluded. ANSWER: Thanks for this suggestion. The text lists the main organisational websited we checked to identify guidelines ("Cochrane, Joanna Briggs Institute, European Network for Health Technology Assessment (EUNETHTA), Enhancing the Quality and Transparency of Health Research (EQUATOR) network, Grading of Recommendations Assessment, Development and Evaluation (GRADE))", as suggested by the peer reviewer. Also, the text is now more explicit about the process followed to select resources. See answers to comments below regarding this specific issue.More detailed description of the resource selection process has been added to the text (see detailed modifications in the next comments) ○ Importantly, more detail could presented on how the authors selected the most “relevant” or “best” resources, especially where there may have been multiple candidate documents. ANSWER: Thanks for this suggestion. A paragraph was added to the eligibility criteria section "The resources were selected based on the authors expert judgement, prioritising those resources which were endorsed or part of a guideline from the organisations cited Page 24 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 above, and those which were more recent. " ○ Results: As a navigation guide, I found that citing the resources within the text did not tell me everything I wanted to know. ANSWER: We have added a table in the Appendix where the references and resources considered in the review are classified into guidelines to conduct SR (or chapters of those guidelines), reporting guidelines, primary methods papers and references to examples used in the manuscript. ○ Methods: It would be helpful to include a brief description of your role in selecting and writing this summary of guidance, ANSWER: Thanks, we will be more explicit on this issue. A sentence has been added to the data selection and extraction section "For each pre-defined section, the authors selected and summarized the methods that were considered to be more rigorous and widely accepted, prioritizing major methods applicable to all reviews over more controversial methods, or methods which required highly specialized knowledge. The text organises the results pedagogically with the aim to highlight key differences between review types, present the key characteristics of each method, and be a comprehensive tool that contains the most relevant advice based on the authors judgement." ○ It might be helpful to note that the summary of guidance presented is just that, a brief summary, and that authors who are new to systematic reviews should consult the more detailed documents for complete guidance. ANSWER: Thanks for the suggestion, we have clarified this point in the text. A sentence has been added to the key results section of the discussion, stating "This project does not aim to be a standalone tool for a researcher to find complete guidance on how to conduct and report a review, but rather it aims to be a signpost pointing out to the resources where researchers may find in depth guidance to develop their reviews." ○ Results: It might be particularly helpful to note where there is disagreement in the literature in relation to particular areas of guidance. ANSWER: As far as possible, we have avoided presenting controversial advice, although we are aware that any topic can be approached differently by different researchers or even institutions. It could be quite daunting to be comprehensive in identifying all the controversies in the different steps of conducting a review, as the volume of publications on methods for reviews is extremely large and diverse. Additionally, we think that the discussion or even the identification of issues where controversy exists may fall outside of the project scope, as it might reduce the usefulness to a new researcher which needs to find clear guidance on a topic, even if there are other alternative methods available. We have stressed the subjectiveness and risk of implicit selection biases in the discussion "An inherent limitation of this project is its methodology based on a selection of resources and summary of guidance informed by expert opinion, which may be susceptible to implicit selection biases or lack of comprehensiveness. " ○ Title and abstract: It would be helpful to briefly define the scope of this review, especially in the context of a journal such as F1000Research, which publishes in a wide range of scientific fields. ANSWER: Thanks, this is a very useful suggestion.Title has been modified to "Toolkit of methodological resources to conduct systematic reviews in health care: reviews on prevalence, prognosis, diagnosis and interventions". The concept has also been introduced in the abstract ("This work identifies and describes the most relevant methodological resources to conduct high-quality reviews that answer health care questions regarding Page 25 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 prevalence, prognosis, diagnostic accuracy and effects of interventions") ○ Competing interests: The authors could note any roles with Cochrane and the GRADE Working Group. ANSWER: A sentence was added to the Data selection and extraction "The authors are members of CIBERESP (Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública - Biomedical Research Center Network of Epidemiology and Public Health), hold active roles within Cochrane and the GRADE Working Group," ○ Results: In the first two paragraphs, you discuss some specific points in relation to resources available for diagnostic and prognosis reviews. As all review types appeared to cite both guidance handbooks and additional primary methods studies, I wasn’t clear on the point you were trying to make in this section. ANSWER: We realize we didn't make our point clear, that was to illustrate what guideline handbooks had been identified by type of review. We've rewritten both paragraphs aiming to be more clear. "We identified guidance handbooks, primary studies and reporting guidelines as a result of the bibliographic searches. The resources selected are presented in the Appendix. We have identified methodological guidelines dedicated to the development of prevalence SRs14, global prognosis15, and prognostic factor SRs 16, 17, 18 . During the performed search, we identified methodological manuals to develop prognostic model SRs in the series of publications from the PROGRESS project19, and in the resource compilation from Cochrane's Prognosis Methods Group ." ○ There were some key guidance documents that I would have expected to see, ANSWER: .Thanks for the bringing forth this issue, which was overlooked in the manuscript. The issue of narrative synthesis of results has been cited in several sections the paper (statistical synthesis, quality of evidence, results report). An explicit mention to narrative synthesis has been added to the Statistical synthesis section "In those cases when a quantitative synthesis is precluded, the SR will be restricted to a narrative synthesis. A narrative synthesis should not simply summarize the findings from the included studies in order to draw conclusions about the body of evidence, but instead should be a more formal process which includes a formulation of the theory of how the intervention works, why and for whom, the exploration of the relationships in the data, and the assessment of the robustness of the synthesis. 59". Certainty of evidence from narrative synthesis has been mentioned "Certainty of evidence can be assessed too when no quantitative synthesis is possible. 65". Also, the SWiM paper has been cited in the REsults report section"Additionally, the SWiM guideline is available for reporting intervention SRs where the effects of interventions are synthethised narratively without metanalysis, focusing on the key features of narrative information synthesis (grouping of studies, presentation of data and summary text, and appropriate discussion of limitations of this type of synthesis)." Network metanalysis have been mentioned in the Statistical methods and Quality of evidence sections. A cite to Chapter 11 in the handbook has been added to both sections Thanks, we've added the GRADE diagnostic references to the paper. These references were incorporated to the corresponding Quality of evidence section, modifying the paragraphs on diagnostic SR. ○ In the text on risk of bias and Table 2, is it worth noting that the number of items in RoB 2 varies depending on the effect of interest and the included study designs? Page 26 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 ANSWER: Thanks, we've added this information. A sentence has been added to the risk of bias section ("The number of questions may vary, depending on the effect of interest and the design of the study assessed"). A footnote has been added to Table 2 "The number of items in the risk of bias tools may vary depending on the effect of interest and the included study designs, as well as the addition or suppresion of index questions by the researchers to tailor the tool to the SR ." ○ In Table 3, as there are more variations on the measures to combine (e.g. Ratio of Means), and statistical methods for meta-analysis of efficacy of interventions (including the inverse-variance method, which is mentioned for other types), could it be helpful to note in the table somewhere that these are common characteristics, not an exhaustive list? ANSWER: Thanks, we've corrected the oversight of not mentioning the inverse variance method for continuous outcomes in the efficacy of interventions SR, and have clarified in the text that the table is by no means exhaustive. The text introducing table 3 now reads "Table 3 shows a non-exhaustive compilation of the main characteristics of the meta- analysis methods and the main software commands for each type of SR". The inverse- variance method has been added to the efficacy of interventions row ○ When discussing fixed effect and random effects models, it may be helpful to note that they differ in relation to assumptions and heterogeneity, as you mention something briefly about analysis using random-effects models later in the paper that relies on understanding this. ANSWER: This is certainly an important point. Although there is already an explicit link between choice of model and heterogeneity in the previous paragraph ("the evaluation of the existing clinical and statistical heterogeneity in the set of studies, which will inform us /.../ 2) what meta-analysis model we should apply"), we agree to further stress this point as suggested. We've modified the existing sentence on random and fixed-effects models "Both models are based on different assumptions regarding distribution of effects and heterogeneity across studies, and they differ in their application and interpretation" ○ On page 9, under “Efficacy of intervention reviews”, I think hazard ratios are listed in error against binary outcomes as well as time-to-event outcomes. Perhaps you meant risk ratios? ANSWER: You're right, it should have read 'risk ratios' instead of 'hazard ratios'. hazard ratio' substituted by 'risk ratio' ○ In the section on assessing certainty of the evidence, it would be helpful to describe the GRADE approach in the first paragraph (it is currently named without description in the third paragraph). ANSWER: Thanks, we agree. We've added an explicit reference to the GRADE system "Certainty of evidence is best evaluated with the GRADE system . Certainty in the obtained estimates for each one of the key SR outcomes or factors is classified as high, moderate, low or very low" ○ Discussion: I would also acknowledge under limitations of this review that the selection of resources and summary of guidance were informed by expert opinion, and that others may have selected different resources or made different recommendations. ANSWER: While we have strived to draw a unbiased selection of the best resources, it is right to point out this potential limitation. We've added the sentence "An inherent limitation of this project is its methodology based on a selection of resources and summary of Page 27 of 28 F1000Research 2020, 9:82 Last updated: 07 OCT 2020 guidance informed by expert opinion, which may be susceptible to implicit biases or lack of comprehensiveness" ○ Reference 1: I think there may be an error – should this be a reference to the 2019 edition of the Cochrane Handbook, rather than an older paper by Higgins in Cochrane Methods on RoB 2? ANSWER: Yes, you are right. References 1 and 56 were interchanged, and have now been corrected. New reference 1 and new reference 56 ○ Reference 20: could be based on the references to individual chapters, i.e. Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. ANSWER: Thank you, the reference has been modified as suggested ○ Reference 63: It’s great to see the Spanish Manual GRADE cited. If I am correct, this is a translation of the 2013 GRADE Handbook (apologies if I am wrong). Would it be helpful to cite some of the more recent GRADE papers as well, as they reflect the most up to date guidance? I note you also cite the paper on using standard language to express GRADE, as well as the recent Cochrane Handbook Chapter, so perhaps this is sufficient. ANSWER: We wish to keep the original 2013 reference, as the most comprehensive manual on GRADE. However, we will substitute the reference of the Spanish version for the English version. Reference 63 substituted for the reference to the original version Competing Interests: No competing interests were disclosed. The benefits of publishing with F1000Research: • Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com Page 28 of 28