Journal of Clinical Epidemiology 135 (2021) 42e53 GRADE RELATED PAPERS Challenges in applying the GRADE approach in public health guidelines and systematic reviews: a concept article from the GRADE Public Health Group Michele Hilton Boona,*, Hilary Thomsona, Beth Shawb, Elie A. Aklc,d, Stefan K. Lhachimie,f, Jes opez-Alcaldeg, Miloslav Klugarh, Leslie Choii, Zuleika Saz-Parkinsonj, us L Reem A. Mustafac,k, Miranda W. Langendaml, Olivia Cranem, Rebecca L. Morganc, Eva Rehfuessn, Bradley C. Johnstono, Lee Yee Chongp, Gordon H. Guyattc, unemannq, Srinivasa Vittal Katikireddia, For the GRADE Working Group Holger J. Sch€ a MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glasgow, Glasgow G3 7HR, UK b Center for Evidence-based Policy, Oregon Health & Science University, Portland, OR 97201 USA c Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street W, Hamilton, Ontario L8S 4K1, Canada d Department of Internal Medicine, American University of Beirut, Beirut, Lebanon e Department for Health Services Research, Institute of Public Health and Nursing Research, University of Bremen, Grazer Straße 4, 28359 Bremen, Germany f Health Sciences Bremen, University of Bremen, 28359 Bremen, Germany g Department of Paediatrics, Obstetrics & Gynaecology and Preventative Medicine, Universitat Autonoma de Barcelona; Faculty of Health Sciences, Universidad Francisco de Vitoria (UFV)-Madrid; Clinical Biostatistics Unit, Hospital Universitario Ramon y Cajal (IRYCIS); CIBER Epidemiology and Public Health; Cochrane Associate Centre of Madrid, Madrid, Spain h Faculty of Medicine, Czech National Centre for Evidence-Based Healthcare and Knowledge Translation (Cochrane Czech Republic, The Czech Republic Centre for Evidence-Based Healthcare; JBI Centre of Excellence, Masaryk University GRADE Centre), Institute of Biostatistics and Analyses, Masaryk University, 625 00 Brno, Czechia i The Department of Vector Biology, Partnership for Increasing the Impact of Vector Control, Liverpool School of Tropical Medicine, Liverpool, UK j European Commission, Joint Research Centre, Ispra, Italy k Departments of Medicine and Biomedical & Health Informatics, University of Missouri-Kansas City, Kansas City, MO 66160 USA l Department of Clinical Epidemiology, Amsterdam University Medical Centres, University of Amsterdam, Biostatistics and Bioinformatics, Amsterdam Public Health Research Institute, Amsterdam, the Netherlands m National Institute for Health and Care Excellence (NICE), Level 1A, City Tower, Piccadilly Plaza, Manchester M1 4BT, UK n Institute for Medical Informatics, Biometry and Epidemiology, Pettenkofer School of Public Health, LMU Munich, Munich, Germany o Department of Nutrition, Texas A&M University, College Station, TX, USA p Cochrane Public Health and Health Systems Network, University of Oxford, Oxford, UK Funding: This work was supported by the UK Medical Research Council (grant numbers MC_UU_12,017/13 and MC_UU_12,017/15), Scottish Gov- ernment Chief Scientist Office (grant numbers SPHSU13 and SPHSU15), and NHS Research Scotland (grant number SCAF/15/02). Leslie Choi is supported by PIIVeC, the Partnership for Increasing the Impact of Vector Control and READ-It, the Research, Evidence and Development Initiative. This work was also partly supported through a grant from the Global Malaria Program, WHO (grant number 709319). PIIVeC is funded by the UK Medical Research Council (grant number MR/P027873/1) through the Global Challenges Research Fund. READ-IT and the CIDG editorial base are funded by the UK aid from the UK Government for the benefit of developing countries (Grant: 300342-104). The views expressed in this article do not necessarily reflect the views of the fund- ing bodies. Declarations of interest: No interests to declare: MHB, EAA, LC, LYC, OC, GHG, SVK, SKL, JLA, RLM, RM, ZSP, ER, ZSP, BS, and HT. HJS is co-chair of the GRADE Working Group but has no direct financial conflict of interest. BCJ: As part of his recruitment to Texas A&M University, BCJ re- ceives funds from Texas A&M AgriLife Research to support investigator-initiated research related to saturated and polyunsaturated fats. Support from Texas A&M AgriLife institutional funds are from interest and investment earnings, not a sponsoring organization, industry, or company. MK declares the following affiliation with the following organizations that could be perceived as a direct or indirect conflict of interest: Director of Cochrane Czech Republic, Director of The Czech Republic Center for Evidence-Based Healthcare; JBI Center of Excellence, Director of Masaryk University GRADE Center. [For reference, from the GRADE project groups ToR: ‘‘GRADE conceptual articles discuss concepts that may not have been piloted on examples and that may not neces- sarily result in GRADE guidance. All GRADE conceptual articles should include text explicitly stating that they are not GRADE guidance in the title, ab- stract, and in the body of the article. It should also explain why the article is a conceptual article and not a GRADE guidance article, and, when possible, to describe if it is in the process to be developed as GRADE guidance.’’] * Correspondence: Michele Hilton Boon, MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glas- gow, G3 7HR, UK. Tel: þ44 141 353 7500. E-mail address: michele.hiltonboon@glasgow.ac.uk (M. Hilton Boon). https://doi.org/10.1016/j.jclinepi.2021.01.001 0895-4356/Ó 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/). M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 43 q Department of Health Research Methods, Michael G DeGroote Cochrane Canada and McMaster GRADE Centres, and WHO Collaborating Centre for Infectious Diseases, Research Methods and Recommendations, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada Accepted 12 January 2021; Published online 18 January 2021 Abstract Background and Objective: This article explores the need for conceptual advances and practical guidance in the application of the GRADE approach within public health contexts. Methods: We convened an expert workshop and conducted a scoping review to identify challenges experienced by GRADE users in public health contexts. We developed this concept article through thematic analysis and an iterative process of consultation and discussion conducted with members electronically and at three GRADE Working Group meetings. Results: Five priority issues can pose challenges for public health guideline developers and systematic reviewers when applying GRADE: (1) incorporating the perspectives of diverse stakeholders; (2) selecting and prioritizing health and ‘‘nonhealth’’ outcomes; (3) interpreting outcomes and identifying a threshold for decision-making; (4) assessing certainty of evidence from diverse sources, including nonrandomized studies; and (5) addressing implications for decision makers, including concerns about conditional recommendations. We illustrate these challenges with examples from public health guidelines and systematic reviews, identifying gaps where conceptual advances may facilitate the consistent application or further development of the methodology and provide solutions. Conclusion: The GRADE Public Health Group will respond to these challenges with solutions that are coherent with existing guidance and can be consistently implemented across public health decision-making contexts. Ó 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Keywords: GRADE; Public health; Guidelines; Nonrandomized studies; Health policy; Social determinants 1. Introduction 1.2. Background 1.1. Overview Public health is concerned with ‘‘preventing disease, For the past decade, the GRADE Working Group has prolonging life, and promoting health through the orga- produced tools and guidance with the aim of reducing un- nized efforts of society.’’ [4]. Public health encompasses necessary confusion and variation in rating the certainty three domainsdhealth protection, health services, and in synthesized evidence and improving the transparency health improvement [5]dwithin a disciplinary approach and methodological rigor of guidelines. Several GRADE which recognizes the impact of social determinants on indi- outputs and project groups have specifically addressed pub- vidual and population health outcomes and the importance lic health topics and used examples from public health of reducing health inequalities. Decisions made by public guidelines [1e3]. At the same time, public health guideline health practitioners, policymakers, and organizations may developers and others using GRADE in public health- relate to interventions that are not implemented at the indi- related contexts have identified ongoing challenges. To vidual level, including health system reform, regulation of address these challenges and consider the solutions, the unhealthy commodities (such as taxation of alcohol or GRADE Working Group approved in 2017 the formation sugar-sweetened beverages), infrastructure development, of the GRADE Public Health Project Group. This article, and social security policies. Depending on the approach which is the first output of this project group, is not taken, policy goals may be competing or may produce con- GRADE guidance but a GRADE concept article. A tradictory results, for example, when addressing health in- GRADE concept article is a new type of communication equalities [6]. Decisions about such interventions and from the GRADE Working Group that discusses the need situations involve stakeholders from many different profes- for conceptual advances in GRADE methodology. sions and organizations. Consensus on critical outcomes This article reports the results of activities undertaken to may be challenging to achieve and evidence from well- identify some of the challenges relating to GRADE and pub- conducted randomized studies may be lacking. However, lic health guidelines, specifically through an initial stake- despite any lack of randomized studies, decisions must be holder workshop and a scoping review. We propose based on the best available evidence. priority areas for further consideration and explain the ratio- In this context, the GRADE approach has strengths which nale for these choices. We present four case studies of public are well recognized. GRADE’s structured and transparent health guidelines and systematic reviews (Boxes 1-4) to methods for integrating evidence into decisions have been illustrate how these challenges have been encountered in tested and implemented by guideline developers worldwide. practice. The article concludes by summarizing the types Previous research has reported the adoption and further of solutions that may be proposed for these challenges and development of GRADE methods in public health contexts the next steps for the GRADE Public Health Group. [7e11]. Previous research has also investigated stakeholder 44 M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 cerns about weak or conditional recommendations. In addi- What is new? tion, participants raised the issue of the broad range of stakeholders involved in public health decision-making Key findings and how to address the differences and competing interests  Nine studies of guideline developers’ experiences that may arise when guidelines apply to sectors other than reported challenges when using GRADE in public health. Further challenges in applying GRADE included the health and related fields. need to integrate the perspectives of different disciplines and professions in areas such as planning, housing, trans-  The GRADE Public Health project group will port, and social security; the need to address competing respond to these challenges by producing guidance policy priorities and resource considerations across govern- relating to social determinants, population-level mental agencies and geographical boundaries; and the outcomes, and rating certainty of evidence from in- involvement of both public sector and commercial interests. terrupted time series. The formation of the GRADE Public Health project group was subsequently approved by the GRADE Guid- What this adds to what was known ance Group in October 2017 with the overarching goal to  Thematic analysis indicates the challenges relate to advance GRADE methodology in the development of pub- diverse perspectives and outcomes, decision- lic health guidelines. The work of the GRADE Public making thresholds, and nonrandomized studies. Health Group builds on previously developed and in-  Case studies drawing on Cochrane reviews and development GRADE guidance and related research on public health guidelines provide examples of how topics including equity, complex interventions, modeling these challenges are encountered in practice. studies, nonrandomized studies (NRS), and environmental and occupational health [21e27]. Here, we particularly What is the implication and what should change focus on interventions that impact on whole populations now? or large population groups (i.e., are not delivered at the in-  Detailed examples, new guidance, and targeted dividual level or whose benefits are realized at a population training materials should be developed to support level). Examples of such interventions include regulation of public health and public policy audiences using unhealthy commodities (e.g., restrictions on transfats in GRADE. processed foods and minimum unit pricing of alcohol); the provision of new infrastructure (e.g., clean water and sanitation, affordable housing, and modifications to trans- port); vaccination programs; and health system reforms views and experiences with GRADE and confirmed that (e.g., the integration of health and social care services). GRADE principles and processes are applicable to public health systematic reviews and guidelines [12e14]. Howev- er, challenges have also been identified for which various 2. Methods solutions have been proposed [12,14e20]. These chal- We conducted a scoping review with the aim of investi- lenges, along with the existence of multiple proposed solu- gating (i) experiences of applying GRADE within public tions, represent areas in which further elaboration of health systematic reviews and guideline development and GRADE guidance and production of exemplars may be (ii) existing research activity in this area. We included beneficial to ensure consistent application of the methodol- peer-reviewed published research on experiences, barriers, ogy, support the production of high-quality systematic re- and facilitators encountered in public health contexts in views and evidence-based guidelines, and allow for the adoption of GRADE. We did not include methodolog- further improvement of the GRADE system. ical or technical documents that reported only the methods To explore experiences of GRADE users and lay the and not the experiences or reflections of reviewers or devel- foundations for a GRADE public health project group, opers. We searched Medline and Embase (date range guideline developers and systematic reviewers were invited 2000e2018; see online supplement for search strategies); to participate in a workshop on GRADE for public health at used the snowballing technique to identify additional rele- the Global Evidence Summit in Cape Town, South Africa, vant studies; searched the reference lists of included arti- in September 2017. Twenty participants contributed to a cles; and contacted experts in the field. We performed a facilitated discussion of challenges encountered when narrative synthesis using thematic analysis to identify key applying GRADE to public health topics. Participants re- issues and challenges reported in the literature. We then ported issues with agreeing on critical outcomes, assessing developed the analysis through an iterative process of certainty when the body of evidence includes findings from consultation and discussion conducted with GRADE Work- disciplines other than health, making appropriate use of ing Group members electronically and at three GRADE nonrandomized studies, and addressing stakeholder con- Working Group meetings. M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 45 Case study 1 Cochrane systematic reviews on unconditional cash transfers Contributed by: Stefan Lhachimi Background Unconditional cash transfers (UCTs, money provided without obligation) are an intervention that addresses a key social determinant of health, i.e., income. Two recent Cochrane reviews [54,55] assessed the health effects of UCTs: the first in the context of (humanitarian or natural) disasters and the second as a social protection intervention for reducing poverty and vulnerabilities (e.g., orphanhood, old age, or HIV infection). Both reviews included studies conducted in low- and middle- income countries only and had as primary outcomes the use of health services and health outcomes. Secondary outcomes were relevant social determinants of health (e.g., assets, education, labor force participation, parenting quality, and extreme poverty) and health care expenditure. Examples of challenges Challenges 2 and 3: Most studies included in the review of UCTs as a social protection intervention were conducted by economists. Outcome reporting in the economic literature may not take into account considerations necessary for evidence synthesis, such as choosing and defining outcomes in common with already existing studies on the same topic or reporting intracluster correlation coefficients in cluster randomized trials. Some studies have dozens of very similar outcomes (e.g., number of goats, number of cows, number of chickens, etc.), making it difficult to choose which one to report in a systematic review. Moreover, for many outcome measures used (in particular indices of food security and diet diversity), no agreed minimal important difference exists. The Summary of Findings table went through several iterations during the review process before agreement was reached on which and how many outcomes to report. Challenges 4 and 5: In the review of UCTs for disaster relief, no high-quality evidence was identified. In addition, all included studies investigated UCTs as a response to only one type of disaster, namely droughts, which are different from most disasters in that droughts can be anticipated with a relatively long lead time before the consequences of the disaster materialize. Accordingly, the evidence was seriously indirect as well as low quality. Nevertheless, the UN at the World Humanitarian Summit in Istanbul 2016 called for making UCTs the default option for help during disasters. Intuitively, this seems plausible as UCTs in a short-term disaster context are (compared to in-kind transfers) swift and fairly easily to administer. However, this view does not account for potential crowding out effects of UCTs, i.e., unintended negative effects of UCT provision on other types of financial assistance during disasters. 3. Results the absence of a pooled effect estimate, difficulties with the evidence-to-recommendations process, and concerns We identified nine studies [12e16,28e31] that specif- about the resources required to implement GRADE meth- ically described the experiences of guideline development odology [14]. agencies or systematic review groups in applying GRADE Five reports [12,13,15,28,30] have described experi- to public health topics. ences of using GRADE in the context of World Health Or- Rehfuess and Akl gathered insights from 25 stake- ganization (WHO) guideline development. Akl et al holders in 12 organizations about their experiences and reported their experiences of applying GRADE in the views of using GRADE in the field of public health [14]. development of a WHO Department of HIV/AIDS guide- They grouped the challenges identified by participants into line on public health interventions for men who have sex six categories: with men and transgender people [12]. They describe the solutions they developed within a single guideline for the  Complexity of public health interventions. following nine challenges:  Choice of outcomes and outcome measures.  Ability to discriminate between different types of  Heterogenous and complex interventions. observational studies.  Paucity of trial data.  Use of nonepidemiological evidence.  Selecting outcomes of interest.  GRADE terminology.  Using indirect evidence.  GRADE and guideline development process.  Integrating values and preferences.  Considering resource use. Issues relating to GRADE terminology included defini-  Addressing social and legal barriers. tions of ‘‘quality of evidence,’’ the use of the term ‘‘obser-  Wording of recommendations. vational study,’’ and different understandings of terms such  Developing global guidelines. as ‘‘low-quality evidence’’ and ‘‘weak recommendations.’’ Participants reported challenges relating to the methodolog- Dedios et al. evaluated the methodological quality and ical difficulties of systematically reviewing nonrandomized implementability of guidelines produced by the WHO studies, applying GRADE criteria to narrative synthesis in Department of Nutrition for Health and Development 46 M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 Case study 2 Cochrane systematic reviews to inform WHO malaria vector control guidelines Contributed by: Leslie Choi Background In 2016, the WHO commissioned the development of new guidelines for malaria vector control [56] in partnership with the Cochrane Infectious Diseases Group (CIDG) based at the Liverpool School of Tropical Medicine. CIDG contributed eight systematic reviews [57e64] (five de novo, two updates, and one previously published review) that were used to inform the recommendations made in these guidelines. Examples of challenges Challenges 2 and 3: Vector control tools are typically public health interventions distributed at a community level. To evaluate efficacy at a community level, appropriate study designs with applicable outcomes are required. The main challenge encountered with these guidelines is how to tailor the guidelines for the correct target audience. Are they for individuals wanting to protect themselves from malaria or for national malaria program planners? Paradoxically, increased protection for some individuals may translate into increased risk for others in the community who are not as well protected. Leading on from this, it is difficult to assess whether the study evidence included in the systematic reviews demonstrates community protection. For example, the systematic review on topical repellents combined studies that distributed the intervention at an individual level and those that distributed at a community level. Therefore, it was unclear if the conclusions drawn from that review were applicable to both contexts. Challenge 4: Some modelling studies have suggested that poor coverage of vector control tools leads to more harm than good in a community by protecting a few individuals at the expense of the majority. However, these findings were difficult to capture and reconcile with the evidence from RCTs within the Summary of Findings and by extension the Evidence to Decision (EtD) framework. [15]. They suggested that difficulties encountered in their or strength of recommendations as issues. Alexander evaluation may reflect limitations of GRADE guidance, et al. investigated why WHO guideline developers make specifically citing NRS, time requirements, and wording ‘‘discordant recommendations,’’ i.e., strong Case study 3 Czech national public health guideline on 1-day surgery Contributed by: Miloslav Klugar Background One-day surgery was identified as a public health topic within the national Clinical Practice Guidelines project by the Guarantor Committee (representatives of important policymakers in the Czech Republic from the Ministry of Health, Institute of Health Information and Statistics, Czech Health Research Council, and health insurance organizations). One-day surgery has various definitions in different health systems worldwide; the definition for the Czech health system agreed by guideline panellists is ‘‘surgical performance (diagnostic and therapeutic) for hospitalization not exceeding 1 day (up to 24 hours of hospitalization, including one overnight stay).’’ In the Czech Republic, there are several surgical procedures in some hospitals in the 1-day surgery regime, while in other hospitals patients may be routinely hospitalized for the same surgical procedures for 3-6 days. One-day surgery clearly involves surgeons as a key audience, but also health care users, health care providers, and policymakers. In addition, following GRADE guidance, it was clear that social work and community care stakeholders needed to be engaged, as timely discharge from hospital and patient safety following discharge depend on social and community care. Examples of challenges Challenge 1: Initially the guideline panel membership included only surgeons and methodologists and the focus was on identifying which surgical procedures should be covered by 1-day surgery. There was initial resistance to inviting allied health professionals, patients, and other relevant stakeholders to join the panel. Furthermore, after framing the guideline questions the panel realized that social work/community care stakeholders should also be on the panel and that representation was needed from health and social insurance organizations, as well as homecare organizations. Identifying representatives from different organizations/sectors and managing the work of such a diverse guideline panel presented a challenge. Challenge 2: Although the Population, Intervention, Comparison, Outcome format is straightforward in its application to public health questions, the panel faced a challenge in prioritizing clinical versus social and community care outcomes, particularly in terms of the identification and management of postsurgical complications and patient safety after discharge from hospital. M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 47 Case study 4 European Breast Guidelines  pez Alcalde and Zuleika Saz Parkinson  s Lo Contributed by: Jesu Background The European Commission’s Joint Research Center (JRC) supports European Union policies with independent scientific evidence [65]. The JRC coordinates the development of the European guidelines for breast cancer screening and diagnosis [66] (in short, the European Breast Guidelines). These guidelines follow GRADE [67]. The European Breast Guidelines provide recommendations on population-based breast cancer screening programs and diagnostic procedures for breast cancer [10,68,69]. The target audience of the guidelines is heterogeneous, covering health care providers, service users, and policymakers. Examples of challenges Challenges 1 and 2: To define the perspective of the European Breast Guidelines was not straightforward. Some questions are purely clinical, taking an individual perspective, while others have a population perspective. These aspects influence the selection of the guideline questions and outcomes. The prioritization of the outcomes to determine the effects of breast cancer screening programs is also challenging; some panellists may take the safety of screening interventions for granted and be reluctant to include harms as outcomes. Challenges 3 and 4: The GRADE approach requires a multidisciplinary team with specific skills, which may not be available in all public health institutions. Therefore, certain tasks must be outsourced. The first guideline developed with GRADE in a public health institution can be logistically complicated, but the replication in future guidelines is expected to be more straightforward. Challenge 5: As RCTs for some questions are scarce, or even thought to be unethical, the use of GRADE is expected to generate conditional recommendations. The misconception that GRADE gives too much weight to RCT evidence may partly explain this. recommendations based on low or very low certainty evi- experts [16]. The ECDC Project on a Framework for Rating dence. Their interviews with guideline panel members Evidence in Public Health subsequently reported the dis- identified several challenges or barriers to GRADE applica- cussions and conclusions of an international expert meeting tion, including insufficient guidance on prioritizing out- on evaluating and grading evidence on infectious disease comes and rating NRS, different professional prevention and control [31]. This report concluded that backgrounds among panel members, and a political envi- GRADE can be applied successfully to guideline questions ronment that favors strong recommendations [13]. In a concerning the incidence and prevalence of diseases and further study by Alexander et al, interviews with GRADE suggested that some types of nonrandomized evidence, methodologists also identified the political environment as such as a body of evidence from interrupted time series a driver of ‘‘discordant’’ recommendations as well as issues (ITS) designs, should be rated more highly in GRADE as- with the management of conflicts of interest, urgent time- sessments. This is an area for which the GRADE Working scales, and the feasibility of implementing recommenda- Group has sought examples and operationalization since its tions [28]. Most recently, Gopinathan and Hoffman inception but failed to receive or identify clear methodolog- interviewed 35 senior WHO staff on their views about ical suggestions that would allow rating ITS higher. changes in WHO guideline development because its pro- Finally, Burford et al. (2012) summarized the benefits cesses were reformed starting in 2007 [30]. Among several and challenges when applying GRADE to Cochrane public themes identified in the interviews, challenges were identi- health systematic reviews [29]. These challenges related to fied relating to the assessment of NRS and qualitative NRS and rating certainty of evidence, as well as what to do studies, concerns which the authors note were subsequently in the absence of a pooled effect size. The article addressed addressed with the introduction of ROBINS-I [32] and some misconceptions relating to initial ratings of random- GRADE-CERQual [33]. ized and observational studies and noted that concerns Two reports [16,31] relate to guidelines or projects at the about low certainty of evidence are not unique to public European Center for Disease Prevention and Control health. The authors further noted that GRADE is particu- (ECDC). Within a review of evidence-based guideline larly helpful in addressing certain issues commonly methodology, the authors described issues with GRADE’s encountered when synthesizing public health evidence, terminology and treatment of NRS and noted an absence including the importance of context and the heterogeneity of guidance relating to economic and epidemiological that arises when combining findings from different public models, studies of incidence and prevalence of disease, health programs and health systems. and microbiological studies. They further noted the need The studies identified in the scoping review range in to integrate other aspects of decision-making with GRADE publication date from 2011 to 2018. During this period, de- processes, such as prioritization of topics and selection of velopments in GRADE methodology occurred that 48 M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 Table 1. Key challenges, examples, and proposed solutions in applying GRADE guidance to public health topics Challenge Examples Solutions 1. Incorporating diverse perspectives Convening guideline panels with diverse Forthcoming GRADE Public Health perspectives from different sectors, article, e.g., Czech national public health Addressing social determinants in public guideline on 1-day surgery (Box 3) health systematic reviews and Combining individual and population guidelines using GRADE perspectives, e.g., malaria vector Adapt GRADE training materials for control reviews (Box 2) and European nonhealth and policy audiences Breast Guidelines (Box 4) 2. Selecting and prioritizing outcomes Addressing individual and population outcomes in the same guideline or review, particularly when benefits and harms differ depending on the perspective taken, e.g., Cochrane reviews and WHO guideline on malaria vector control (Box 2) Achieving consensus on critical outcomes across different sectors, e.g., health care, social work, community care, and insurers in Czech national public health guideline on 1-day surgery (Box 3) Health vs. nonhealth outcomes, e.g., in transport policy 3. Interpreting outcomes and identifying Outcomes that have no defined Forthcoming GRADE Public Health a threshold for decision-making thresholds, e.g., Cochrane reviews of article, unconditional cash transfers (Box 1) Considerations when applying GRADE to Small changes may have important health population-level outcomes in public impacts at population level but not at health systematic reviews and individual level, e.g., taxation to guidelines address health externalities Different thresholds will have implications for modelling of the population health effects of policies and public health interventions 4. Assessing certainty of evidence from Cochrane reviews of environmental Contribute worked examples from public diverse sources, including interventions to reduce sugar- health reviews and guidelines to other nonrandomized studies sweetened beverage consumption and GRADE project groups, especially the interventions to reduce ambient air GRADE NRS group pollution found large differences in Operationalize criteria for when internal validity both within and across interrupted time series may avoid NRS typical selection and confounding bias Natural experiments may involve and be rated as moderate certainty randomization or allocation that is ‘‘as good as’’ random, and may be the best possible evidence for a policy-related question but are treated as nonrandomized or observational studies and generally start with a rating of low- quality evidence 5. Addressing implications for decision Making strong recommendations despite Collaboration with GRADE EtD group makers, including concerns about very low certainty evidence, e.g., ongoing since May 2019 conditional recommendations unconditional cash transfers for disaster relief (Box 1) addressed many, if not most, of the challenges identified. studies to be rated as high-quality evidence. In addition, ar- For example, GRADE changed its terminology about the ticles have since addressed the use of GRADE in the quality of evidence to be more flexible. Furthermore, absence of meta-analyses [18,34]. Overall, the existing GRADE had already altered the traditional evidence hierar- literature is consistent in identifying certain challenges in chy by allowing a body of evidence from observational using GRADE within public health systematic reviews print & web 4C=FPO M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 49 Fig. 1. Process of developing the scope and priorities of the GRADE Public Health Group. and guidelines, particularly regarding diverse stakeholder health care managers, health policymakers, patients, and perspectives within political environments, rating the cer- carers. The Health in All Policies approach is arguably tainty of nonrandomized evidence and wording or strength where this becomes most challenging, for example, when of recommendations. Most of the included studies make considering the health impacts of migration policy [35]. suggestions or recommendations to address these issues, In addition, to address the wider determinants of health, and many solutions have been developed. However, addi- public health guidelines may be targeted at professionals, tional solutions require further investigation, development, policymakers, and other stakeholders from nonhealth disci- and testing to ensure a systematic approach that is coherent plines and sectors whose perspectives on health evidence with existing GRADE guidance while also meeting the may vary; indeed, sometimes the primary responsibility needs of stakeholders. for financing and/or implementing public health interven- tions lies within these sectors. For example, the National Institute for Health and Care Excellence guideline on phys- 4. Analysis: key challenges and solutions when ical activity and the environment is aimed at local govern- applying GRADE guidance to public health topics ment and their contractors, employers, and community To formulate an approach to addressing the issues iden- organizations responsible for public spaces, housing author- tified through the scoping review, the GRADE Public ities, public transport planners and providers, and organiza- Health Group organized the challenges into themes, map- tions that support people with limited mobility [36]. The ped these to current GRADE Working Group activities, way that such varied audiences frame policy questions, identified gaps, discussed and presented potential activities the extent to which these audiences value health protection and outputs, and reached consensus on a program of work and improvement, and the priorities placed on various pol- including publications, contributions to other project icy approaches may all differ substantially [37], but need to groups, and training. The project group also identified be understood if the guideline is to achieve its aims. and discussed case studies through which the relevance Furthermore, different professions and stakeholder groups and applicability of the five themes were explored and may represent different ‘‘cultures of evidence’’ in which confirmed in a variety of public health contexts, including legislation, regulations, and other contextual factors may both systematic reviews and guidelines. Figure 1 summa- take priority over scientific research [38]. rizes the process of developing the scope and priorities of the GRADE Public Health Group, leading to a focus on five 4.2. Challenge 2: selecting and prioritizing outcomes key challenges. The five key challenges are described narratively below. Differences in perspectives and in cultures of evidence Table 1 summarizes the analysis of the challenges, gaps, also translate into challenges in selecting relevant outcomes and the project group response, including existing solu- and agreeing which outcomes are critical to decision- tions, planned outputs, and other activities. Boxes 1-4 pre- making. For example, in transportation policy any recom- sent the case studies, which have been selected to represent mendations about traffic management would ideally a diversity of public health topics (social determinants of consider all relevant outcomes, including respiratory health, health, infectious diseases, organization of health care, changes in physical activity levels, emergency admissions, and screening) drawn from both systematic reviews and and safety for vulnerable road users. However, transport guidelines (ten Cochrane reviews, one national public policymakers might not view these outcomes as critical health guideline, and one pan-national guideline develop- when weighed against traffic flow, commuting times, road ment program). traffic collisions, costs, provision of goods, or public opinion. Relatedly, challenges may arise when benefits are realized to a greater extent by communities than by in- 4.1. Challenge 1: incorporating diverse perspectives dividuals who receive treatment (e.g., immunizations which Many health guidelines are intended for multidisci- lead to herd immunity) [26]. A further challenge relates to plinary audiences, including different health professions, the potential trade-off between population health and health 50 M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 equity, in which a population health benefit may be studies [45]. Two recent Cochrane reviews, one on environ- achieved at the expense of widening inequalities. For mental interventions to reduce sugar-sweetened beverage example, a public health intervention such as screening or consumption [46] and one on interventions to reduce wellness checks may succeed in improving a population’s ambient air pollution [47] used nonrandomized intervention overall morbidity and mortality, but if the improved out- studies where no randomized studies were available; how- comes are predominantly achieved in a subgroup of higher ever, there were large differences in internal validity both socioeconomic status (SES), health inequality may inadver- within and across NRS. Existing GRADE guidance has tently be increased by widening the difference between the noted that the potential exists for different NRS designs outcomes of high versus low SES subgroups. The GRADE to provide moderate-quality evidence but that examples equity guidelines [21,24,26,27] are helpful in addressing are yet to be identified [25]. An active research agenda ex- this challenge. Finally, advancing health in the context of ists in the area of NRS and evidence synthesis, including the Sustainable Development Goals implies that health ben- the activity of the GRADE NRS group, the development efits may need to be weighed against broader environ- of ROBINS-I within the Cochrane Risk of Bias Methods mental, economic, and social impacts and their potential Group, and further adaptations of ROBINS-I for different interactions [39e41]. NRS designs [32,48]. Addressing this challenge by identi- fying relevant examples from public health policy and applying these new tools therefore seems a promising cata- 4.3. Challenge 3: interpreting outcomes and identifying lyst through which solutions to other challenges may also a threshold for decision-making emerge. The interpretation of evidence for a given outcome needs to be informed by knowledge of what change in that 4.5. Challenge 5: addressing implications for decision outcome would make a meaningful difference in a given makers, including concerns about conditional context. A change in a health outcome that might be recommendations deemed too small to be of importance to a patient may be perceived differently from a whole population perspec- As demonstrated in the scoping review, the predomi- tive, where small effects in many individuals may be nance of ‘‘weak’’ recommendations in public health guide- magnified to differences that are important for public health lines has been of concern to guideline developers and interventions and policy [42], as can be seen, for example, policymakers. GRADE has addressed this issue by chang- in population-wide salt reduction programs [43]. Further- ing the terminology (weak recommendations are in public more, many interventions at the population level are imple- health primarily labeled conditional and the conditions mented for primarily nonhealth reasons but act on key should be specified). It is already recognized that public determinants of health, for example, transport infrastruc- health produces several situations in which action may be ture, housing, and general taxation, and therefore may have strongly recommended despite low or very low certainty, profound health and health equity impacts. In these circum- including: stances, the effect size of an intervention may not be as  Life-threatening situations. informative for decision-making as the expected direction  Uncertain benefit but certain harm. of overall health impact (e.g., likely results in health gains  Potential equivalence of effectiveness in which one or health harms). Relatedly, equipoise may not exist for the option is clearly more or less risky or costly. primary nonhealth outcome [44]. These considerations of  Potential for catastrophic harm [49]. population-level impact, social determinants of health and inequalities, and lack of equipoise may all influence views Early strong recommendations in the context of very low on what constitutes an important difference and the certainty evidence emphasized that GRADE had recog- required degree of precision to support decision-making. nized this problem [50]. However, examples may still be encountered, such as in fiscal policy, where economists might strongly recommend austerity policies while public 4.4. Challenge 4: assessing certainty of evidence from health might produce a weak recommendation in terms of diverse sources, including NRS health outcomes. Grading recommendations as ‘‘condi- In public health, where randomized studies are less com- tional’’ rather than ‘‘weak’’ may partially address this issue, mon and often infeasible in comparison with other areas of but guideline developers may also need to better contextu- health, some types of NRS may provide greater certainty alize and ground such recommendations in ‘‘what else we than others when investigating the health effects of policy might know.’’ A Bayesian decision theory approach has or social interventions. For example, natural experiment been proposed in the context of this problem [51e53] studies may address selection bias and confounding and could be further explored. A key challenge for GRADE through designs such as ITS or regression discontinuity, in public health is therefore to identify how to reconcile the which may support stronger causal inference than other tension between the methodologically correct presentation observational designs such as cohort and case-control of evidence and recommendations as per GRADE and the M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 51 implications of strong vs. conditional recommendations public health topics in coordination with other from the perspective of decision makers in political GRADE project groups, including the NRS and EtD environments. groups. [Boxes 1-4: Case studies: four examples of challenges of  Adaptation of GRADE training materials for public applying GRADE in public health systematic reviews and health and public policy audiences. guidelines]. By providing a single, carefully documented, rigorous, and transparent method of evaluating evidence, GRADE 5. Discussion and conclusion serves to increase the trustworthiness, implementation, The views sourced from a participatory workshop, a and adaptability of systematic reviews and guidelines, scoping review, and four case studies suggest that key pub- while also helping to reduce research waste. It is therefore lic health-related challenges in application of GRADE our aim to address and mitigate any barriers to using methodology include the incorporation of stakeholder per- GRADE in public health and, through the work of the spectives from outside the health sector; agreement on the GRADE Public Health group, to build bridges and increase interpretation and prioritization of diverse outcomes; defini- mutual understanding across the wide range of policy and tion and interpretation of thresholds for population health research areas that intersect in the population health and nonhealth outcomes; assessing the certainty of evi- context. dence from diverse sources and NRS designs; and address- ing implications for decision makers, including the presentation of strength of recommendations to ensure util- CRediT authorship contribution statement ity and acceptability for stakeholders. These challenges may be categorized into three different Michele Hilton Boon: Conceptualization, Methodol- types, as per the different types of solutions that may be ogy, Writing - original draft, Writing - review & editing. proposed. The first type of solution is to identify the Hilary Thomson: Conceptualization, Writing - review & training needs of public health guideline developers (and editing. Beth Shaw: Conceptualization, Investigation, their stakeholders) in understanding and using the GRADE Writing - review & editing. Elie A. Akl: Conceptualization, concept, i.e., as per existing guidance. As different disci- Methodology, Writing - review & editing. Stefan K. Lha- plines or ‘‘cultures of evidence’’ come together in public chimi: Writing - original draft, Writing - review & editing. health, it will take time to speak a common (GRADE) lan- us Lopez-Alcalde: Writing - original draft, Writing - re- Jes guage. The second type of solution has to do with applying view & editing. Miloslav Klugar: Writing - original draft, existing GRADE guidance, including current develop- Writing - review & editing. Leslie Choi: Writing - original ments, to the public health context. As identified within draft, Writing - review & editing. Zuleika Saz-Parkinson: the scoping review, most of the existing GRADE guidance, Writing - original draft, Writing - review & editing. Reem both on rating certainty of the evidence and moving from A. Mustafa: Methodology, Writing - review & editing. evidence to decision, is already applicable to the public Miranda W. Langendam: Methodology, Writing - review health context but would benefit from translation and exam- & editing. Olivia Crane: Writing - review & editing. Re- ples, so that its relevance to public health practice and pol- becca L. Morgan: Writing - review & editing. Eva Re- icy can be more readily recognized and a population hfuess: Writing - review & editing. Bradley C. perspective can be consistently reflected in all steps. The Johnston: Writing - review & editing. Lee Yee Chong: third type of solution is to advance the GRADE methodol- Writing - review & editing. Gordon H. Guyatt: Methodol- ogy where existing guidance falls short, i.e., to develop new ogy, Writing - review & editing. Holger J. Sch€ unemann: GRADE guidance where necessary and appropriate. Conceptualization, Methodology, Writing - review & edit- The GRADE Public Health Group has considered the ing. Srinivasa Vittal Katikireddi: Conceptualization, challenges in the context of other GRADE outputs and Methodology, Investigation, Writing - review & editing. project groups, as well as other ongoing research in guide- line methodology [41,70], with the aim of avoiding unnec- essary duplication of effort. The solutions proposed by the Acknowledgments group include the following package of work: The authors gratefully acknowledge the participants at  Further GRADE concept articles or guidance to the initial GRADE Public Health workshop at the Global address social determinants in public health system- Evidence Summit in Cape Town, South Africa, for sharing atic reviews and guidelines using GRADE, consider- their experiences and views. The authors also thank the ations when applying GRADE to population-level following members of the GRADE Working Group for outcomes and criteria to assess certainty of interrup- comments on previous drafts of this article: Paul Garner, ted time series. Thomas Harder, Rachel Kettle, Hugh McGuire, Paul Mont-  The ongoing development and dissemination of gomery, Susan Norris, Juan Pablo Pe~na-Rosas, Nancy San- detailed examples of the application of GRADE to tesso, Ein-Soon Shin, Toni Tan, Gregory Traversy, and 52 M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 Vivian Welch. Article history: Concepts and cases in devel- [15] Dedios MC, Esperato A, De-Regil LM, Pena-Rosas JP, Norris SL. opment were presented on slides at GRADE Working Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation. Imple- Group meetings in Rome (2017), Manchester (2018), and mentation Sci 2017;12(1):39. Hamilton (2019). [16] European Centre for Disease Prevention and Control. Evidence-based methodologies for public health [Technical report]. 2011. Available at https://ecdc.europa.eu/en/publications-data/evidence-based- Supplementary data methodologies-public-health. Accessed February 6, 2020. [17] Harder T, Abu Sin M, Bosch-Capblanch X, Bruno C, de Carvalho Supplementary data to this article can be found online at Gomes H, Duclos P, et al. Towards a framework for evaluating and https://doi.org/10.1016/j.jclinepi.2021.01.001. grading evidence in public health. Health Policy 2015;119(6):732e6. [18] Thayer KA, Sch€unemann HJ. Using GRADE to respond to health References questions with different levels of urgency. Environ Int 2016;92-93: 585e9. [1] Alonso-Coello P, Sch€unemann HJ, Moberg J, Brignardello- [19] Florez ID, Morgan RL, Falavigna M, Kowalski SC, Zhang Y, Etxean- Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision dia-Ikobaltzeta I, et al. Development of rapid guidelines: 2. A quali- (EtD) frameworks: a systematic and transparent approach to making tative study with WHO guideline developers. Health Res Policy Syst well informed healthcare choices. 1: Introduction. BMJ 2016353. 2018;16(1):62e9. [2] Moberg J, Oxman AD, Rosenbaum S, Sch€unemann HJ, Guyatt G, [20] Sch€unemann H, Hill S, Guyatt G, Akl EA, Ahmed F. The GRADE Flottorp S, et al. The GRADE Evidence to Decision (EtD) framework approach and Bradford Hill’s criteria for causation. J Epidemiol for health system and public health decisions. Health Res Policy Syst Community Health 2011;65:392e5. 2018;16(1):45. [21] Akl EA, Welch V, Pottie K, Eslava-Schmalbach J, Darzi A, Sola I, [3] Sch€unemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, et al. GRADE equity guidelines 2: considering health equity in Guyatt G, et al. GRADE Guidelines: 16. GRADE evidence to deci- GRADE guideline development: equity extension of the guideline sion frameworks for tests in clinical practice and public health. J Clin development checklist. J Clin Epidemiol 2017;90:68e75. Epidemiol 2016;76:89e98. [22] Cuello-Garcia CA, Morgan RL, Brozek J, Santesso N, Verbeek J, [4] Acheson D. Public Health in England: The Report of the Committee Thayer K, et al. A scoping review and survey provides the rationale, of Inquiry into the Future Development to the Public Health Func- perceptions, and preferences for the integration of randomized and tion. London: The Stationery Office; 1988. nonrandomized studies in evidence syntheses and GRADE assess- [5] Griffiths S, Jewell T, Donnelly P. Public health in practice: the three ments. J Clin Epidemiol 2018;98:33e40. domains of public health. Public Health 2005;119(10):907e13. [23] Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, [6] Benach J, Malmusi D, Yasui Y, Martınez JM. A new typology of pol- et al. GRADE: assessing the quality of evidence in environmental and icies to tackle health inequalities and scenarios of impact based on occupational health. Environ Int 2016;92-93:611e6. Rose’s population approach. J Epidemiol Community Health 2013; [24] Pottie K, Welch V, Morton R, Akl EA, Eslava-Schmalbach JH, 67:286e91. Katikireddi V, et al. GRADE equity guidelines 4: considering health [7] Ahmed F, Temte JL, Campos-Outcalt D, Sch€unemann HJ, equity in GRADE guideline development: evidence to decision pro- Recommendation AEB, Group AEBRW. Methods for developing cess. J Clin Epidemiol 2017;90:84e91. evidence-based recommendations by the advisory committee on im- [25] Sch€unemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, munization practices (ACIP) of the U.S. Centers for disease control Thayer K, et al. GRADE guidelines: 18. How ROBINS-I and other and prevention (CDC). Vaccine 2011;29(49):9171e6. tools to assess risk of bias in nonrandomized studies should be used [8] Dahm P, Oxman AD, Djulbegovic B, Guyatt GH, Murad MH, to rate the certainty of a body of evidence. J Clin Epidemiol 2019; Amato L, et al. Stakeholders apply the GRADE evidence-to- 111:105e14. decision framework to facilitate coverage decisions. J Clin Epidemiol [26] Welch VA, Akl EA, Guyatt G, Pottie K, Eslava-Schmalbach J, 2017;86:129e39. Ansari MT, et al. GRADE equity guidelines 1: considering health eq- [9] Schunemann HJ, Hill SR, Kakad M, Vist GE, Bellamy R, uity in GRADE guideline development: introduction and rationale. J Stockman L, et al. Transparent development of the WHO rapid Clin Epidemiol 2017;90:59e67. advice guidelines. PLoS Med 2007;4(5):786e93. [27] Welch VA, Akl EA, Pottie K, Ansari MT, Briel M, Christensen R, [10] Schunemann HJ, Lerda D, Dimitrova N, Alonso-Coello P, et al. GRADE equity guidelines 3: considering health equity in Grawingholt A, Quinn C, et al. Methods for development of the Eu- GRADE guideline development: rating the certainty of synthesized ropean commission initiative on Breast cancer guidelines recommen- evidence. J Clin Epidemiol 2017;90:76e83. dations in the era of guideline transparency. Ann Intern Med 2019; [28] Alexander PE, Li S-A, Gionfriddo MR, Stoltzfus RJ, Neumann I, 171(4):273e80. Brito JP, et al. Senior GRADE methodologists encounter challenges [11] Thornton J, Alderson P, Tan T, Turner C, Latchem S, Shaw E, et al. as part of WHO guideline development panels: an inductive content Introducing GRADE across the NICE clinical guideline program. J analysis. J Clin Epidemiol 2016;70:123e8. Clin Epidemiol 2013;66:124e31. [29] Burford BJ, Rehfuess E, Schunemann HJ, Akl EA, Waters E, [12] Akl EA, Kennedy C, Konda K, Caceres CF, Horvath T, Ayala G, et al. Armstrong R, et al. Assessing evidence in public health: the added Using GRADE methodology for the development of public health value of GRADE. J Public Health 2012;34(4):631e5. guidelines for the prevention and treatment of HIV and other STIs [30] Gopinathan U, Hoffman SJ. Institutionalising an evidence-informed among men who have sex with men and transgender people. BMC approach to guideline development: progress and challenges at the Public Health 2012;12:386. World Health Organization. BMJ Glob Health 2018;3(5):e000716. [13] Alexander PE, Gionfriddo MR, Li SA, Bero L, Stoltzfus RJ, [31] Harder T, Abu Sin M, Bosch-Capblanch X, Bruno C, de Carvalho Neumann I, et al. A number of factors explain why WHO guideline Gomes H, Duclos P, et al. Towards a framework for evaluating and developers make strong recommendations inconsistent with GRADE grading evidence in public health. Health Policy 2015;119(6): guidance. J Clin Epidemiol 2016;70:111e22. 732e6. [14] Rehfuess EA, Akl EA. Current experience with applying the GRADE [32] Sterne JAC, Hernan MA, Reeves BC, Savovic J, Berkman ND, approach to public health interventions: an empirical study. BMC Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias Public Health 2013;13:9. in non-randomised studies of interventions. BMJ 2016;355:i4919. M. Hilton Boon et al. / Journal of Clinical Epidemiology 135 (2021) 42e53 53 [33] Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, [51] Fischer AJ, Threlfall A, Meah S, Cookson R, Rutter H, Kelly MP. Gulmezoglu M, et al. Using qualitative evidence in decision making The appraisal of public health interventions: an overview. J Public for health and social interventions: an approach to assess confidence Health 2013;35(4):488e94. in findings from qualitative evidence syntheses (GRADE-CERQual). [52] Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, PLoS Med 2015;12(10):e1001895. et al. The GRADE Working Group clarifies the construct of certainty [34] Murad MH, Mustafa RA, Sch€unemann HJ, Sultan S, Santesso N. Rat- of evidence. J Clin Epidemiol 2017;87:4e13. ing the certainty in evidence in the absence of a single estimate of [53] Threlfall AG, Meah S, Fischer AJ, Cookson R, Rutter H, Kelly MP. effect. Evid Based Med 2017;22(3):85e7. The appraisal of public health interventions: the use of theory. J Pub- [35] Juarez SP, Honkaniemi H, Dunlavy AC, Aldridge RW, Barreto ML, lic Health 2015;37(1):166e71. Katikireddi SV, et al. Effects of non-health-targeted policies on [54] Pega F, Liu SY, Walter S, Lhachimi SK. Unconditional cash transfers migrant health: a systematic review and meta-analysis. Lancet Glob for assistance in humanitarian disasters: effect on use of health ser- Health 2019;7(4):e420e35. vices and health outcomes in low- and middle-income countries. Co- [36] National Institute for Health and Care Excellence. Physical activity chrane Database Syst Rev 2015;(9):CD011247. and the environment. NICE guideline [NG90]. 2018. Available at [55] Pega F, Liu SY, Walter S, Pabayo R, Saith R, Lhachimi SK. Uncon- https://www.nice.org.uk/guidance/ng90. Accessed February 6, 2020. ditional cash transfers for reducing poverty and vulnerabilities: effect [37] Stone D. Policy paradox: The art of political decision making. New on use of health services and health outcomes in low- and middle-in- York: WW Norton; 1997. come countries. Cochrane Database Syst Rev 2017;(11):CD011135. [38] Lorenc T, Tyner EF, Petticrew M, Duffy S, Martineau FP, Phillips G, [56] World Health Organization. Guidelines for malaria vector control. et al. Cultures of evidence across policy sectors: systematic review of 2019. Available at https://www.who.int/malaria/publications/atoz/ qualitative evidence. Eur J Public Health 2014;24:1041e7. 9789241550499/en/. Accessed February 6, 2020. [39] Griffiths SM. The Sustainable Development Goals: an agenda for us [57] Choi L, Pryce J, Garner P. The combination of indoor residual spray- all. Perspect Public Health 2019;139(5):224e5. ing with insecticide-treated nets versus insecticide-treated nets alone [40] Morton S, Pencheon D, Squires N. Sustainable Development Goals for preventing malaria. Cochrane Database Syst Rev 2017;(6): (SDGs), and their implementation: a national global framework for CD012688. health, development and equity needs a systems approach at every [58] Choi L, Wilson A. Larviciding to control malaria. Cochrane Database level. Br Med Bull 2017;124(1):81e90. Syst Rev 2017;(7):CD012736. [41] Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, [59] Gleave K, Lissenden N, Richardson M, Choi L, Ranson H. Piperonyl but- Baltussen R. The WHO-INTEGRATE evidence to decision frame- oxide (PBO) combined with pyrethroids in insecticide-treated nets to pre- work version 1.0: integrating WHO norms and values and a vent malaria in Africa. Cochrane Database Syst Rev 2018;11:CD012776. complexity perspective. BMJ Glob Health 2019;4(Suppl 1):e000844. [60] Maia MF, Kliner M, Richardson M, Lengeler C, Moore SJ. Mosquito [42] Rose G. Strategy of prevention: lessons from cardiovascular disease. repellents for malaria prevention. Cochrane Database Syst Rev 2018; BMJ 1981;282:1847. 2:Cd011595. [43] Webster J, Waqanivalu T, Arcand J, Trieu K, Cappuccio FP, [61] Pluess B, Tanser FC, Lengeler C, Sharp BL. Indoor residual spraying Appel LJ, et al. Understanding the science that supports for preventing malaria. Cochrane Database Syst Rev 2010Cd006657. population-wide salt reduction programs. J Clin Hypertens (Green- [62] Pryce J, Choi L, Richardson M, Malone D. Insecticide space spraying wich, Conn) 2017;19(6):569e76. for preventing malaria transmission. Cochrane Database Syst Rev [44] Petticrew M, McKee M, Lock K, Green J, Phillips G. In search of so- 2018;11:Cd012689. cial equipoise. BMJ 2013;347:f4016. [63] Pryce J, Richardson M, Lengeler C. Insecticide-treated nets for pre- [45] Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: venting malaria. Cochrane Database Syst Rev 2018;(11):CD000363. an overview of methods, approaches, and contributions to public [64] Walshe DP, Garner P, Adeel AA, Pyke GH, Burkot TR. Larvivorous health intervention research. Annu Rev Public Health 2017;38(1): fish for preventing malaria transmission. Cochrane Database Syst Rev 39e56. 2017;(12):CD008090. [46] von Philipsborn P, Stratil JM, Burns J, Busert LK, Pfadenhauer LM, [65] European Commission. JRC in brief 2019. Available at https://ec. Polus S, et al. Environmental interventions to reduce the consumption europa.eu/jrc/en/about/jrc-in-brief. Accessed February 6, 2020. of sugar-sweetened beverages and their effects on health. Cochrane [66] European Commission. European Breast Guidelines 2019. Available at Database Syst Rev 2019;6:Cd012292. https://ecibc.jrc.ec.europa.eu/recommendations/. Accessed February 6, [47] Burns J, Boogaard H, Polus S, Pfadenhauer LM, Rohwer AC, van 2020. Erp AM, et al. Interventions to reduce ambient particulate matter [67] European Commission. Methods of the European guidelines for breast air pollution and their effect on health. Cochrane Database Syst cancer screening and diagnosis 2016. Available at https://ecibc.jrc.ec. Rev 2019;5:Cd010919. europa.eu/documents/20181/82470/GDGþMethodology/a8fb475b- [48] Morgan RL, Thayer KA, Santesso N, Holloway AC, Blain R, b5c6-474f-80f2-3ae8b9c27461. Accessed February 6, 2020. Eftim SE, et al. A risk of bias instrument for non-randomized studies [68] European Commission. Scope of the European guidelines for breast of exposures: a users’ guide to its application in the context of cancer screening and diagnosis. 2017. Available at https://ecibc.jrc. GRADE. Environ Int 2019;122:168e84. ec.europa.eu/-/scope-of-the-european-guidelines-for-breast-cancer- [49] Andrews JC, Sch€ unemann HJ, Oxman AD, Pottie K, Meerpohl JJ, screening-and-diagnos-1. Accessed February 6, 2020. Coello PA, et al. GRADE guidelines: 15. Going from evidence to [69] Sch€unemann HJ, Lerda D, Quinn C, Follmann M, Alonso-Coello P, recommendationddeterminants of a recommendation’s direction Rossi PG, et al. Breast cancer screening and diagnosis: a synopsis and strength. J Clin Epidemiol 2013;66:726e35. of the European Breast guidelines. Ann Intern Med 2020;172:46e56. [50] Schunemann HJ, Hill SR, Kakad M, Bellamy R, Uyeki TM, [70] Montgomery P, Movsisyan A, Grant SP, Macdonald G, Rehfuess EA. Hayden FG, et al. WHO rapid advice guidelines for pharmacological Considerations of complexity in rating certainty of evidence in sys- management of sporadic human infection with avian influenza A tematic reviews: a primer on using the GRADE approach in global (H5N1) virus. Lancet Infect Dis 2007;7:21e31. health. BMJ Glob Health 2019;4(Suppl 1):e000848.