Journal of Clinical Epidemiology 150 (2022) 51e62 REVIEW ARTICLE Evidence to decision frameworks enabled structured and explicit development of healthcare recommendations Jose F. Meneses-Echaveza,b,*, Julia Bidondea,c, Juan Jose Yepes-Nu~ nezd,e, Tina Poklepovic Pericicf, Livia Puljakg, Malgorzata M. Balah, Dawid Stormani, Mateusz J. Swierzh, Joanna Zaja˛ch, Camila Montesinos-Guevaraj, Yuan Zhangk, Nathaly Chavez Guapol, Holger Sch€ unemannk, Signe Flottorpm, Pablo Alonso-Coellon a Norwegian Institute of Public Health, Oslo, Norway b Facultad de Cultura Fısica, Deporte y Recreacion. Universidad Santo Tomas, Bogota, Colombia c School of Rehabilitation Science, College of Medicine, University of Saskatchewan, Canada d Universidad de los Andes, School of Medicine, Bogota, Colombia e Pulmonology Service, Internal Medicine Section, Fundacion Santa Fe de Bogota University Hospital, Bogota, Colombia f Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia g Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Zagreb, Croatia h Chair of Epidemiology and Preventive Medicine, Department of Hygiene and Dietetics, Systematic Reviews Unit, Jagiellonian University Medical College, Krakow, Poland i Chair of Epidemiology and Preventive Medicine, Department of Hygiene and Dietetics, Systematic Reviews Unit, Department of Adult Psychiatry, University Hospital Jagiellonian University Medical College, Krakow, Poland j Centro de Investigaci on en Salud Publica y Epidemiologıa Clınica (CISPEC), Facultad de Ciencias de la Salud Eugenio Espejo, Universidad UTE, Quito, Ecuador k Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada l Facultad de Enfermerıa y Rehabilitacion, Programa de Fisioterapia, Universidad de la Sabana, Chıa, Colombia m Institute of Health and Society, University of Oslo, Oslo, Norway n Iberoamerican Cochrane Center, Biomedical Research Institute Sant Pau (IIB Sant Pau-CIBERESP), Barcelona, Spain Accepted 7 June 2022; Published online 13 June 2022 Abstract Objective: The aim of this study is to identify and describe the processes suggested for the formulation of healthcare recommendations in healthcare guidelines available in guidance documents. Methods: We searched international databases in May 2020 to retrieve guidance documents published by organizations dedicated to guideline development. Pairs of researchers independently selected and extracted data about the characteristics of the guidance document, including explicit or implicit recommendation-related criteria and processes considered, as well as the use of evidence to decision (EtD) frameworks. Results: We included 68 guidance documents. Most organizations reported a system for grading the strength of recommendations (88%), half of them being the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach. Two out of three guidance documents (66%) proposed the use of a framework to guide the EtD process. The GRADE-EtD framework was the most often reported framework (19 organizations, 42%), whereas 20 organizations (44%) proposed their own multicriteria frameworks. Using any EtD framework was related with a more comprehensive set of recommendation-related criteria compared to no framework, especially for criteria like values, equity, and acceptability. Conclusion: Although limited, the use of EtD frameworks was associated with the inclusion of relevant recommendation criteria. Among the EtD structured frameworks, the GRADE-EtD framework offers the most comprehensive perspective for evidence-informed Conflict of interest statement: The authors declare that they have no Funding Source: Jose F. Meneses-Echavez received funding from Uni- competing interests. Jose Meneses-Echavez is a doctoral candidate in Pub- versidad Santo Tomas, Bogota, Colombia (FODEIN project code: lic Health and Methodology of Biomedical Research, at the Department of 2115005). Pediatrics, Obstetrics, Gynecology and Preventive Medicine at Universitat * Correspondence author: Researcher, Division for Health Services. Autonoma de Barcelona, Spain. Pablo Alonso-Coello, Signe Flottorp, and Norwegian Institute of Public Health, Sandakerveien 24C, Building D11, Holger Sch€ unemann were involved in the development of the GRADE-EtD 4th floor office 434, Oslo, Norway 0477. Tel.: þ47 486 89 405. framework. E-mail address: jose.meneses@fhi.no (J.F. Meneses-Echavez). https://doi.org/10.1016/j.jclinepi.2022.06.004 0895-4356/Ó 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/). 52 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 decision-making processes. Ó 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Keywords: Practice guidelines; Decision making; Evidence-based practice; Methods; Evidence-to-decision framework; Clinical practice guidelines; GRADE; Health recommendations; Certainty of evidence; Systematic reviews; Evidence synthesis 1. Introduction criteria being proposed, all the EtD frameworks aim to offer a comprehensive list of criteria needed to be considered by Guidelines for clinical, public health, and health policy, both decision makers and guideline developers. from here on guidelines, enhance decision making by trans- One of the most popular frameworks is the GRADE-EtD. lating complex scientific research findings into recommenda- There exists a growing body of research about the role of the tions for practice [1,2]. Guideline developers share the same GRADE-EtD framework in guideline development [12]. The aim, establishing standards of care backed by strong scienti- GRADE Working Group developed the EtD framework for fic evidence; however, they do not share the same methodo- different types of decisions, including recommendations [4]. logical expertise and resources for guideline development Details about the development process of the GRADE-EtD [1,2]. According to the National Academy of Medicine, trust- framework are available elsewhere [13]. The GRADE-EtD worthy guidelines should ‘‘be developed by a knowledge- framework aims to help panel members use evidence in a able, multidisciplinary panel of experts and representatives structured and transparent way to inform healthcare decisions. from key affected groups; be based on a systematic review Besides, the framework helps guideline development groups of the existing evidence; consider important patient sub- (GDG) to considering the most relevant criteria that influence groups and patient preferences as appropriate; be based on a decision by structuring discussions, identifying reasons for an explicit and transparent process that minimizes distor- disagreements, and building the basis for a transparent deci- tions, biases, and conflicts of interest; provide a clear expla- sion [4,13] (Fig. 1). The GRADE-EtD framework has been nation of the logical relationships between alternative care used by international organizations including the World options and health outcomes and provide ratings of both Health Organization (WHO), the European Commission the quality of evidence and the strength of the recommenda- [7], and major scientific societies [14e16]. Some panelists tions; and be reconsidered and revised as appropriate when have acknowledged the relevance of the framework in facili- important new evidence warrants modifications’’ [2]. tating both structured panel meetings and the consideration of The process of moving from evidence to recommenda- all relevant criteria pertaining to decision making [17]. tions represents a cornerstone of guideline development The controversy around the best fit for purpose frame- [2,3]. This process implies a systematic and transparent work remains in the literature as new proposals continue integration of the evidence supporting the criteria influ- to emerge even in recent years [10]. In fact, despite the encing the recommendation, as expressed by item 13 in progress achieved in studying the importance of various the Guidelines 2.0 checklist [3]. Due to the broad variety recommendation-related criteria, to our knowledge, little of recommendation-related criteria suggested by interna- is known on how GDGs work, the EtD processes they tional organizations [2e7], the McMaster checklist, a tool follow, and the criteria that organizations consider when for guideline development, points out that a framework out- formulating recommendations. This gap constrains the lining the criteria to be considered to arrive at a decision transparency of reporting of the methods through which (e.g., the magnitude of the difference between benefits healthcare recommendations were developed and affects and harms, and resource use) should guide the recommen- the extent to which end users can understand the evidence dation formulation process [3]. grounding the recommendations and whether these might A number of frameworks have been proposed for ad- be implemented in their own setting. dressing this process, such as the GRADE-EtD (Grading Therefore, our study addresses the following question: of Recommendations Assessment, Development, and Eval- What are the processes and frameworks suggested for the uation evidence to decision) framework [4], the ‘‘decision- formulation of recommendations in healthcare guidelines making triangle’’ [8], the Evidence and Value: Impact on available in guidance documents? DEcision Making framework [6], or the Guidance for Prior- ity Setting in Health Care (GPS-Health) [9]. Some research groups and organizations have launched other EtD frame- works and processes, mostly based on the criteria and sub- 2. Methods criteria contained in the GRADE-EtD framework [10,11]. 2.1. Design Some of EtD frameworks emphasize specific criteria, such as equity in the GPS-Health [9], or ethics as the ‘‘decision- Methodological study. We followed the methods for the making triangle’’ [8]. Despite the variation in terms of the conduct of a systematic review (e.g., double data J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 53 2.3. Literature searches What is new? We systematically searched the Guidelines International Network library, MEDLINE, Google Scholar, and The Co- Key finding chrane Methodology Register using key terms such as  One in three organizations do not report a struc- ‘‘handbook*,’’ ‘‘guideline methodology,’’ ‘‘clinical practice tured Evidence to Decision (EtD) process in their guideline*,’’ and ‘‘manual.’’ Then, we inspected the refer- guidance documents. ences listed in the included documents and websites of rele-  Using an EtD framework is associated with a more vant organizations identified in previous research projects comprehensive process, including additional [1,20e22]. These procedures (search and inspection of ref- criteria, like values, equity, and acceptability. erences) were first conducted during October 2018, and further updated in May 2020. The searches had no language What this adds to what was known? restrictions except for Google Scholar, which was restricted  The EtD process in guidance documents from to English. We contacted experts in the field. Additional file guideline development organizations is often 1 describes the search strategies. unstructured.  Most organizations consider desirable effects, 2.4. Study selection and data extraction undesirable effects, and certainty of the evidence We designed and piloted an ad hoc data abstraction form of effects. Other criteria, such as patients’ values based on the WHO handbook for guideline development [7] and preferences, cost-effectiveness, equity, because of its comprehensiveness, relevance, and formative acceptability, or feasibility are less frequently influence on other guideline documents. The form covered considered. information on the main characteristics of the guidance documents, as well as characteristics of the methodology What is the implication and what should change [e.g., panel composition and management of conflicts of in- now? terest (COI)], and information on the EtD process,  Guidance documents for guideline development including the use of frameworks [4]. We defined a struc- require improvements in the reporting of the EtD tured process as any systematic series of steps taken to process. fulfill a goal, and a framework as any structure of concepts  Guideline development organizations should underlying a structured process, in this case the process of follow a more systematic and explicit methodology formulating recommendations (EtD process). We explored for formulating recommendations. the full-text of each guidance document for information about the suggested EtD process, including the use of  A more complete and detailed reporting of the EtD frameworks, and listed all recommendation-related criteria process should be included in guideline develop- considered. Both selection and data extraction processes ment guidance documents. were undertaken independently by pairs of reviewers, with  Organizations may use our findings to either pre- discrepancies solved by consensus or by involving a third pare their own guidance documents or to reassess reviewer, if needed. their existing ones by identifying gaps in EtD criteria or to follow suggestions for improvement. 2.5. Data management and analysis Pairs of reviewers read the guidance documents and ex- tracted the information pertaining to the recommendation formulation process (e.g., direct quotes). We held rounds of vir- extraction) [18]. We published the protocol in Open Sci- tual meetings to ensure accuracy and completeness of the data. ence Framework [19]. All data extraction forms were then compiled by one reviewer (J.F.M.E.), who run descriptive analysis, with frequencies and 2.2. Inclusion criteria percentages (e.g., characteristics of the organizations, panel composition, and COIs management). Based on the EtD pro- We included the latest version of methodological guid- cess and the recommendation-related criteria identified in the ance documents for guideline development, published guidance documents, we created the following categories: from 2003 as organization reports, journal articles, or briefs, developed by institutions responsible for guideline 1. Use of any framework for the EtD process development in any field of health care. We discarded 1.1. Use of GRADE-EtD framework guidance documents dealing with processes of updating 1.2. Use of other EtD framework or adapting guidelines as well as those endorsing included documents. 2. No framework 54 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 The last step of the analysis involved the study of the Most of the guidance documents provided some degree of probability of addressing each criterion across the different information about the composition of guideline panels. A categories of frameworks presented above (i.e., any frame- multidisciplinary panel, including a steering group, method- work vs. no framework; GRADE-EtD vs. other EtD frame- ologists with expertise in evidence synthesis, and health pro- work; GRADE-EtD vs. no framework; other EtD fessionals specialized in the topic of interest was the most framework vs. no framework), for which we estimated odds common composition. More than half of the documents ratios (OR) with their corresponding 95% confidence inter- (64%) suggested the involvement of patients or consumer vals (CI). We used R-project software for the analysis [23]. representatives (Additional file 4). Half of the guidance doc- J.F.M.E. carried out the statistical analysis, and a second uments presented a definition of COIs and presented a pro- reviewer (P.A.C.) audited it for accuracy. cess to report the COIs. Although 87% of the guidance documents specified who should report COIs, only half of them stated that any declaration of COIs should be reviewed 3. Results before making appointments to the GDG and specified the types of COIs to be declared (i.e., financial and nonfinan- The search resulted in a total of 8,838 records. After we cial). Half of the documents (59%) described a process for removed 268 duplicates, 8,570 records remained. We the management of COIs (see Additional file 4). excluded 8,478 at full-text and assessed the 92 full-text doc- uments for eligibility (Additional file 2). Of these, we 3.1. Panel composition and conflicts of interest excluded 24 documents, either because they were not a guid- management ance document, were a previous version of an included document, were impossible to retrieve, or were publications 3.1.1. Rating the quality of evidence and grading the related to documents already included. We included a total strength of recommendations. of 68 guidance documents from 14 countries (Fig. 2). Almost all guidance documents (93%) suggested a struc- Additional file 3 presents the names of the organizations tured approach or system for rating the quality of the evi- and their corresponding guidance documents included in this dence; most of them (85%) contained a specific section or study. The median publication year of the documents was chapter. GRADE was the most common approach suggested 2015 (range 2003e2020). Scientific societies published most (53%), followed by the approaches proposed by the Austra- of the documents (58%), followed by governmental (20%) lian National Health and Medical Research Council and the and supranational organizations (13%). Nearly half of the Scottish Intercollegiate Guidelines Network (SIGN), re- documents were from North America (45%), followed by ported in three documents each (4%). Of note, seven docu- Europe (37%), Asia and Oceania (6%), and South America ments (10%) suggested rating systems that were based on (3%). See Figure 3 and Additional file 4 for further details GRADE or other approaches. Finally, nine documents on the characteristics of the included guidance documents. (13%) proposed their own systems, mainly based on the pre- Figure 2 does not show six guidance documents published vious approaches. See Additional file 4 for further details. by European organizations, WHO, and World Confederation Most of the guidance documents (88%) proposed a system for Physical Therapy. for grading the strength of recommendations; half of them Fig. 1. GRADE-EtD frameworks workflow. Illustration: Sarah Rosenbaum, Norwegian Institute of Public Health. GRADE-EtD, Grading of Recommendations Assessment, Development, and Evaluation evidence to decision J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 55 Fig. 2. Selection process. suggested the GRADE approach. Around a fifth of the docu- (1 document), NICE (National Institute for Health Care and ments (22%) suggested their own approach, whereas 14% of Excellence; 1 document), and US Preventive Services Task the documents did not report any. Other approaches were Force (USPSTF; 1 document). Five documents (8%) proposed SIGN (3%), Oxford Center for Evidence-Based Medicine other approaches, mostly adapted from GRADE (Table 1). Number of Guidance Documents > 10 guidance documents 6-9 guidance documents 2-5 guidance documents 1 guidance documents Fig. 3. Geographical distribution of the included guidance documents. 56 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 3.2. Recommendation formulation guidance documents that suggested an EtD framework considered a more comprehensive set of criteria than those Two out of three guidance documents (66%) suggested who did not suggest any framework. All guidance docu- a structured process for formulating recommendations, ments had a major focus in their EtD frameworks on desir- with the entire panel being involved in more than half able effects, undesirable effects, and certainty of the (56%) of the documents. In contrast, 28% of the guidance evidence of effects; guidance documents that suggested documents failed to report who was involved in the pro- any framework for guiding the EtD process had higher re- cess of formulating recommendations. The technical team porting rates (O80%) than those who suggested no frame- needed to share preliminary material (e.g., evidence sum- work (52%). Other criteria such as equity, acceptability, maries and evidence-to-recommendation tables) with the and feasibility were reported in less than half of the docu- guideline panel ahead of GDG meetings in 40% of the ments that guided the EtD process through a systematic guidance documents, and 48% of the documents reported framework (any EtD framework), whereas considerably that the technical team should make preliminary judg- lower rates of use were observed among the documents that ments about the quality of the evidence. A lower number did not suggest a framework to guide the EtD process (e.g., of documents (37%) declared that preliminary judgments from 0% to 22%). Of note, one document reported on the on the certainty of the evidence or any other factor (e.g., consideration of legal consequences (Domus Medica) and values and preferences, equity, resources required) should on bioethical considerations [Italian National Center for be made before the panel meetings. A smaller number of Clinical Excellence, Quality, and Security (CNEC)], documents (22%) suggested the same for deciding about respectively. the direction and strength of recommendations (10%) Among the documents that suggested a framework for (Table 1). guiding the EtD process, those that suggest the GRADE- EtD framework addressed a larger number of criteria 3.3. Use of evidence to decision frameworks compared to those that suggested another framework for the EtD process (Table 3). Rates of use differed between The GRADE-EtD framework was the most often re- the two categories. For instance, 95% of the guidance doc- ported (42%), followed by NICE’s framework (8%), uments that suggested the GRADE-EtD framework re- SIGN (4%), and USPSTF (4%). Twenty guidance docu- ported the consideration of patients’ values relative to ments (76%) reported their own multicriteria framework. 46% of those that used another EtD framework. A larger Nearly half of the documents (57%) provided guidance difference was observed for equity considerations, 42% for formulating recommendations when there is insuffi- in the GRADE-EtD framework and 11% in another cient evidence or no evidence available, or low/very EtD framework, respectively. Table 2 presents all the low-quality evidence. Most of the documents (82%) sug- recommendation-related criteria reported in the guidance gested a method to reach agreement among guideline documents. Results from the bivariate analysis are pre- panel members [e.g., consensus, voting (majority rule), sented in the section below. or nominal group techniques] (Table 1). The documents failed to provide specific definitions of the criteria. Although the wording of some 3.4. Setting, perspective, and subgroup considerations recommendation-related criteria may vary across the docu- We found that 42% and 24% of the guidance documents ments, which might be explained by the organization pref- that included an EtD framework specified the setting and erences, this refers to the same set of criteria described in perspective, respectively. These rates were considerably Table 2. Criteria wording variation are expected for organi- lower among documents that did not suggest any frame- zations that adapt a framework and process for arriving at work (17% setting and 9% perspective). Guidance docu- recommendations. Thus, some organizations may have ad- ments suggesting the GRADE-EtD framework were the dressed the magnitude of the problem before assessing the most common ones in reporting the inclusion of both evidence pertaining to the EtD process, and therefore might setting and perspective (42%), followed by those that sug- not report this as an independent criterion in their EtD gested another EtD framework (42% and 11%, respec- framework. tively). The inclusion of subgroups considerations showed To illustrate this better, the GRADE-EtD framework pre- a similar pattern among the guidance documents (i.e., sents a set of additional considerations, the so-called 31% in the use of any EtD framework and 4% in the use detailed judgments, that assist the panel when considering of no framework). the evidence that underlie the main criteria [4]. For example, legal consequences, reported by Domus Medica as a specific criterion, would be covered by the GRADE- 3.5. Recommendation-related criteria EtD framework as one of the detailed judgments included A total of 14 recommendation-related criteria were under feasibility (i.e., Are there important legal or bureau- identified across the guidance documents. Overall, cratic constraints that make it difficult or impossible to J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 57 Table 1. Recommendation formulation information reported in the included guidance documents Recommendation formulation information n (%) Contains specific section in the document 55 (81%) Details of people involved Entire panel/GDG, 8 (56%) Panel þ other, 5 (7%) Subgroup of the panel, 3 (4%) Not specified, 19 (28%) Technical team shares materials (e.g., evidence summaries) with the 27 (40%) guideline panel ahead of meeting Technical team or someone else makes preliminary judgments on the 25 (37%) different criteria (e.g., certainty of evidence) Technical team or someone else makes preliminary judgment on the 15 (22%) strength of recommendations (e.g., strong, conditional) Technical team or someone else makes preliminary judgments about 7 (10%) the direction of recommendations (e.g., in favor, against) Approach to grading the strength of recommendations 60 (88%) Approach suggested for grading the strength of recommendations GRADE, 35 (51%) NHMRC, 2 (3%) SIGN, 2 (3%) CEBM, 1 (1.5%) NICE, 1 (1.5%) USPSTF, 1 (1.5%) Adapted systems (5, 7%) GRADE þ NICE, 1 (1.5%) NICE þ SIGN, 2 (3%) GRADE þ SIGN, 1 (1.5%) GRADE þ AHRQ þ USPSTF, 1 (1.5%) Other or not specified, 21 (36%) Use of a framework for the EtD process 45 (66%) Frameworks suggested for the EtD process GRADE-EtD, 19 (42%) Other approaches, 26 (58%) Own approach, 20 (76%) NICE, 2 (8%) SIGN, 1 (4%) USPSTF, 1 (4%) SIGN þ NICE, 1 (4%) GRADE þ SIGN þ AHRQ, 1 (4%) Explicit method to reach agreement among panel members (e.g., 56 (82%) consensus, nominal group techniques) Documentation of judgements made 27 (38%) AHRQ, The Agency for Healthcare Research and Quality (USA); CEBM, The Center for Evidence-Based Medicine, based in the Nuffield Depart- ment of Primary Care Health Sciences at the University of Oxford; EtD, evidence to decision; GDG, guideline development group; GRADE, The Grading of Recommendations Assessment, Development, and Evaluation; NHMRC, National Health and Medical Research Council (Australia); NICE, The National Institute for Health Care and Excellence (UK); SIGN, Scottish Intercollegiate Guidelines Network; USPSTF, The US Preventive Services Task Force. cover the intervention?). The same principle applies to stakeholders who would disapprove of the intervention ethical considerations, which is suggested by the CNEC, morally, for reasons other than its effects on people’s auton- and is addressed as a detailed judgment under acceptability omy (such as regarding ethical principles such as no malef- by the GRADE-EtD framework [i.e., Are there key icence, beneficence, or justice)?] [4]. 58 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 Table 2. Recommendation-related criteria in the EtD process All guidance documents Any framework No framework GRADE-EtDa Other framework Criteria 68 (100%) 45/68 (66%) 23/68 (34%) 19/45(42%) 26/45(58%) Problem priority 40 (59%) 30 (67%) 10 (43%) 12 (63%) 18 (69%) Desirable effects 49 (72%) 37 (82%) 12 (52%) 17 (89%) 20 (77%) Undesirable effects 50 (73%) 38 (84%) 12 (52%) 17 (89%) 21 (81%) Certainty of the evidence of effects 50 (73%) 38 (84%) 12 (52%) 18 (95%) 20 (77%) Values (outcome importance) 35 (51%) 30 (67%) 5 (22%) 18 (95%) 12 (46%) Balance of effects 39 (57%) 32 (71%) 7 (30%) 17 (89%) 15 (58%) Resources required 37 (54%) 31 (69%) 6 (26%) 18 (95%) 13 (50%) Certainty of evidence of required resources 17 (25%) 16 (36%) 1 (4%) 9 (47%) 7 (27%) Cost-effectiveness 36 (53%) 30 (67%) 6 (26%) 16 (84%) 14 (54%) Equity 11 (16%) 11 (24%) 0 8 (42%) 3 (11%) Acceptability 19 (28%) 17 (38%) 2 (9%) 9 (47%) 8 (31%) Feasibility 23 (34%) 18 (40%) 5 (22%) 9 (47%) 9 (35%) GRADE, The Grading of Recommendations Assessment, Development, and Evaluation; EtD, evidence to decision. a Guidance documents that suggested the GRADE-EtD frameworks for the process of formulating recommendations. 3.6. Drawing conclusions as part of the evidence to 3.7. Bivariate analysis decision process As stated in the methods section, we express the results Overall, 41% of the guidance documents reported a of the bivariate analysis as the probabilities of addressing process to summarize the judgments made about the each recommendation-related criterion in the different cat- different recommendation-related criteria. This step egories of the EtD process (i.e., any framework vs. no was more common in the documents that suggested the framework; GRADE-EtD vs. other EtD framework; use of a framework (49%) compared to documents that GRADE-EtD vs. no framework; other EtD framework vs. did not (26%). We did not observe major differences in no framework). The following is a summary of the main this step between those suggesting the use of the findings. We refer the reader to Additional file 5 for a full GRADE-EtD framework and another framework (47% description of the data. vs. 50%, respectively). Similarly, the justification of The use of an EtD framework for guiding the recom- the recommendation’s strength and direction was more mendation formulation compared to no framework resulted common in documents that used a framework (53% vs. in higher probability of incorporating both perspectives 17%, respectively). This trend was also observed for (OR 2.8, 95% CI 0.6e13.8) and subgroup considerations other steps, such as considerations for relevant sub- (OR 7.2; 95% CI 0.9e57.9). Similarly, the documents that groups, implementation, monitoring, and evaluation, as incorporated the GRADE-EtD framework were more likely well as about the formulation priorities for further to incorporate these criteria in the recommendation formu- research (Table 3). lation process than those that suggested another EtD Table 3. Drawing conclusions All guidance documents Any framework No framework GRADE-EtDa Other framework Criteria 68 (100%) 45/68 (66%) 23/68 (34%) 19/45 (42%) 26/45 (58%) Summary of the judgments made about 28 (41%) 22 (49%) 6 (26%) 9 (47%) 13 (50%) the different criteria considered Justification of the recommendation 28 (41%) 24 (53%) 4 (17%) 11 (58%) 13 (50%) Subgroup considerations 13 (19%) 12 (27%) 1 (4%) 7 (37%) 5 (19%) Implementation considerations 38 (56%) 31 (69%) 7 (30%) 14 (74%) 17 (65%) Monitoring and evaluation considerations 30 (44%) 25 (56%) 5 (22%) 14 (74%) 11 (42%) Research priorities 19 (28%) 18 (40%) 1 (4%) 9 (47%) 9 (35%) GRADE, The Grading of Recommendations Assessment, Development, and Evaluation; EtD, evidence to decision. a Guidance documents that suggested the GRADE-EtD frameworks for the process of formulating recommendations. J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 59 Any framework GRADE-EtD framework 20 Other EtD framework 10 OR (95% CI) 5 1 Undesirable Certainty of Paents’ Certainty of Cost- Problem Desirable Balance of Resources Equity Acceptability Feasibility the evidence values evidence of priority effects effects effects required effecveness of effects resources required Fig. 4. Bivariate analysis for the recommendation-related criteria in the guidance documents. The odds ratios are based on the comparison of each category of EtD framework relative to the use of no framework. The vertical lines illustrate the 95% confidence intervals. GRADE, The Grading of Recommendations Assessment, Development and Evaluation; EtD, evidence to decision. framework or no framework, with ORs ranging from 1.4 to GRADE-EtD framework were more likely to incorporate 8.4, respectively. cost-effectiveness considerations in the recommendation The probability of using all the recommendation-related formulation process than those that suggested another criteria identified in this study were higher in documents framework (OR 1.6; 95% CI 0.6e3.9) or no framework that suggested the use of any EtD framework relative to (OR 3.2; 95% CI 1.1e9.8) (Fig. 4). no framework, as well as for the documents that suggested Of note, the odds of including recommendation-related the GRADE-EtD framework compared to those that sug- criteria such as equity, acceptability, and feasibility fol- gested another framework or no framework (Fig. 4). lowed the same pattern. That is, the use of any EtD frame- For instance, guidance documents that suggested the use work and the GRADE-EtD framework resulted in higher of any EtD framework were more likely to consider pa- probability of including those criteria when formulating tients’ values in the recommendation formulation process recommendations relative to the use of no framework or compared to those that did not follow any framework another EtD framework. (OR 3.1; 95% CI 1e8.9). The odds of including patients’ Finally, and in line with the associations observed for the values were twofold greater in documents that suggested recommendation-related criteria, the documents that sug- the GRADE-EtD framework relative to those that sug- gested both the use of any framework or the GRADE-EtD gested another framework (OR 2; 95% CI 0.8e5.4). The framework were more likely to provide a justification of the OR increased to 4 when the GRADE-EtD was compared judgements made, implementation considerations, as well as to no framework (OR 4.4; 95% CI 1.4e13.9) (Fig. 4). monitoring and evaluation considerations than the documents The guidance documents that suggested any EtD frame- that suggested no framework or another framework. work were more likely to present evidence on the balance Additional file 5 presents further details. between desirable and undesirable effects when formu- lating recommendations than documents that did not sug- gest any framework (OR 2.3; 95% CI 0.9e6.1). The odds were larger for the comparison of the documents that sug- 4. Discussion gested the GRADE-EtD framework with those that used no 4.1. Main findings framework (OR 2.9; 95% CI 1.1e8.6). Similar odds were observed for the criterion related to resources required. Our study presents a complete and systematic evaluation Suggesting the use of an EtD framework was associated of the frameworks and processes suggested when moving with higher odds of including cost-effectiveness consider- from evidence to recommendations in guideline develop- ations when formulating recommendations relative to the ment. We documented the diversity of both frameworks use of no framework (OR 2.5; 95% CI 0.9e7.1). Alike and processes in this area. GRADE emerged as the most other criteria, the documents that suggested the use of the common approach for grading both the certainty of the 60 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 evidence and the strength of recommendations. Our ana- under acceptability in the GRADE-EtD frameworks. There- lyses revealed that only slightly more than half of the orga- fore, an organization might consider treating ethical consid- nizations presented a structured process for formulating erations as a separate criterion, rather than as a detailed recommendations, in their guidance documents. However, judgment under acceptability [4]. the analyses also showed that there are gaps in important aspects like the sharing of preliminary material or judge- 4.3. Limitations and strengths ments through the recommendation formulation process. Fourteen recommendation-related criteria were identi- Our study had some limitations. We do not rule out hav- fied in the guidance documents. The use of an EtD frame- ing excluded documents that organizations may have pub- work was associated with higher probabilities of addressing lished in other languages. Nevertheless, we do not a comprehensive set of recommendation-related criteria anticipate missing relevant information, as our searches compared to the use of no framework, especially for criteria covered a representative number of guidance documents like patients’ values and preferences, equity, and accept- derived from the most relevant organizations in the field. ability. Similarly, the use of the GRADE-EtD framework Regarding the strengths, we followed methodological was associated with higher probabilities of including those standards for the conduct and reporting of evidence synthe- criteria when compared to other EtD frameworks or no sis studies, such as systematic search of the literature, dou- framework. However, caution is advised when drawing in- ble screening process, and a priori piloting of the data ferences from the analysis due to the small number of guid- extraction procedures. Data were analyzed through iterative ance documents included in our analysis. consensus between researchers. The project team was stra- tegically positioned to carry out this study, as the re- searchers are experienced in working with guideline 4.2. Our results in the context of previous research panels and are experts in evidence synthesis methods and Previous research suggests that the GRADE-EtD frame- the guideline development process. work is a widely used tool for assisting comprehensive and transparent evidence-informed decision making 4.4. Implications for practice and research [12,13,24,25]. Our findings confirm this notion, because the methods developed by the GRADE Working Group Our findings provide a systematic analysis of different were the most used approach. For example, a case study criteria contained in the frameworks proposed for the conducted by members of the GRADE working group recommendation formulation process among guidance doc- [12] showed that the GRADE-EtD coverage framework uments for guideline development. Different stakeholders had been fully accepted by some stakeholders in the United (e.g., patients and clinicians) may use our findings to better States [24]. Similarly, a more recent real-time assessment understand the processes the organizations followed to of the use of the GRADE-EtD framework in guideline formulate recommendations (e.g., quality of the evidence panels concluded that it was essential to structure panel assessment, COI management, voting dynamics, and EtD meetings and ensure the consideration of all relevant criteria considered). This will help them decide whether criteria [17]. In that study, formal GRADE-EtD’s criteria recommendations can be implemented in their own setting. guided 94% of panel discussions, whereas other external It is important to remark that the information extracted criteria guided the remaining 6% of the discussions (e.g., from the guidance documents was reliant on how complete clinical experience, political environment, and legal impli- the information about each criterion was presented in the doc- cations). Nevertheless, both the extent to which the criteria uments, and so was our analysis. As an example of this, stat- contained in the GRADE-EtD framework restraint discus- ing the organization used the GRADE-EtD framework did not sions among panelists, and the need for additional criteria equal the inclusion of all the recommendation-related criteria remain unexplored [17]. proposed by the framework; this was also true for other Despite the diversity of both, frameworks and processes frameworks proposed by the organizations. Organizations in this area, there is still ongoing debate and research on the may use our findings to either prepare their own guidance adequacy and validity of available frameworks for different documents or to reassess their existing ones by identifying types of decisions [26] and organizations [10]. Recently, gaps in EtD criteria or to follow suggestions for improvement. different authors have suggested the need for including In addition, guidance documents might provide clear defini- additional criteria in the GRADE-EtD framework. For tions for each EtD criterion, so that GDG members under- example, burden of treatment [5] or human rights and so- stand what evidence to look for and how to put this in a ciocultural acceptability, equality and nondiscrimination, decision-making context (e.g., considerations on equity or societal implications, and health system considerations by acceptability). All organizations should follow a more system- other authors [10]. The need for tailoring for different types atic and explicit methodology for formulating recommenda- of organizations or goals was considered in the GRADE- tions and ensure a complete reporting in their guidance EtD frameworks in the original publications. For example, documents. These might enhance transparency and credibility, ethical considerations are considered as a detailed judgment enabling end users to determine how much confidence they J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 61 can have in the recommendations; facilitate later adaptation to References contexts other than the ones where they were originally devel- [1] Alonso-Coello P, Martinez Garcia L, Carrasco JM, Sola I, Qureshi S, oped; and improve usability and communicability of the EtD Burgers JS. The updating of clinical practice guidelines: insights frameworks. from an international survey. Implementation Sci 2011;6:1e8. We systematically documented the use of EtD frame- [2] IOM (Institute of Medicine). Clinical Practice Guidelines We Can works in guidance documents; however, the extent to which Trust. Washington, DC: The National Academies Press; 2011. [3] Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, the mention of the different criteria in the guidance docu- Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development ments translates into a real use in published guidelines is of a comprehensive checklist for a successful guideline enterprise. still unknown. Besides, because both guideline developers CMAJ 2014;186(3):E123e42. and panel members have expressed favorable experiences [4] Alonso-Coello P, Sch€unemann HJ, Moberg J, Brignardello-Petersen R, with the use of EtD frameworks [12,17,24], further qualita- Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frame- works: a systematic and transparent approach to making well tive research exploring the use of the frameworks among informed healthcare choices. 1: Introduction. BMJ 2016;353:i2016. panelists and methodologist is also warranted. Our findings [5] Dobler CC, Harb N, Maguire CA, Armour CL, Coleman C, could also foster further research on the need of modifica- Murad MH. Treatment burden should be included in clinical practice tions of available frameworks. guidelines. BMJ 2018;363:k4065. [6] Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and Value: Impact on DEcisionMaking–the EVIDEM framework and potential applications. BMC Health Serv 5. Conclusion Res 2008;8:1e16. [7] World Health Organization. WHO handbook for guideline develop- The use of systematic and structured processes for moving ment. 2nd ed. Geneva: World Health Organization; 2014. from evidence to decisions is still limited among international [8] Tannahill A. Beyond evidence–to ethics: a decision-making frame- organizations devoted to guideline development. The use of work for health promotion, public health and health improvement. EtD frameworks facilitates the inclusion of relevant recom- Health Promot Int 2008;23(4):380e90. mendation criteria. Among the structured frameworks, the [9] Norheim OF, Baltussen R, Johri M, Chisholm D, Nord E, Brock D, et al. Guidance on priority setting in health care (GPS-Health): the inclusion of GRADE-EtD framework is the most widely used approach, equity criteria not captured by cost-effectiveness analysis. Cost effective- offering the most comprehensive perspective for evidence- ness and resource allocation. Cost Eff Resour Alloc 2014;12:1e8. informed decision-making processes. More complete and [10] Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, detailed reporting in the guidance documents is warranted. Baltussen R. The WHO-INTEGRATE evidence to decision frame- work version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health 2019;4(Suppl 1):e000844. [11] Morgan RL, Thayer KA, Bero L, Bruce N, Falck-Ytter Y, Ghersi D, CRediT authorship contribution statement et al. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int 2016;92-93:611e6. JFME: conceptualization, methodology, data extraction, [12] Neumann I, Brignardello-Petersen R, Wiercioch W, Carrasco- formal analysis, original draft preparation, project adminis- Labra A, Cuello C, Akl E, et al. The GRADE evidence-to-decision trator, rewriting, and editing. PAC: conceptualization, framework: a report of its testing and application in 15 international methodology, formal analysis, original draft preparation, guideline panels. Implementation Sci : IS 2016;11:1e8. [13] Rosenbaum SE, Moberg J, Glenton C, Sch€unemann HJ, Lewin S, and editing. HS and SF: methodology, supervision, orig- Akl E, et al. Developing evidence to decision frameworks and an inter- inal draft preparation, rewriting, and editing. JB, JJYN, active evidence to decision tool for making and using decisions and TPP, LP, MMB, DS, MS, JZ, CMG, YZ, and NCG: data recommendations in health care. Glob Challenges 2018;2(9):1e9. extraction, formal analysis, original draft preparation, [14] American College of Chest Physicians (CHEST). Available at https:// rewriting, and editing. All authors read, provided feedback, www.chestnet.org/Guidelines-and-Resources/About-CHEST-Guidelin es/Guideline-Development. Accessed January 15, 2021. and approved the final version of this manuscript. [15] Canadian Task Force on Preventive Health Care. Canadian Task Force on preventive health care procedure manual. 2014. Available at https://canadiantaskforce.ca/methods/. Accessed January 15, 2021. Acknowledgments [16] Centro Nazionale per l’Eccellenza Clinica la Qualita e la Sicurezza delle Cure. Manuale metodologico per la produzione di linee guida The authors thank Andrea Juliana Sanabria and Anna Sel- di pratica clinica. 2019. Available at https://snlg.iss.it/wp-content/ va for their support and assistance with the literature searches. uploads/2019/04/MM_v1.3.2_apr_2019.pdf. Accessed January 15, Besides, the authors thank the Norwegian Institute of Public 2021. [17] Li SA, Alexander PE, Reljic T, Cuker A, Nieuwlaat R, Wiercioch W, Health for their support with the publication fee for this study. et al. Evidence to Decision framework provides a structured "road- map" for making GRADE guidelines recommendations. J Clin Epi- demiol 2018;104:103e12. Appendix A [18] Higgins JPT TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interven- Supplementary data tions version 6.2. Available at www.training.cochrane.org/handbook. Accessed January 20, 2021. Supplementary data to this article can be found online at [19] Meneses-Echavez JF, Bidonde J, Yepes-Nu~nez JJ, Storman D, https://doi.org/10.1016/j.jclinepi.2022.06.004. Puljak L, Zaja˛c J, et al. Moving from evidence to decisions when 62 J.F. Meneses-Echavez et al. / Journal of Clinical Epidemiology 150 (2022) 51e62 making recommendations in healthcare: a protocol of an analysis [23] R Core Team. R. In: A language and environment for statistical of clinical practice guidelines guidance documents. Charlottes- computing. Vienna, Austria: I: R Foundation for Statistical ville, VA: Open Science Framework; 2020. Computing; 2017. [20] Vernooij RW, Sanabria AJ, Sola I, Alonso-Coello P, Martinez Garcia L. [24] Dahm P, Oxman AD, Djulbegovic B, Guyatt GH, Murad MH, Guidance for updating clinical practice guidelines: a systematic review of Amato L, et al. Stakeholders apply the GRADE evidence-to- methodological handbooks. Implementation Sci 2014;9:1e9. decision framework to facilitate coverage decisions. J Clin Epidemiol [21] Selva A, Sanabria AJ, Pequeno S, Zhang Y, Sola I, Pardo- 2017;86:129e39. Hernandez H, et al. Incorporating patients’ views in guideline [25] Meneses-Echavez JF, Rosenbaum S, Rada G, Flottorp S, Moberg J, development: a systematic review of guidance documents. J Clin Alonso-Coello P. Users’ experiences with an interactive Evidence Epidemiol 2017;88:102e12. to Decision (iEtD) framework: a qualitative analysis. BMC Med [22] Schunemann HJ, Oxman AD, Akl EA, Brozek JL, Montori VM, Inform Decis making 2021;21:1e11. Heffner J, et al. Moving from evidence to developing recommenda- [26] Morgan RL, Kelley L, Guyatt GH, Johnson A, Lavis JN. Decision- tions in guidelines: article 11 in Integrating and coordinating efforts making frameworks and considerations for informing coverage deci- in COPD guideline development. An official ATS/ERS workshop sions for healthcare interventions: a critical interpretive synthesis. J report. Proc Am Thorac Soc 2012;9(5):282e92. Clin Epidemiol 2018;94:143e50.