Florez et al. Implementation Science (2020) 15:79 https://doi.org/10.1186/s13012-020-01036-5 RESEARCH Open Access Assessment of the quality of recommendations from 161 clinical practice guidelines using the Appraisal of Guidelines for Research and Evaluation– Recommendations Excellence (AGREE-REX) instrument shows there is room for improvement Ivan D. Florez1,2* , Melissa C. Brouwers3, Kate Kerkvliet2, Karen Spithoff2, Pablo Alonso-Coello4, Jako Burgers5, Francoise Cluzeau6, Beatrice Férvers7, Ian Graham8, Jeremy Grimshaw8, Steven Hanna9, Monika Kastner10, Michelle Kho11, Amir Qaseem12 and Sharon Straus13 Abstract Objective: To assess the quality of recommendations from 161 clinical practice guidelines (CPGs) using AGREE-REX- D (Appraisal of Guidelines REsearch and Evaluation-Recommendations Excellence Draft). Design: Cross-sectional study Setting: International CPG community. Participants: Three hundred twenty-two international CPG developers, users, and researchers. Intervention: Participants were assigned to appraise one of 161 CPGs selected for the study using the AGREE-REX- D tool Main outcome measures: AGREE-REX-D scores of 161 CPGs (7-point scale, maximum 7). (Continued on next page) * Correspondence: ivan.florez@udea.edu.co 1 Department of Pediatrics, Universidad de Antioquia, Calle 67 # 53-108, 050001 Medellin, Colombia 2 Department of Oncology, McMaster University, Juravinski Site, G2 Wing, 1280 Main Street West, Hamilton, Ontario L8S 4L8, Canada Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Florez et al. Implementation Science (2020) 15:79 Page 2 of 8 (Continued from previous page) Results: Recommendations from 161 CPGs were appraised by 322 participants using the AGREE-REX-D. CPGs were developed by 67 different organizations. The total overall average score of the CPG recommendations was 4.23 (standard deviation (SD) = 1.14). AGREE-REX-D items that scored the highest were (mean; SD): evidence (5.51; 1.14), clinical relevance (5.95; SD 0.8), and patients/population relevance (4.87; SD 1.33), while the lowest scores were observed for the policy values (3.44; SD 1.53), local applicability (3,56; SD 1.47), and resources, tools, and capacity (3.49; SD 1.44) items. CPGs developed by government-supported organizations and developed in the UK and Canada had significantly higher recommendation quality scores with the AGREE-REX-D tool (p < 0.05) than their comparators. Conclusions: We found that there is significant room for improvement of some CPGs such as the considerations of patient/population values, policy values, local applicability and resources, tools, and capacity. These findings may be considered a baseline upon which to measure future improvements in the quality of CPGs. Keywords: Clinical practice guidelines, Practice guidelines, Recommendations, Quality of health care, AGREE tool, Implementability the overall methodological quality of CPGs and to serve Contributions to the literature as a methodological blueprint to inform CPG develop-  We applied the AGREE II and the recently developed tool ment and reporting [5–7]. The AGREE II focuses on the (AGREE-REX draft version), to assess quality, credibility, and entire CPG development process. As its complement, the implementability of 161 international clinical practice AGREE-REX (Appraisal of Guidelines REsearch and Evaluation-Recommendations EXcellence) was designed guidelines (CPGs). The AGREE REX draft tool was applied by to focus specifically on CPG recommendations and the 322 guidelines’ developers, users and researchers from 51 justifications that underpin them [8]. Its development countries. was in response to data demonstrating high-quality CPG  -The scores of the AGREE REX draft tool items were higher in processes, although necessary, are not always sufficient those items related to the quality of the evidence and the to yield individual CPG recommendations that are clin- clinical relevance. The items related to patients and ically credible and implementable [9, 10]. population relevance and implementation relevance scored The prototype of the AGREE-REX (the AGREE-REX- in the mid-range, while the items related to patients/popula- D) and the AGREE II was applied to 161 guidelines. In this article, we present the results of this assessment, tion or policy values, the alignment of values, the local ap- identify areas for CPG recommendation improvement, plicability, and the resouces, tools, and capacity items scored and compare the evaluative information garnered by low. both tools.  CPGs produced by government-supported organizations scored higher on all the items of the AGREE-REX draft tool Materials and methods than those produced by professional societies or other types This study represents a component of a larger program of groups, and CPGs produced in UK and Canada scored of research designed to create the AGREE-REX version higher in selected items in comparison to USA and inter- 1 (AGREE-REX-D); the technical components of this national CPGs program of research are reported elsewhere [8]. Our main study was designed to create the AGREE REX tool  The correlations between the overall AGREE-REX draft tool following a mixed-methods project, and this manuscript and AGREE II domains were low, except for the applicability presents the cross-sectional study that summarizes the domain where the correlation was modest. assessment of the selected CPGs during the development of the AGREE REX-D). This study received ethics ap- Introduction proval from the Hamilton Integrated Research Ethics Clinical practice guidelines (CPGs) are systematically de- Board (project #13-700). veloped statements informed by a systematic review of evidence and an assessment of the benefits and harms of Participants alternative care options with the aim of optimizing pa- Participants included CPG developers, clinicians, imple- tient care [1–3]. However, concerns about variation in menters, and other users. They were purposefully re- the quality of CPGs and their resultant recommenda- cruited through a variety of channels including social tions exist in the literature [1, 3, 4]. The AGREE II is an media and CPG organizations, such as the Guidelines established instrument, used internationally, to evaluate International Network (G-I-N), G-I-N North America Florez et al. Implementation Science (2020) 15:79 Page 3 of 8 regional community, Knowledge Translation (KT) Outcomes Canada, Canadian Agency for Drugs and Technologies AGREE-RE-DX scores in Health (CADTH), Canadian Partnership Against The prototype of AGREE-REX-D comprised 11 items Cancer, Cancer Care Ontario, and to investigators within 4 themes (Table 1). Each item was rated using a known in the CPG research community. The study was 7-point scale applied to two quality attributes, with also advertised on the AGREE social media accounts higher scores reflecting higher quality. The two attri- (Facebook and Twitter), and My AGREE PLUS (online butes were the following: platform for appraising CPGs with the AGREE II tool, www.agreetrust.org) registered users were invited to  Extent to which quality features were documented participate. in the CPG  Extent to which quality features were considered in formulating the recommendations. CPGs CPGs in multiple clinical specialty areas were col- The instrument concludes with two general quality as- lected from the Agency for Healthcare Research and sessments: overall credibility and overall implementabil- Quality (AHRQ) National Guidelines Clearinghouse ity of the CPG recommendations. database [11]. Using the database’s advanced search function, we identified CPGs that were (1) published AGREE II evaluations between 2013 and 2015; (2) written in English lan- For exploratory purposes, the CPGs were also assessed, guage, and (3) no more than 50 pages in length for independently, using the AGREE II by two members of the CPG core document. The resulting list of CPGs the AGREE Scientific team. The AGREE II includes 23 was reviewed and the following were excluded: guide- items within 6 domains and 2 overall assessments [5]. lines addressing organizational rather than clinical The 23 items are assessed with a 7-point scale (1 = topics, technology assessments; CPGs not available for strongly disagree; 7 = strongly agree), with high scores free to the public; and CPGs for which the link in reflecting more favorable quality results. Discrepancies the database were not functional. Descriptive informa- in scoring were resolved by consensus when required. tion was extracted from each CPG, including type of authoring organization (government supported vs. Scoring professional society vs other/not clear), disease topic For each CPG, an AGREE-REX-D item score was de- (cancer vs. non-cancer), and country of authoring rived for each of the 11 items by averaging scores on the group (USA, UK, Canada, or international). 7-point scale between the two raters. A mean overall AGREE-REX-D score was calculated for each CPG by averaging across the 11 items. Finally, mean scores for Procedure overall credibility and overall implementability items Participants received individualized password-protected were derived by averaging scores between the two raters. access to the study materials, which included links to a AGREE II tool mean domain scores were derived by downloadable PDF format of the AGREE-REX-D, the summing the scores across the two appraisers and stand- CPG to which they were randomly assigned, and the on- ardizing them as a percentage of the maximum possible line survey platform (LimeSurvey) to record their scores. score a CPG could achieve for that domain [5]. Before Participants were asked to review the AGREE-REX-D manual and items, read the CPG, and then evaluate it by Table 1 AGREE-REX applying the tool and recording their item ratings in AGREE-REX domains and items LimeSurvey. Participants were provided with no formal Domains Items training or orientation of the tool by members of the 1. Evidence justification 1. Evidence team. The AGREE-REX-D manual provided definitions 2. Clinical applicability 2. Clinical relevance of the items and instructions on how to assess and score justification 3. Relevance to patients/ them. An email reminder was sent at 2 weeks from the populations 4. Implementation relevance participant’s initial start date informing them of their 3. Values justification 5. Guideline developer values deadline in 1 week. Deadline extensions were given 6. Target user values when requested. Evaluations were completed between 7. Patient population values May 2016 and March 2017. Participants were offered a 8. Policy values 9. Alignment of values $50 CAD pre-paid virtual gift card for completing the study. All communication with participants was done by 4. Feasibility considerations 10. Local applicability 11. Resources, capacity, and tools the staff of AGREE Scientific office. Florez et al. Implementation Science (2020) 15:79 Page 4 of 8 these scores were summed and calculated, the independ- Table 2 Participants demographics (n = 322) ent appraisers were required to reach a consensus on Demographic characteristics Frequency n (%) any AGREE II item scores that were two or more points Sex apart on the 7-point scale. Female 202 (62.5) Male 115 (35.7) Sample size calculation I prefer not to disclose 5 (1.6) The sample size calculation was based on a separate Age methodological goal to conduct a reliability study of the AGREE-REX-D tool based on the interrater reliability 19 or younger 2 (0.6) outcome. Based on consensus by the team, we made the 20–29 49 (15.2) following assumptions: two raters per CPG, an intraclass 30–39 100 (31.1) correlation coefficient of 0.6, and a confidence interval 40–49 83 (25.8) from 0.5 to 0.7. We determined that we required 316 50–59 63 (19.6) participants to appraise 158 CPGs: each participatant 60–69 23 (7.1) rated one CPG using the AGREE-REX-D and each CPG 70 and older 2 (0.6) was rated by two independent raters. Additional infor- Experience with AGREE II mation on the details of the sample size calculation can be found elsewhere [8] No experience 70 (21.7) Some experience 122 (37.9) Analytical framework Experienced 88 (27.3) Descriptive measurements were used to summarize the Very experienced 42 (13) AGREE-REX item and overall scores. A series of one- Geographic location way ANOVA tests was used to examine mean differ- North America 177 (55) ences in the AGREE-REX-D item scores and the overall Europe 76 (23.6) score as a function of the following characteristics: type Asia 24 (7.5) of authoring organization (government-supported vs. professional societies vs. other), disease topic (cancer vs. South America 32 (9.9) not cancer), and country of development (USA vs. UK Africa 7 (2.2) vs. Canada vs. international). International guidelines Oceania 6 (1.9) category included guidelines co-developed by two or First language more countries or developed by international organiza- English 188 (58.4) tions or societies. Descriptive measures were used to Spanish 51 (15.8) summarize AGREE II domain scores. Finally, correla- Italian 14 (4.3) tions between mean overall AGREE-REX-D scores and Chinese 13 (4) AGREE II domain scores were calculated. Analyses were performed using Stata 15.0 (StataCorp. 2017. Stata Stat- Dutch 10 (3.1) istical Software: Release 15. College Station, TX: Stata- Portuguese 7 (2.2) Corp LLC). French 4 (1.2) Greek 3 (0.9) Results Ukrainian 3 (0.9) Participants Other 29 (9) Descriptive statistics of the participants are listed in a Participants’ roles with clinical practice guidelines (PG) Tables 2 and 3. CPG developer—clinical expert 85 (26.4) CPGs CPG developer—patient/public representative 15 (4.7) We appraised 161 CPGs. The CPGs targeted a range of CPG developer—methodologist 170 (52.8) diseases and clinical problems including cancer, CPG user—health care provider 102 (31.7) infectious diseases, pregnancy and child birth, mental CPG user—administrator/policy maker/manager 38 (11.8) health, nervous system disorders, respiratory, digestive, CPG user—patient/member of the public 20 (6.2) genitourinary, blood and endocrine disorders, and mus- Researcher 159 (49.4) culoskeletal, among others. With the exception of cancer Other (e.g., librarian, student) 25 (7.8) (n = 38), the number of CPGs for each unique disease a Participants could select more than one role was small (< 8) making other comparisons by disease topic not viable. CPGs were developed by 67 different Florez et al. Implementation Science (2020) 15:79 Page 5 of 8 Table 3 Comparison of mean overall AGREE-REX scores by AGREE-REX-D performance by country of CPG authoring participant demographic feature group Participant demographic t test statistica p value The country of the authoring CPG organization showed Experience vs. no experience 4.04 .001 differences in AGREE-REX quality scores as well. North America vs. all other regions 2.86 .004 Statistically significant differences (i.e., p < 0.05) for five AGREE-REX items (implementation relevance, target English vs. non-English 1.056 .290 user values, policy values, local applicability, and re- PG developers Vs. PG users and researchers − 2.29 .023 sources, tools, and capacity), and the mean overall a Equal variances assumed AGREE-REX score were found. Differences as a function of authoring group approached, but did not reach, statis- international organizations (see Additional file 1 tical significance for the overall implementability assess- Appendix 1). Organizations that produced the CPGs ment. For each of these comparisons, the CPGs were government-supported in less than a third of produced in the UK and Canada showed higher scores. cases (n = 46; 28.6%), and they were authored by The item scores of CPGs published from the UK ranged groups most often located in USA (n = 89; 55.3%) or from 3.66 (SD 1.26) to 5.74 (SD 0.90); from Canada the UK (n = 46; 28.6%). CPGs were all published be- ranged from 3.42 (SD 1.0) to 5.87 (SD 0.64); from the tween 2013 and 2015. The list of appraised CPGs can USA ranged from 3.08 (SD 1.47) to 5.06 (SD 1.39); and be accessed in the supplementary file. from international organizations ranged from 2.96 (SD 1.39) to 5.18 (SD 1.44)). In all but one case, overall AGREE-REX-D item means straddled the mid-point of AGREE-REX (see Table 4) the scale where there was a significant difference be- AGREE-REX.D performance for all CPGs tween the groups. The mean overall AGREE-REX score across the 161 CPGs was 4.23 (SD 1.14). There was variability in per- AGREE-REX-D performance by disease formance across the individual 11 items, with 6 that No significant differences emerged between cancer and scored above the middle point of 4.0 on the response non-cancer CPGs scores; this held true for each of the scale. The mean overall credibility and overall imple- AGREE-REX items and the mean overall AGREE-REX- mentability assessments were 4.78 (SD 1.24) and 4.19 D score (p > 0.5; means not presented). (SD 1.23), respectively. AGREE II (see Table 5) The AGREE II domain scores for the CPGs are displayed AGREE-REX-D performance by type of organization in the Table 5. Scope and purpose, and clarity of presen- Statistically significant differences (i.e., p < 0.05) were tation were the domains with the highest scores, while found as a function of organization type for each of the the applicability domain had the lowest score. mean AGREE-REX-D items, the mean overall AGREE- REX-D score, and the overall implementability and AGREE II and AGREE-REX overall credibility assessments. In each case, more favor- The correlations between the overall AGREE-REX-D able ratings were found among CPGs produced by and AGREE II domains were low (r < 0.30) except for government-supported organizations. The item scores of the applicability domain where the correlation was mod- CPGs produced by government-supported organizations est at r = 0.38 [8]. Overall, AGREE-REX scores were (n = 46) ranged from 4.41 (SD 1.11) to 5.95 (SD 0.8); the higher among appraisers with no AGREE II experience scores of CPG produced by professional societies (n = compared to those with AGREE II experience. 109) ranged from 2.99 (SD 1.46) to 5.24 (SD 1.26); and the scores of CPG produced by other types of organiza- Discussion tions (n = 6), ranged from 3.00 (SD 0.89) to 6.17 (SD We appraised 161 CPGs with the prototype of the 0.68). Of note, in 5 of the 11 cases, the AGREE-REX-D AGREE-REX-D tool and the AGREE II tool. The most item means across the organization types fell within the favorable AGREE-REX ratings (means > 5.0) were found positive ends of the response scale (m ≥ 4) despite there for the evidence and clinical relevance items; ratings that being statistically significant differences between them. fell in the more moderate range of the scale (means > In contrast, in 6 of the 11 cases, the overall means of the 4.0 and < 5.0) were found for the patient/population AGREE-REX-D items straddled the mid-point of the relevance, implementation relevance, developers’ values scale—suggesting some organizations tended to perform and users’ values items; and least favorable ratings that lower than the mid-point and others perform higher fell below the mid-point of the scale (means < 4.0) were than the mid-point of the scale. found for patients/population values, policy values, Table 4 aAGREE-REX Scores For Specific Items, Overall Items (across the 11 Items), and Overall Assessments Practice Specific Items Means Overall Means Overall Assessments # Florez et al. Implementation Science Guidelines (PG) PGs Evidence Clinical Patient/ Implement- Values Values Values Values Alignment Local Resources, Mean % Clinical Implement- Features Relevance pop. ation Developers Users Patient/ Policy of Values Applicability Tools, Overall Score Credibility ability Relevance Relevance Pop Capacity Score Mean All Guidelines 5.15 5.47 4.87 4.46 4.61 4.28 3.85 3.44 3.42 3.56 3.49 4.23 54 4.78 4.19 161 (1.33) (1.18) (1.33) (1.52) (1.45) (1.52) (1.61) (1.53) (1.44) (1.47) (1.44) (1.14) (19) (1.24) (1.23) (2020) 15:79 Type of Organization Government 5.61 5.95 5.57 5.48 5.26 4.97 4.83 4.56 4.11 4.52 4.41 5.02 67 5.28 4.85 46 Supported (1.14) (0.8) (0.98) (0.89) (1.2) (1.14) (1.34) (1.15) (1.32) (1.3) (1.11) (0.77) (13) (0.96) (1.07) Professional 4.98 5.24 4.57 4.04 4.30 3.98 3.41 2.99 3.1 3.17 3.06 3.89 48 4.56 3.89 109 Societies (1.35) (1.26) (1.36) (1.55) (1.48) (1.58) (1.52) (1.46) (1.38) (1.39) (1.39) (1.2) (19) (1.31) (1.21) Other or Not 4.83 6.17 5.00 4.42 5.17 4.5 4.42 3.00 3.75 3.33 4.25 4.44 57 4.83 4.50 6 Clear (1.63) (0.68) (1.41) (1.24) (0.93) (1.38) (2.04) (0.89) (1.63) (0.82) (1.29) (0.99) (17) (0.82) (0) p-value 0.021 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 0.004 <0.001 Country US 5.06 5.33 4.68 4.17 4.51 4.12 3.61 3.08 3.31 3.26 3.22 4.03 51 4.62 3.98 89 (1.39) (1.28) (1.4) (1.58) (1.49) (1.62) (1.66) (1.47) (1.54) (1.47) (1.47) (1.21) (20) (1.31) (1.26) UK 5.26 5.74 5.22 5.04 4.86 4.76 4.23 4.16 3.66 4.18 4.12 4.66 61 5.06 4.51 46 (1.23) (0.9) (1.13) (1.25) (1.39) (1.21) (1.31) (1.41) (1.26) (1.48) (1.41) (0.95) (16) (1.06) (1.19) Canada 5.46 5.87 5.17 4.75 4.67 4.5 4.46 3.87 3.79 3.83 3.42 4.53 59 4.92 4.50 12 (1.21) (0.64) (1.19) (1.5) (1.51) (1.58) (2) (1.61) (1.48) (1.27) (1) (1.08) (18) (1.12) (1.1) International 5.18 5.18 4.64 4.18 4.39 3.54 3.64 2.96 2.96 3.18 3.25 3.91 49 4.75 4.21 14 (1.38) (1.44) (1.42) (1.47) (1.47) (1.31) (1.67) (1.39) (1.18) (0.99) (1.14) (0.98) (16) (1.38) (1.03) p-value 0.7 0.1146 0.1192 0.011 0.5495 0.025 0.0932 <0.001 0.2609 0.0036 0.0056 0.0106 0.0106 0.2649 0.0851 a Mean (SD); US United States, UK United Kingdom, Implement implementability, Patient/Pop Patients/population Page 6 of 8 Florez et al. Implementation Science (2020) 15:79 Page 7 of 8 Table 5 Average AGREE II Domain Scores (n = 161 PGs) development groups. These data align with initial ap- AGREE II Domains Mean SD Min Max praisal findings using the original AGREE instrument, in Scope and Purpose 75.3 14.4 33 100 which CPGs developed by government-supported orga- Stakeholder Involvement 56.0 16.4 19 86 nizations also had the most favorable quality scores [15]. CPG panels with more resources (financial and access to Rigour of Development 56.6 16.6 10 91 skilled methodologists) confer quality benefits and set- Clarity and Presentation 81.2 12.3 33 100 ting quality standards too high may have the unintended Applicability 36.5 21.4 0 94 consequence of increasing the disparities between the Editorial Independence 57.4 24.9 0 100 “have much” and “have less” jurdisdictions. Similar dif- ferences and similar concerns were raised in the assess- alignment of values, local applicability and resources, ment of CPGs with the original AGREE instrument [15]. tools and capacity items. CPGs produced by Our study has several limitations. First, we only included government-supported organizations scored higher on English-language CPGs in the analysis. As a result, we all the items of the AGREE-REX-D than those produced have no data on the unique strengths or limitations re- by professional societies or other types of groups, and lated to credibility and implementability of non-English CPGs produced in UK and Canada scored higher in se- CPGs. This provides an opportunity for future research lected items in comparison to USA and international studies. Additionally, in order to optimize the feasibility of CPGs. The confidence intervals around the mean the study and candidates’ interests to participate, we only AGREE-REX scores were large. included CPGs that were less than 50 pages in length (ex- The distribution of the mean scores across the 11 cluding appendices and tables). Although the length of the items is not surprising. CPG methods research has fo- CPG document is not necessarily associated with the qual- cused largely on issues directly relevant to creating the ity, credibility and implementability, the restriction we im- evidence base. As a consequence, some AGREE-REX posed may have resulted in the exclusion of lengthy CPGs concepts are easier to achieve success because there ex- that may have more information and perhaps could have ists tools and resources to support thier operationaliza- been scored higher. In addition, while 161 CPGs were tion (e.g., tools designed by the GRADE working group evaluated, they were not from 161 unique developers. This [12]). In contrast, resources to operaitonalize other con- could potentially be a source of confounding. Finally, the cepts are more elusive. For example, continued meth- penultimate prototype of the AGREE-REX-D was used odological development is needed to adequately measure and not the final version. While there is considerable over- and report values across diverse stakeholder groups so lap between the two, future status reports must account that they are reliable, valid, and usable. Similarly, system- for these differences when reflecting on changes in scores atic strategies to incorporate these perspectives into the over time. framing of recommendations are required [13]. As previously reported with the evaluation of the Conclusion AGREE II [14], lower scores with some AGREE-REX-D As part of the development of the AGREE-REX tool, we items may reflect inadequate reporting and not poor assessed 161 CPG recommendations from different or- quality in methodological execution [6]. Developers may ganizations around the world using the draft version of have followed appropriate steps but not reported them the tool. We found that there is significant room for im- in the CPG documentation and, as a consequence, could provement in some CPG recommendation elements. not be assessed. Also, it is possible that some conceptual The most unfavorable ratings were found in the follow- elements reflected in the AGREE-REX-D (e.g., concepts ing items: patients/population values, policy values, related to implementation activities) are not the respon- alignment of values, local applicability and resources, sibility of the CPG developer directly, but perhaps by an- tools and capacity. It should also be noted that statisti- other party or group within their specific settings [12]. cally significant higher scores were found in guidelines Thus, the AGREE-REX could provide a signal to individ- developed by government-supported organizations (in uals who are ultimately responsible for action about comparison to those produced by professional or spe- where gaps and barriers to this goal exist so that correct- cialist societies or others), and in guidelines developed in ive action can be taken. the UK and Canada (in comparison to those produced Differences in mean overall AGREE-REX-D scores as a in the USA and internationally. function of the type of organization may reflect the Since the AGREE-REX can be used as a methodo- greater interest or great capacity of government- logical blueprint to inform the development and report- supported organizations to seek out a broader range of ing of high-quality recommendations, our findings may values or invest in additional methodological steps that be used as a baseline upon which to measure future im- lead to higher quality scores than do other types of provements in the quality of CPG recommendations. Florez et al. Implementation Science (2020) 15:79 Page 8 of 8 Supplementary information McMaster University, 2C Health Sciences Centre, 1280 Main Street West, Supplementary information accompanies this paper at https://doi.org/10. Hamilton, Ontario L8S 4L8, Canada. 10North York General Hospital, 4001 1186/s13012-020-01036-5. Leslie Street, Toronto, Ontario M2K 1E1, Canada. 11Institute of Applied Health Sciences McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4L8, Canada. 12American College of Physicians, 190 N Independence Mall Additional file 1. List of development organizations of clinical practice West, Philadelphia, PA 19106-1572, USA. 13Li Ka Shing Knowledge Institute of guidelines assessed. St. Michael’s Hospital, 30 Bond Street, Shuter 2-026, Toronto, Ontario M5B 1W8, Canada. Acknowledgements The authors thank the following AGREE-REX Research Team members and Received: 30 May 2020 Accepted: 26 August 2020 collaborators for their input: Onil Bhattacharyya; George Browman; Anna Gagliardi; Peter Littlejohns; Holger Schunemann; and Louise Zitzelsberger. The authors also thank the study participants. References 1. Shiffman RN, Shekelle P, Overhage JM, et al. Standardized reporting of Transparency clinical practice guidelines: a proposal from the conference on guideline The lead author affirms that this manuscript is an honest, accurate, and standardization. Ann Intern Med. 2003;139(6):493–8. transparent account of the study being reported; that no important aspects 2. Qaseem A, Forland F, Macbeth F, et al. Guidelines international network: of the study have been omitted; and that any discrepancies from the study toward international standards for clinical practice guidelines. Ann Intern as planned (and, if relevant, registered) have been explained. Med. 2012;156:525–31. 3. Institute of Medicine. 2011. Clinical practice guidelines we can trust. Authors’ contributions Washington, DC: The National Academies Press. https://doi.org/https://doi. IDF was involved in project conception and design, led data analysis, org/10.17226/13058 (). interpretation of results, and manuscript writing. MCB was involved in 4. Grilli R, Magrini N, Penna A, et al. Practice guidelines developed by specialty project conception and design, interpretation of the results, and critical societies: the need for critical appraisal. Lancet. 2000;355:103–6.5. review of the manuscript. KS and KK were involved in project conception 5. Brouwers M, Kho ME, Browman GP, et al. AGREE II: advancing guideline and design, data collection and analysis, interpretation of the results and development, reporting and evaluation in healthcare. CMAJ. 2010;182: manuscript writing. AQ, JB, FC, BF, MK, SH, MK, JG, IG, PAC, and SS were E839–42. involved in project conception and design, and critical review of the 6. Brouwers MC, Kho ME, Browman GP, et al. Performance, usefulness and manuscript. All authors revised and provided approval of the submitted areas for improvement: development steps towards the AGREE II - part 1. manuscript and agree to act as guarantors of the work. CMAJ. 2010;182:1045–52. 7. Brouwers MC, Kho ME, Browman GP, et al. Validity assessment of items and Funding tools to support application: development steps towards the AGREE II – This project was funded by the Canadian Institutes of Health Research (CIHR), part 2. CMAJ. 2010;182:E472–8. grant #201209MOP-285689-KTR-CEBA-40598. JG holds a Canada Research 8. Brouwers MC, Spithoff K, Kerkvliet K, Alonso-Coello P, Burgers J, Cluzeau F, Chair in Health Knowledge Transfer and Uptake. The funding body did not et al. Development and validation of a tool to assess the quality of clinical influence the design of the study, the collection, analysis, and interpretation practice guideline recommendations. JAMA Netw Open. 2020;3(5):e205535. of the data, or the writing of the manuscript. 9. Nuckols TK, Lim YW, Wynn BO, et al. Rigorous development does not ensure that guidelines are acceptable to a panel of knowledgeable providers. J Gen Intern Med. 2008;23:37–44. Availability of data and materials 10. Watine J, Friedberg B, Nagy E, et al. Conflict between guideline The analyses are available from the corresponding author. methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006;52:65–72. Ethics approval and consent to participate 11. The Agency for Healthcare Research and Quality (AHRQ) National CPG This study has been approved by the Hamilton Integrated Research Ethics Clearinghouse database. https://www.guideline.gov. Accessed 20 May 2013. Board (project number: 13-700). 12. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical Consent for publication research ed). 2008;336(7650):924–6. Not applicable 13. Brouwers MC, Makarski J, Kastner M, on behalf of the GUIDE-M Research Team, et al. The Guideline Implementability Decision Excellence Model Competing interests (GUIDE-M): a mixed methods approach to create an international resource All authors have completed the ICMJE uniform disclosure form at www. to advance the practice guideline field. Implement Sci. 2015;10(1):36. icmje.org/coi_disclosure.pdf and declare: KS and KK had financial support 14. Fervers B, Burgers JS, Haugh MC, et al. Predictors of high quality clinical from the CIHR grant that funded the submitted work; no financial practice guidelines: examples in oncology. Int J Qual Health Care. 2005; relationships with any organizations that might have an interest in the 17(2):123–32. submitted work in the previous 3 years; MCB is the grant holder of the CIHR 15. Burgers JS, Cluzeau FA, Hanna SE, et al. Characteristics of high-quality funding that supported this work; no other relationships or activities that guidelines: evaluation of 86 clinical guidelines developed in ten European could appear to have influenced the submitted work. countries and Canada. Int J Technol Assess Health Care. 2003;19(1):148–57. Author details 1 Department of Pediatrics, Universidad de Antioquia, Calle 67 # 53-108, Publisher’s Note 050001 Medellin, Colombia. 2Department of Oncology, McMaster University, Springer Nature remains neutral with regard to jurisdictional claims in Juravinski Site, G2 Wing, 1280 Main Street West, Hamilton, Ontario L8S 4L8, published maps and institutional affiliations. Canada. 3School of Epidemiology and Public Health, University of Ottawa, 101F, 600 Peter Morand, Ottawa, Ontario K1G 5Z3, Canada. 4Iberoamerican Cochrane Centre, Hospital de la Santa Creu i Sant Pau, C/Sant Antoni M. Claret 167, Pavelló 18 planta 0, 08025 Barcelona, Spain. 5Dutch College of General Practitioners, PO Box 3231, Utrecht, The Netherlands. 6Imperial College London, St. Mary’s Hospital (Room 1070, Queen Elizabeth the Mother Wing), Praed Street, London W2 1NY, UK. 7Département Cancer et Environnement, Centre Léon Bérard, 28 rue Laënnec, 69373, Cedex 08 Lyon, France. 8Ottawa Hospital Research Institute, University of Ottawa, 501 Smyth Road, Box 711, Ottawa, Ontario K1H 8L6, Canada. 9Faculty of Health Sciences,