BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Original research Which actionable statements qualify as good practice statements In Covid-­19 guidelines? A systematic appraisal Omar Dewidar ‍ ‍,1,2 Tamara Lotfi,3,4,5 Miranda Langendam,6 Elena Parmelli,7 Zuleika Saz Parkinson,8 Karla Solo ‍ ‍,3,4,5 Derek K Chu ‍ ‍,3,9 Joseph L Mathew,10 Elie A Akl,3,11 Romina Brignardello-­Petersen,3,4 Reem A Mustafa,3,12 Lorenzo Moja,13 Alfonso Iorio,3,4,9 Yuan Chi,14,15 Carlos Canelo-­Aybar,16 Tamara Kredo ‍ ‍,17,18 Justine Karpusheff,19 Alexis F Turgeon,20,21 Pablo Alonso-­Coello,16 Wojtek Wiercioch,3,4,5 Annette Gerritsen,17 Miloslav Klugar ‍ ‍,22 María Ximena Rojas,23 Peter Tugwell,24,25 Vivian Andrea Welch ‍ ‍,1,2 Kevin Pottie,26 Zachary Munn,27 10.1136/bmjebm-2021-111866 Robby Nieuwlaat,3 Nathan Ford,28 Adrienne Stevens,3,4 Joanne Khabsa ‍ ‍,29 Zil Nasir,3,4 Grigorios I Leontiadis,4,9 ►► Additional supplemental material is published online Joerg J Meerpohl,30,31 Thomas Piggott ‍ ‍,3,4,5 Amir Qaseem,32 only. To view, please visit the journal online (http://​ Micayla Matthews,3,4,5 Holger J Schünemann ‍ ‍,3,4,5,9,33,34 the dx.d ​ oi.​org/​10.​1136/​ eCOVID-­19 recommendations map collaborators bmjebm-​2021-​111866). For numbered affiliations see Abstract end of article. Summary box Objectives To evaluate the development and quality of actionable statements that qualify What is already known about this Correspondence to: as good practice statements (GPS) reported in subject? Prof Holger J Schünemann, COVID-­19 guidelines. ⇒⇒ Good practice statements (GPS) McMaster University Design and setting Systematic review . We (ie, actionable statements about Department of Health searched MEDLINE, MedSci, China National Research Methods Evidence interventions that would do Knowledge Infrastructure (CNKI), databases and Impact, Hamilton, substantially more good than harm of Grading of Recommendations Assessment, Canada; s​ chuneh@​mcmaster.​ or vice versa) do not qualify for Development and Evaluation (GRADE) Guidelines, ca rating the certainty of evidence, NICE, WHO and Guidelines International Network but are important statements in (GIN) from March 2020 to September 2021. We guidelines. The GRADE Working Group included original or adapted recommendations developed five criteria to assess the ►► http://​dx.​doi.​org/​10.​1136/​ addressing any COVID-­19 topic. appropriateness of issuing a GPS. bmjebm-​2022-​111962 Main outcome measures We used GRADE Working Group criteria for assessing the appropriateness of issuing a GPS: (1) clear but did not meet GPS criteria. Of the 2375 and actionable; (2) rationale necessitating the GPS, 85% were clear and actionable; 59% © Author(s) (or their message for healthcare practice; (3) practicality of provided a rationale necessitating the message employer(s)) 2022. Re-­use systematically searching for evidence; (4) likely for healthcare practice, 24% reported the net permitted under CC net positive consequences from implementing the positive consequences from implementing the BY-­NC. No commercial GPS and (5) clear link to the indirect evidence. GPS. Systematic collection of evidence was re-­use. See rights and We assessed guideline quality using the Appraisal deemed impractical for 13% of the GPS, and permissions. Published of Guidelines for Research and Evaluation II tool. 39% explained the chain of indirect evidence by BMJ. Results 253 guidelines from 44 professional supporting GPS development. 173/2375 (7.3%) To cite: Dewidar O, Lotfi T, societies issued 3726 actionable statements. We statements explicitly satisfied all five criteria. The Langendam M, et al. BMJ classified 2375 (64%) as GPS; of which 27 (1%) guidelines’ overall quality was poor regardless Evidence-B­ ased Medicine were labelled as GPS by guideline developers. of the appropriateness of GPS development and 2022;27:361–369. 5 (19%) were labelled as GPS by their authors labelling. BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | 361 BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine The prevalence and quality of GPS in guideline documents has Summary box not been empirically evaluated, particularly during the current COVID-­ 19 pandemic where healthcare professionals, scientific What are the new findings? societies and government agencies invested a substantial amount ⇒⇒ Statements that qualify as GPS constitute more of time and resources in developing clinical practice guidelines to than half of the actionable statements in COVID-­19 reduce information gaps and improve patient outcomes. Further- guidelines; there was rarely any appropriate more, the application of the GRADE criteria for GPS have neither labelling and a lack of transparency in the rationale been operationalised as guidance for those evaluating guidelines for their development. nor for developers of GPS. During the development of the global How might it impact clinical practice in the foreseeable living map of COVID-­19 recommendations and portal for contex- future? tualisation (eCOVID-­19RecMap)14 15 (https://COVID-19.recmap.​ ⇒⇒ We provide a structured framework for GPS org), we identified and evaluated GPS for their appropriateness evaluation. Utilisation of this framework by for development to inform clinical practice. researchers will help monitor the progress around GPS development and evaluate potential barriers Methods slowing the uptake of available guidance by Search guideline developers. We systematically searched MEDLINE (PubMed) from 1 March 2020 to 24 September 2021 using a search string: ((practice guide- Conclusions Statements that qualify as GPS are common in line[PT]) OR (practice guidelines as topic*[MH])) NOT (comment[pt] COVID-­19 guidelines but are characterised by unclear designation or editorial[pt] or letter[pt] or interview[pt] or case reports[pt] or and development processes, and methodological weaknesses. news[pt]), with no restrictions on the language of publication, as part of work to build the eCOVID-­19RecMap.15 We searched ECRI Clinical Guidelines, International Database of GRADE Guidelines (BIGG database), National Institute for Health and Care Excel- Introduction lence (NICE), the World Health Organization (WHO), Centers for Several formal approaches have emerged to structure the process Disease Control and Prevention (US CDC) and Guidelines Interna- of developing health recommendations in guidelines.1 Within tional Network (GIN)’s libraries using an automated web scraping guidelines, there are a variety of actionable statements for appli- approach via Application Process Interfaces (API). We also manu- cation by clinicians, consumers and other stakeholders.2 These ally searched MedSci and China National Knowledge Infrastruc- actionable statement can be further broken down into the catego- ture (CNKI) databases to identify Chinese guidelines. ries of formal recommendations, informal recommendations and Additionally, we manually searched websites of the following good practice statements (GPSs). Formal recommendations use the guideline organisations: Public Health Agency of Canada (PHAC), best available evidence and should be developed based on trans- Scottish Intercollegiate Guidelines Network (SIGN), Canadian Task parent and trustworthy methods.3–6 Such recommendations are Force on Preventive Health Care (CTFPHC), European Centres for the central aim of guideline development. Informal recommenda- Disease Control and Prevention (ECDC). We also contacted guide- tions resemble formal recommendations but they lack reporting or lines developers of all the above organisations to keep us apprised use of rigorous guideline development methods. GPSs, sometimes of any new or updated guidelines.15 referred to as best practice statements, form a separate category of actionable statements that are considered important to issue for healthcare practice.2 GPSs differ from formal and informal recom- Identifying COVID-19 guidelines mendations as they are not typically based on systematic reviews We included guidelines eligible for the eCOVID-­19RecMap with of the evidence and do not include a rating of the certainty of the most recent guideline uploaded on 24 September 2021. evidence using approaches such as Grading of Recommendations These guidelines reported original or adapted recommendations Assessment, Development and Evaluation (GRADE).7 8 The GRADE and were consistent with the WHO definition of practice guide- approach is the most widely used tool for guideline developers to lines while addressing any topic regarding patients at risk for or assess the certainty in effect estimates and subsequently trans- infected with COVID-­19.16 Online supplemental table S1 describes lating the evidence into recommendations using a standardised the definition in detail. We selected guidelines for the eCOVID-­ and transparent evidence to decision framework.7 9 10 19RecMap based on a prioritisation process developed within the Due to the lack of international consensus guidance for eCOVID-­ 19RecMap executive research team (https://COVID-19.​ GPS development and reporting, they are commonly confused recmap.org/about). A topic is a priority if it satisfies one of the with other GRADEd recommendations. For example, GPSs are following in COVID-­19 context1: arises commonly in practice,2 frequently reported as strong recommendations with low or uncertainty in practice,3 new evidence to consider,4 existence of very low-­quality evidence.11–13 To clarify this confusion, GRADE variations in practice,5 important consequences for high resource proposed the following five criteria to assess the appropriateness use/cost,6 not adequately addressed in existing guidelines.17 The of issuing a recommendation as a GPS and differentiate them priority list was refined weekly according to the climate of the from GRADEd recommendations8 : (1) statement is clear and pandemic at the current point in time. actionable, (2) message is necessary regarding healthcare practice, We did not restrict guideline eligibility by population group, (3) implementation of the statement likely to result in large net organisation, country, guideline quality or language. However, positive consequences, (4) summarisation of evidence would be we only extracted and evaluated non-­ English guidelines that poor use of guideline panel’s time and (5) the rationale connecting could be translated to English by members of our multinational the indirect evidence used to support the statement is clear and team. For guidelines with more than one version, we evaluated explicit. the most recent update. Guideline eligibility was determined by 362 BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Table 1 GRADE criteria for evaluating GPS modified from reference8* Signalling question* Description Is the statement clear and actionable? Specific statement that includes the specification of the population of interest. Is the message really necessary in regard to actual healthcare practice? Without the guidance provided by the statement, clinicians might fail to take the appropriate action. Knowledge of that practice among the clinicians who represent the target audience is suboptimal. After consideration of all relevant outcomes and potential downstream Certainty of benefits and harms are great; the values and preferences consequences, implementing the good practice statement results in a are clear; the intervention is cost saving; and the intervention is clearly large net positive consequence? acceptable, feasible and promotes equity. Is collecting and summarising the evidence a poor use of a guideline Poor use of a guideline panel’s time and resources to collect and link the panel’s limited time, energy, or resources (opportunity cost is large)? indirect evidence is an issue of opportunity cost and their time and energy better spent on other efforts to maximise the guideline’s methodologic quality and over-­all trustworthiness. Is there a well-­documented clear and explicit rationale connecting the The rationale should include an explicit statement of the chain of evidence indirect evidence? that supports the recommendation. *The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group developed these criteria for guideline developers (to designate GPS in their guidelines) and those evaluating the appropriateness of GPS. All five criteria should be fulfilled to designate a statement as GPS. GPS, good practice statement. two researchers independently, with consensus or arbitration for a explanations and signalling questions in table 2 and reordered final decision if needed. the original GRADE criteria for the purpose of critical appraisal of GPS. We conducted all the evaluations in duplicate, and an expert Identifying actionable statements that qualify as GPS We identified actionable statements from the included guidelines using the framework proposed by Lotfi et al.2 In brief, state- ments that are actionable in isolation with an expected large net Table 2 Characteristics of included guidelines and good practice benefit, not GRADEd for strength or the certainty of evidence statements or accompanied by a citation for supporting evidence and the N (%) alternative of the stated statement were judged as illogical or did Guideline Source (n=200 guidelines) not conform with ethical norms were qualified as GPS.2 Addition-  WHO 128 (64) ally, researchers extracted statements in the guidelines labelled  Centers for Disease Control and Prevention 25 (13) as best practice or GPSs. We used this approach to identify GPS  Public Health Agency of Canada 12 (6) because there is no universally accepted approach for presenting  European Centre for Disease Prevention and Control 10 (5) GPS in guidelines and they are often inconsistently labelled.13 18  National Institute for Health and Care Excellence 2 (1) Two researchers extracted the statements and experts in guideline  Scottish Intercollegiate Guidelines Network 2 (1) development reviewed them as a quality control step. In addi-  Other 21 (11) tion, we extracted the source, topic (eg, infection prevention and Field (n=200 guidelines)  Public health 160 (80) control, vaccination) and intended user and applicable context of  Health policy and systems 88 (44) each guideline.  Clinical practice 69 (35)  Health technology assessment 3 (2) Evaluating GPS World region (n=200 guidelines) We compared the appropriateness of issuing the GPS labelled by  Global 100 (50) guideline developers with statements that qualified as GPS using  North America 43 (22) the five GRADE criteria in table 1.8 We piloted a form using answer  Europe-­Central Asia 41 (21) options of ‘yes’, ‘probably yes’, ‘probably no’ and ‘no’ and devel-  East-­Asian Pacific 11 (6) oped instructions for how to use the form (online supplemental  South Asia 3 (2) figure S1). Trained methodologists held weekly meetings to opti-  Middle East-­North Africa 2 (1) mise these judgements by discussing examples from guidelines. Recommendation Topic (n=2375 statements)  Infection Control 940 (40) We used the following approach for the judgements: researchers  Vaccination 451 (19) selected ‘yes’ and ‘no’ answers when information supporting or  Health services and systems 446 (19) opposing the qualification of the statement as GPS, respectively,  Planning and monitoring 309 (13) was explicit in the guideline (any primary document or supple-  Treatment and rehabilitation 126 (3) ments). We selected ‘Probably yes’ and ‘Probably no’ when the  Diagnosis 52 (2) information supporting or opposing the qualification of the state-  Screening 51 (2) ment as GPS was implicit, respectively. For the statement to fulfil Target users (n=2375 statements) the GPS criteria, all the criteria ii–v must be answered ‘probably  Healthcare providers and professionals 894 (38) yes’ or ‘yes’. We did not include criterion i as part of the assess-  Public health officials 845 (36) ment for appropriates of issuing the statement as GPS since it is a  General population 321 (14) requirement for any recommendation.8 Online supplemental table  School administrations 258 (11)  Government 57 (2) S3 presents examples of GPS. We then iteratively developed the BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | 363 BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine in guideline development validated them. We resolved disagree- (64%) statements that qualified as GPS in 200 of 253 (79%) guide- ments by consensus in weekly group discussions. lines included on the eCOVID-­ 19RecMap (online supplemental table S2). Those 200 guidelines were included in our analysis. On Guidelines quality appraisal average, 82% of the statements per guideline (range from 2% to To evaluate if the guidelines were developed with rigorous 100%) qualified as GPS. methods, we critically appraised their development process using the Appraisal of Guidelines for Research and Evaluation (AGREE) Characteristics of GPs II tool for three out of six domains that were deemed important for Table 3 shows that 64% of the guidelines were published by WHO guideline credibility: scope and purpose, rigour of development and 13% by the CDC. One hundred and sixty (80%) guidelines were and editorial independence.19 The other AGREE domains (stake- in the field of public health and 50% were produced for global holder involvement, clarity of presentation domain and appli- use. Forty per cent of the GPS provided guidance on infection cability) were not included in the evaluation as they are not as control while the remaining were on a variety of topics including critical for determining the overall quality of the guideline. Two vaccination, planning and monitoring health services, screening, researchers independently conducted the evaluations of the guide- diagnosis and treatment. The GPS targeted a range of users: 38% lines and a guideline development expert subsequently reviewed were nominally intended for healthcare providers and profes- them. The scores of each domain item were assessed on a seven-­ sionals and 36% targeted public health officials. The remaining point scale; 0% if each reviewer scored a 1 (minimum value) and GPSs were intended to be used by individuals outside the health- 100% for a score of 7 (maximum value) by both reviewers. We care setting, patients, caregivers and the public. One guideline was identified discrepancies when a difference of 3 points or more per translated from French to English while the remaining guidelines item between the reviewers was found. We resolved these discrep- were published in English. ancies by consensus or a third reviewer. The final score per item was calculated as the average of scores between reviewers after Issuing GPS according to guideline characteristics and statement resolution of discrepancies if any. We extracted the information topic from the guidelines into the GRADEpro (www.gradepro.org) app Figure 2 presents the associations between issuing GPS based on through a new module that allows the creation of GPS. We then the guideline organisation, field, region, and recommendation included the GPS in the RecMap (https://covid19.recmap.org/​ topic. Guidelines published in the field of clinical practice were recommendations?recommendationFormality=gps). less likely to publish statements that qualify as GPS as compared with formal/informal recommendations, while guidelines in health systems and public health were more likely. Guidelines published Patient and public involvement statement by WHO, CDC, PHAC, ECDC and SIGN were more likely to issue We partnered with public representatives from the Cochrane statements as GPS with varying strengths of association. GPS Consumer network in the development and conduct of the were more frequently issued in guidelines published for European-­ eCOVID-­ 19RecMap project. The representatives participated in Central Asian use (OR 2.01, 95% C.I 1.54 to 2.62). In contrast, weekly calls of the project executive team where this project was guidelines published for global and North American use were less reviewed for relevence of content and provided contextual feed- likely to issue statements as GPS. Issuing GPS was more common back. The representatives were not involved in the extraction and in statements regarding infection control (OR 1.63, 95% C.I 1.37 evaluation of the GPS. The larger eCOVID-­19RecMap investigator to 1.93), planning and monitoring (OR 1.32, 95% C.I 1.03 to 1.71) team also reviewed the design and conduct of the project and and health services and systems (OR 3.05, 95% C.I 2.30 to 4.05). provided feedback accordingly. Statements considering diagnosis (OR 0.40, 95% C.I 0.27 to 0.61), treatment and rehabilitation (OR 0.16, 95% C.I 0.12 to 0.20) and Statistical analysis screening (OR 0.32, 95% C.I 0.22 to 0.47) were less likely to be Characteristics of the included guidelines and judgements for issued as GPS. Statements concerning vaccination were also asso- each of the GPS evaluation criteria were summarised as percent- ciated with being issued as GPS (OR 1.24, 95% C.I 1.00 to 1.53). ages. Univariate ORs were used to examine the association between guideline and statement characteristics with issuing of Evaluation of development process of the GPS GPS. AGREE II scores were calculated according to the AGREE II Only 27/2375 (1%) of the identified statements that qualified as manual and reported using the median and IQR. All analyses and GPS were actually labelled as GPS by the guideline developers. figures were conducted with R V.4.1.1 software. GPS evaluation Of those, 23/27 (85%) statements satisfied all the GPS criteria and AGREE II scores were stratified by labelling of GPS by guide- (ii–v) with implicit and explicit rationales for development. ‘Clear line developers. and actionable’ was judged as ‘yes’ in 89%, 2% were judged as ‘probably yes’ and 3.7% were judged as ‘probably no’ (figure 3). Results For the criterion ‘necessity of the message for healthcare’, 63% of Characteristics of eligible guidelines the GPS were judged as ‘yes’ and 37% were judged as ‘probably We identified 4533 records through PUBMED, MedSci, hand- yes’. Eleven per cent of those GPS were judged as ‘yes’ for the searching and 11 guideline databases and websites. We excluded criterion relating to net positive consequences from implementing 1401 (31%) guidelines after deduplication and title screening, and the statement, while 82% were judged as ‘probably yes’. For the a further 700 (25%) after screening at full text. Of the identified criterion relating to usefulness of collection and summarisation COVID-­19 guidelines, 412 were related to care in the context of of evidence, 4% of the GPS were judged as ‘yes’, 82% as ‘prob- COVID-­19 and 1746 pertained directly to COVID-­19. The guide- ably yes’ and 15% as ‘probably no’. Fifty-­six per cent provided lines pertaining directly to COVID-­19 were eligible for publishing an explicit statement explaining the chain of indirect evidence on the eCOVID-­ 19RecMap. Of those guidelines, 253 were supporting the development of the GPS and were judged as ‘yes’ extracted and evaluated since the formal launch in November for this criterion. Judgements ‘probably yes’ was assigned to 56% 2020 to September 2021 (figure 1). We identified 2375 of 3726 of the GPS for this criterion. 364 BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Figure 1 PRISMA chart for guidelines eligible for the eCOVID-­19RecMap. BIGG, International Database of Grade Guidelines; CCITC, Changes of Care in Times of COVID-­19; CDC: Centers for Disease Control and Prevention; ECDC, European Centres for Disease Control and Prevention; GIN, Guidelines International Network, NICE, National Institute for Health and Care Excellence; PHAC, Public Health Agency of Canada; SIGN, Scottish Intercollegiate Guidelines Network; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-­Analyses The reporting of implicit or explicit rationales supporting the the guidelines scored over 60% for all three domains. Figure 4 development of statements that qualified as GPS (n=2348) was shows that the six guidelines with labelled GPS scored a median generally similar to those statements labelled as GPS by guide- of 81% (IQR 64–85) in the domain ‘Scope and purpose’, but only line developers. Of those, 2205/2348 (94%) statements satisfied 9.4%, (IQR, 8.3–27 for the domain ‘methodological rigour’ and 0% all the GPS criteria (ii–v) with implicit and explicit rationales. (IQR) 0–0) for the domain ‘editorial independence’. The 194 guide- Notable differences in proportion of statements supported with an lines reporting statements that qualified as GPS scored similarly. explicit rationale were found for criteria ‘statement leads to large Two of those guidelines scored over 60% for all three domains. net positive consequence’ and ‘summarising evidence is a poor use of a guideline development group’s time’, with more frequent Discussion reporting for statements reported as GPS. In contrast, explicit Our evaluation of COVID-­19 recommendations using a novel clas- rationales explaining the chain of indirect evidence supporting sification that anatomises guidelines into actionable statements2 the development of the GPS was more common for statements not shows that guideline developers include advice that frequently reported as GPS, compared with statements reported as GPS (56% qualifies as GPS, (64% of our eligible statements of which 94% vs 39%, respectively). satisfied all the GPS criteria ii–v with implicit and explicit ration- ales) although developers rarely label them as GPS. Accordingly, Quality of guidelines reporting GPS the evaluation of GPS development processes proved challenging. The AGREE II evaluation of the six guidelines reporting statements Statements were more likely to be issued as GPS in European-­ labelled as GPS based on the three domains of interest showed Central Asia guidelines in the field of public health, specifically that the overall quality of these guidelines was limited; none of statements concerning infection control, planning and monitoring BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | 365 BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Table 3 Improving the good practice statement evaluation framework Evaluation questions Explanation and signalling questions Judgement Is collecting and summarising the ►► Would the investigation of the effect of intervention result only in high certinty indirect evidence? that is, cannot Y/PY/PN/N evidence a poor use of a guideline directly investigate the effect of the intervention by comparing to the alternative of the intervention as it would panel’s limited time and energy not be sensible/ethical) Answer ‘Yes’ (opportunity cost is large)? ►► Does the evaluator believe that the alternative of the intervention is highly unlikely to be chosen due to ethical and human right issues? Answer ‘Probably yes’ Is the message really necessary in ►► Do the authors provide a rationale in the text of the guideline to why this message is necessary? Answer ‘Yes’ Y/PY/PN/N regard to actual healthcare practice? ►► Does the evaluator believe that the statement is relevant to healthcare practice? Answer ‘Probably yes’ After consideration of all relevant ►► Is there any information referenced that the implementation of the good practice statement would have a net Y/PY/PN/N outcomes and potential downstream positive impact on health outcomes, as well as on relevant Evidence to Decision criteria (eg, equity)? Answer ‘Yes’ consequences, does implementing the ►► Does the evaluator believe that the implementation of the good practice statement would have a net positive good practice statement likely results impact on health outcomes, as well as on relevant Evidence to Decision) criteria? Answer ‘Probably yes’ in a large net positive consequence? Is there a well-­documented clear ►► Is there a description in the guideline text of the chain of linked indirect evidence, used to infer the net desirable Y/PY/PN/N and explicit rationale connecting the consequences (mainly large health benefits) on the implementation of the good practice statement? Answer ‘yes’ indirect evidence? ►► Does the evaluator believe that there is a chain of linked indirect evidence that can infer the net desirable consequences (mainly large health benefits) on the implementation of the good practice statement? Answer ‘Probably yes’ Is the statement clear and actionable? ►► Does the statement specify what actions are needed while specifying population or setting in the standard PIC Y/PY/PN/N format? Answer ‘Yes’ ►► Does the statement specify what action is needed while specifying population or setting but not in the standard PIC format? Answer ‘Probably yes’ Outcome is not relevant for the actionable statement as not all outcomes can be addressed in an actionable statement. Outcomes are also not typically part of a recommendation. PIC, Population, Intervention, Comparator. and health systems. We found only a few GPS that were supported needed in the way guideline developers conduct GPS develop- by rationales for their development regardless of how the guideline ment. The high prevalence of GPS may be explained by the uncer- developers labelled them. Overall, the quality of most guidelines tainty and rapid spread of COVID-­19, leading to a lack of direct including formal and informal recommendations was poor and, evidence and immediate need for guidance, reducing the rigour of similar to GPS, the recommendations were often not supported the guideline development process. by rationales for their development. Particularly, the reported Our evaluation shows that the most poorly described criteria editorial independence of the guidelines was very low, which were the net consequences of implementing the statement and could question their trustworthiness. Guidelines to overcome the the usefulness of summarising and collecting the evidence. For COVID-­ 19 pandemic would serve healthcare professionals and the former, many rationales are presumed to be ‘straightforward’ services better if included GPS were clearly identified and devel- and based on general knowledge, hence guideline developers may oped through an explicit process. If GPSs are not transparently have been reluctant to document this rationale for each statement. reported by developers, it is likely that they can be misinterpreted. For example, in statements regarding infection control (approx- Thus, in the accompanying article20, we provide operationalised imately 50% of the statements), the interventions aim to prevent and structured implementation of GRADE guidance for the devel- transmission. Although net consequences are not often stated, it opment of GPS. Our findings suggest that significant changes are is implicitly clear that new cases (and deaths) might be prevented. However, for the latter criterion, the judgement rests on the belief of a guideline panel that they have high confidence in the indirect evidence. A formal documentation is needed to ensure that these statements should truly be issued. Strengths and limitations The strengths of this study include the first systematic evaluation of a large sample of COVID-­19 GPS irrespective of language, topic, publication source or date of development. We used criteria previ- ously proposed by the GRADE Working Group for GPS but created explanations and signalling questions in addition to response options, which allowed us to differentiate between statements explicitly or implicitly supported by a proper rationale (table 2). All judgements were conducted in duplicate and reviewed by an expert in guideline development after developing guidance for this approach. Our work has several limitations. First, we did not assess if state- Figure 2 Association of guideline and statement characteristics with ments GRADEd as low or very low certainty were GPS rather than issuing statements that qualify as good practice statements. Reference formal recommendations. It has been shown that GPS are often incor- was issuing actionable statements other than good practice statements. Dashed line corresponds to univariate OR of 1.00. We were not able to rectly GRADEd,12 18 therefore, despite their abundance in COVID-­19 evaluate associations for guideline regions: South Asia and East Asian guidelines, the actual proportion of GPS may be even higher. Second, Pacific and NICE guideline organisation with issuing good practice despite the use of the most recent version of each guideline, this eval- statements due to absence of other types of statements. CDC, Centers uation is limited by its cross-­sectional nature. Temporal changes in the for Disease Control and Prevention; ECDC, European Centres for Disease Control and Prevention; GPS, good practice statement; NICE, National quality of GPS can be assessed in the future as more updated versions Institute for Health and Care Excellence; PHAC, Public Health Agency of of guidelines and recommendations become available. Third, this is Canada; SIGN, Scottish Intercollegiate Guideline Network. the first time this approach to identifying GPS is used and, despite 366 BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Figure 3 Distribution of judgements for good practice statement (GPS) criteria. Annotations correspond to percentage of statements with their respective judgement. GDG, guideline development group. face validity using established criteria8 and the rigorous methods Comparison with other work applied (eg, duplicate judgements by extensively trained raters and Previous work reported that GPS are commonly issued in non-­ validated by experts in guideline development), further validation is COVID-­19 guidelines.12 18 A retrospective evaluation of discordant required. Fourth, our assessment depended on the completeness of recommendations (low or very low confidence in the estimate of reporting in the guidelines and not necessarily the guideline conduct effect) in WHO guidelines identified 29 (18%) as GPS. Similarly, or methods. Fifth, we acknowledge that the nature of the judgement a study produced by the Endocrine Society found 43 (35.6%) of is contingent on a judgement informed by the expertise and knowl- discordant statements were GPS, further indicating that GPS are edge of the evaluator, which may have been variable. To increase prone to misjudgement.12 18 Our findings show that GPS are prev- confidence, all judgements were completed by two trained reviewers alent in guidelines and may be even more commonly used during and verified by an expert in guideline development to validate the public health emergencies. The COVID-­19 crisis may have impacted decisions methodologically. Our multidisciplinary team also includes developers’ ability and capacity to produce more rigorous guidance, content experts of various clinical knowledge who were engaged forcing them to balance methodological rigour with speed. when needed. Implications for guideline users and developers First, our study shows that guideline developers should explic- itly report the use of GPS in the guideline development process. When not explicitly labelled, two approaches using signalling questions on whether a GPS is justified for development were proposed in prior work.8 The first involves identifying that the alternative of the statement is absurd or does not conform with ethical norms. The phrasing of the statement may present a source of confusion when identifying the alternative. Hence, may be unreliable when identifying GPS. The second method involves acknowledging that the collection of high-­certainty indirect evidence to review and support the statement would be a time-­consuming process (criterion iv: summarisation of evidence would be poor use of guideline panel’s time). The latter method requires more expertise and familiarity with the Figure 4 AGREE II assessment (three domains) of guidelines stratified by field of the statement. In turn, users can assess if GPS were labelling of good practice statements by guideline developers. Guidelines appropriately developed using our methodology. containing statements labelled by guideline developers as GPS (n=6) Second, most of the guidelines were produced for global and guidelines containing statements that qualify as GPS (n=194). The thickness of the plot represents the kernal density estimation to show use but guidelines developed in regions other than high the distribution shape of the data. The three lines represent the median income countries (North America and Europe) were scarce. and lower (25%) and upper (75%) quartiles based on density estimates. Thus, implementing the GPS in other settings, especially in Wider sections of the plot represent a higher probability that guidelines low-­income settings, may not be feasible. For example, GPS will take on the given value; the slimmer sections represent a lower probability. AGREE, Appraisal of Guidelines for Research and Evaluation; recommending increasing surveillance for farm workers and GDG, guideline development group; GPS, good practice statement. their close contacts or maintaining humidity level indoors BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | 367 BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine 11 between 30% and 50% is heavily dependent on resources and Department of Internal Medicine, American University of Beirut, Beirut, influenced by organisational aspects. Lebanon 12 Third, adherence to our updated guidance for the operational- Internal Medicine, Division of Nephrology and Hypertension, University isation and implementation of GPS development20 may improve of Kansas School of Medicine, Kansas City, Kansas, USA 13 Department of Health Product Policy and Standards, World Health the transparency in the process of developing and reporting of Organization, Geneve, Switzerland GPS and help direct guideline developers’ resources and efforts 14 Yealth Network, Beijing Yealth Technology Co., Ltd, Beijing, China to what is needed and avoid the inappropritate issuing of GPS. 15 Cochrane Campbell Global Ageing Partnership, London, UK For example, the European Commission Initiative of Breast Cancer 16 Iberoamerican Cochrane Center, Biomedical Research Institute Sant Guidelines on Breast Cancer Screening and Diagnosis21 reported Pau-­CIBERESP, Barcelona, Spain their GPS in a supplementary document and provided detailed 17 Cochrane South Africa, South African Medical Research Council, Cape descriptions of the rationales supporting them. Town, Western Cape, South Africa 18 Clinical Pharmacology, Department of Medicine, Stellenbosch Implications for research University, Stellenbosch, Western Cape, South Africa 19 We evaluated the GPS primarily through information provided National Institute for Health and Care Excellence, London, UK 20 in the guideline and judgement of the evaluators. Our evalua- Centre de Recherche du Centre Hospitalier Affilié Universitaire de Québec (CHA), CHA-­Hôpital de l'Enfant-­Jésus, Université Laval, Quebec, tion of COVID-­19 GPS using the previously published five criteria Quebec, Canada for GPS provided us with insight that improvements to the GPS 21 Department of Anesthesiology and Critical Care Medicine, Université framework are required to ensure reproducible and valid future Laval, Québec City, Québec, Canada evaluations of GPS. Our suggested framework for evaluating GPS 22 Czech National Centre for Evidence-­Based Healthcare and Knowledge builds on our incorporation of judgements with response options Translation, Institute of Biostatistics and Analyses, Masaryk University, that we applied in our evaluation. We also provide a specific Brno, Czech Republic order, explanations and signalling questions for using the criteria 23 Department of Clinical Epidemiology and Public Health, Institut for GPS evaluation (table 2). For example, the assessment if the d’Investigació Biomèdica Sant Pau IIB Sant Pau, Barcelona, Spain 24 statement is actionable and clear was placed at the end of the Department of Medicine, University of Ottawa Faculty of Medicine, evaluation as it is not specific to GPS and does not impact on the Ottawa, Ontario, Canada 25 appropriatness of the rationale for its development. Furthermore, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada it is not specific to GPS, but is relevant for all actionable state- 26 Deparatment of Family Medicine, University of Ottawa, Ottawa, Ontario, ments. We found that using the criterion summarising evidence Canada would be poor use of guideline panel’s time as the first criterion 27 Joanna Briggs Institute, University of Adelaide, Adelaide, South for the evaluation, helps with differentiating the GPS from other Australia, Australia types of actionable statements although is sometimes a difficult 28 Department of HIV, Hepatitis and Sexually Transmitted Infections, World judgement to make. Further testing of this framework by other Health Organization, Geneva, Switzerland research teams is required, along with specific GRADE guidance 29 Clinical Research Institute, American University of Beirut, Beirut, for the development and evaluation of GPS. Lebanon 30 Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany Conclusions 31 Medical Center - University of Freiburg, Institute for Evidence in The large number of GPS in COVID-­19 guidelines emphasises their Medicine, Freiburg, Germany importance in guidelines especially during public health emergen- 32 American College of Physicians, Philadelphia, Pennsylvania, USA 33 cies, when there is a need for urgent guidance and there is a lack of Department of Biomedical Sciences, Humanitas University, Milan, Italy 34 direct evidence to inform decision making. Our evaluation shows that Cochrane Canada, Hamilton, Ontario, Canada improvements are needed in the presentation, transparent reporting and the rationale for GPS development beyond the existing GRADE Correction notice This article has been corrected since it first guidance. Furthermore, we need studies to monitor the progress published. ORCID has been added for Miloslav Klugar. around GPS development and evaluate potential barriers slowing the Twitter Omar Dewidar @okdewidar uptake of available guidance by guideline developers. Acknowledgements We would like to acknowledge the research Author affiliations collaborators that were involved in the screening of guidelines 1 Methods Centre, Bruyère Research Institute, Ottawa, Ontario, Canada and the extraction and evaluation of the GPS included in the 2 School of Epidemiology and Public Health, University of Ottawa, Ottawa, eCOVID-­19 recommendations map. Ontario, Canada 3 Contributors OD, TL, ML, ZSP, EP and HJS contributed to Department of Health Research Methods, Evidence and Impact, the study conception and design. KS designed and ran the McMaster University, Hamilton, Ontario, Canada 4 Michael G DeGroote Cochrane Canada and McMaster GRADE Centres, literature searches. OD, JK, ZN and MM screened literature and McMaster University, Hamilton, Ontario, Canada conducted the data extraction and evaluation. Authors provided 5 WHO Collaborating Center for Infectious Diseases, Research Methods feedback on the conceptual approach used in this study. All and Recommendations, McMaster University, Hamilton, Ontario, Canada authors provided critical review, interpretation and approval 6 Amsterdam University Medical Centers, University of Amsterdam, of the final manuscript. Other members of the eCOVID-­19 Amsterdam, Netherlands recommendations map contributed to the screening, data 7 Joint Research Centre, European Commission, Ispra, Italy extraction and data validation but did not meet authorship 8 Instituto de Salud Carlos III, Agencia de Evaluación de Tecnologías criteria. The corresponding author attests that all listed authors Sanitarias, Madrid, Spain 9 meet authorship criteria and that no others meeting the criteria Department of Medicine, McMaster University, Hamilton, Ontario, Canada have been omitted. HJS acts as guarantor and accepts full 10 Department of Pediatrics, Post Graduate Institute of Medical Education responsibility for the work and/or the conduct of the study, had and Research, Chandigarh, India access to the data, and controlled the decision to publish. 368 BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | BMJ EBM: first published as 10.1136/bmjebm-2021-111866 on 15 April 2022. Downloaded from http://ebm.bmj.com/ on July 17, 2023 at Sant Paul Hospital Consortia. Protected by copyright. General medicine Funding CIHR (FRN VR4-­172741 & GA3-­177732) for COVID-­19 References recommendation mapping. AFT is the Chairholder of the Canada 1 Woolf S, Schünemann HJ, Eccles MP, et al. Developing clinical practice Research Chair in Critical Care Neurology and Trauma. guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and presentation and deriving recommendations. Competing interests HJS, AS, VAW report grants from Canadian Implement Sci 2012;7:61. Instituites of Health during the conduct of the study—FRN VR4-­ 2 Lotfi T, Hajizadeh A, Moja L, et al. A taxonomy and framework for 172741 & GA3-­177732. RAM reports grants from WHO, grants identifying and developing actionable statements in guidelines suggests from ASH, grants from ACR, other from Boehringer ingelheim avoiding informal recommendations. J Clin Epidemiol 2022;141:161-­171. international, grants from NIDDK, outside the submitted work. 3 Oxman AD, Fretheim A, Schünemann HJ, et al. Improving the use of EA, PA-­C and HJS report contribution to the development of the research evidence in Guideline development: introduction. Health Res original five criteria for assessing the appropriateness of issuing Policy Syst 2006;4:12. 4 Qaseem A, Forland F, Macbeth F, et al. Guidelines international network: good practice statements. The remaining authors have nothing toward international standards for clinical practice guidelines. Ann Intern else to declare. Med 2012;156:525–31. Patient and public involvement Patients and/or the public were 5 Schünemann HJ, Best D, Vist G, et al. Letters, numbers, symbols and involved in the design, or conduct, or reporting, or dissemination words: how to communicate grades of evidence and recommendations. plans of this research. Refer to the Methods section for further CMAJ 2003;169:677. details. 6 Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful Patient consent for publication Not applicable. guideline enterprise. CMAJ 2014;186:E123–42. Provenance and peer review Not commissioned; externally peer 7 Guyatt G, Oxman AD, Akl EA, et al. Grade guidelines: 1. Introduction-­ GRADE evidence profiles and summary of findings tables. J Clin Epidemiol reviewed. 2011;64:383–94. Data availability statement All data relevant to the study 8 Guyatt GH, Alonso-­Coello P, Schünemann HJ, et al. Guideline panels are included in the article or uploaded as online supplemental should seldom make good practice statements: guidance from the grade information. All relevant data included. Working group. J Clin Epidemiol 2016;80:3–7. 9 Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and Supplemental material This content has been supplied by why is it important to clinicians? BMJ 2008;336:995–8. the author(s). It has not been vetted by BMJ Publishing Group 10 Guyatt GH, Oxman AD, Vist GE, et al. Grade: an emerging consensus on rating Limited (BMJ) and may not have been peer-­reviewed. Any quality of evidence and strength of recommendations. BMJ 2008;336:924–6. opinions or recommendations discussed are solely those of 11 World Health O. Chapter 14: Strong recommendation when the evidence the author(s) and are not endorsed by BMJ. BMJ disclaims all is low quality. In: Who Handbook for Guideline development. 2014. 2 edn. liability and responsibility arising from any reliance placed Geneva: World Health Organization, 2014. on the content. Where the content includes any translated 12 Brito JP, Domecq JP, Murad MH, et al. The endocrine Society guidelines: when the confidence CART goes before the evidence horse. J Clin material, BMJ does not warrant the accuracy and reliability of Endocrinol Metab 2013;98:3246–52. the translations (including but not limited to local regulations, 13 Ponce OJ, Alvarez-­Villalobos N, Shah R, et al. What does expert opinion clinical guidelines, terminology, drug names and drug dosages), in guidelines mean? A meta-­epidemiological study. Evid Based Med and is not responsible for any error and/or omissions arising 2017;22:164–9. from translation and adaptation or otherwise. 14 COVID19 recmap. Available: https://covid19.recmap.org Open access This is an open access article distributed in 15 Lotfi T, Stevens A, Akl EA, et al. Getting trustworthy guidelines into the hands of decision-­makers and supporting their consideration of contextual accordance with the Creative Commons Attribution Non factors for implementation globally: recommendation mapping of Commercial (CC BY-­NC 4.0) license, which permits others COVID-­19 guidelines. J Clin Epidemiol 2021;135:182–6. to distribute, remix, adapt, build upon this work non-­ 16 Organization WH. Who guidelines, 2021. Available: https://www.who.int/​ commercially, and license their derivative works on different publications/who-guidelines terms, provided the original work is properly cited, appropriate 17 Wiercioch W. Priority topics for panel engagement in health Guideline credit is given, any changes made indicated, and the use is development McMaster University, 2020. non-­commercial. See: http://creativecommons.org/licenses/by-​ 18 Alexander PE, Brito JP, Neumann I, et al. World Health organization strong nc/4.0/. recommendations based on low-­quality evidence (study quality) are frequent and often inconsistent with grade guidance. J Clin Epidemiol 2016;72:98–106. ORCID iDs 19 Brouwers MC, Kho ME, Browman GP, et al. Agree II: advancing Omar Dewidar http://orcid.org/0000-0001-6420-887X Guideline development, reporting and evaluation in health care. CMAJ Karla Solo http://orcid.org/0000-0001-6134-9140 2010;182:E839–42. Derek K Chu http://orcid.org/0000-0001-8269-4496 20 Dewidar O, Lotfi T, Langendam MW. Good or best practice statements: proposal for the operationalisation and implementation of grade guidance. Tamara Kredo http://orcid.org/0000-0001-7115-9535 BMJ Evid Based Med 2022. [Epub ahead of print: 15 April 2022]. Miloslav Klugar http://orcid.org/0000-0002-2804-7295 doi:10.1136/bmjebm-2022-111962 Vivian Andrea Welch http://orcid.org/0000-0002-5238-7097 21 European Commission of breast cancer (ECIBC). Organising breast cancer Joanne Khabsa http://orcid.org/0000-0002-4336-3501 screening programmes, 2021. Available: https://healthcare-quality.jrc.ec.​ Thomas Piggott http://orcid.org/0000-0003-1643-5386 europa.eu/european-breast-cancer-guidelines/organisation-of-screening-​ Holger J Schünemann http://orcid.org/0000-0003-3211-8479 programme BMJ Evidence-­Based Medicine December 2022 | volume 27 | number 6 | 369