Open access Protocol Protocol of an interdisciplinary consensus project aiming to develop an AGREE II extension for guidelines in surgery George A Antoniou ‍ ‍,1,2 Dimitris Mavridis,3,4 Sofia Tsokani,3 Manuel López-­Cano,5 Iván D Flórez,6,7 Melissa Brouwers,6 Sheraz R Markar,8 Gianfranco Silecchia,9 Nader K Francis,10 Stavros A Antoniou11,12 To cite: Antoniou GA, ABSTRACT Mavridis D, Tsokani S, et al. Strengths and limitations of this study Introduction Appraisal of Guidelines for Research Protocol of an interdisciplinary and Evaluation (AGREE II) is an instrument that informs consensus project aiming ►► This is the first project to address guideline develop- development, reporting and assessment of clinical practice to develop an AGREE II ment and reporting in surgery. guidelines. Previous research has demonstrated the need extension for guidelines ►► It will combine statistical considerations, conceptual in surgery. BMJ Open for improvement in methodological and reporting quality parameters to be derived from qualitative synthesis 2020;10:e037107. doi:10.1136/ of clinical practice guidelines specifically in surgery. We and a formal Delphi process. bmjopen-2020-037107 aimed to develop an AGREE II extension document for ►► It will involve a panel of stakeholders from a variety application in surgical guidelines. ►► Prepublication history and of scientific, cultural and geographical backgrounds. Methods and analysis We have performed a structured additional material for this ►► The project will not address specific disciplines of literature review and assessment of guidelines in surgery paper are available online. To surgery. view these files, please visit using the AGREE II instrument. In exploratory analyses, the journal online (http://​dx.​doi.​ we have identified factors associated with guideline org/​10.​1136/​bmjopen-​2020-​ quality. We have performed reliability and factor analyses 037107). to inform the development of an extension document. We guidelines evaluate, summarise and contex- will summarise this information and present it to a Delphi tualise research evidence into actionable Received 18 January 2020 panel of stakeholders. We will perform iterative Delphi recommendations.3 As such, guidelines have Revised 26 May 2020 rounds and we will summarise the final results to develop a direct impact on delivery of healthcare and Accepted 13 July 2020 the extension instrument in a dedicated consensus surgical services. It is therefore of paramount conference. importance to ensure the highest quality stan- Ethics and dissemination Funding bodies will not be dards in developing and reporting guidelines. involved in the development of the instrument. Research A great amount of scientific endeavour in ethics committee and Health Research Authority approval the past few years has focused on the quality was waived, since this is a professional staff study only of scholarly work, including clinical practice and no duty of care lies with the National Health Service guidelines.4 Reporting standards have been to any of the participants. Conflicts of interest, if any, will be addressed by reassigning functions or replacing developed for virtually all study designs and participants with relevant conflicts. The results will be have been summarised by the Enhancing the disseminated through publication in peer reviewed Quality and Transparency of health Research journals, the funders’ websites, social media and direct (EQUATOR) Network.5 Appraisal of Guide- contact with guideline development organisations and lines for Research and Evaluation (AGREEII) peer-­reviewed journals that publish guidelines. constitutes a framework for developing, appraising and reporting clinical practice guidelines.6 It is endorsed by major interna- © Author(s) (or their INTRODUCTION tional and national agencies, including the employer(s)) 2020. Re-­use Research evidence is the primary source to WHO and the National Institute for Health permitted under CC BY-­NC. No commercial re-­use. See rights inform medical practice forming the corner- and Care Excellence (NICE) in the UK.7 8 and permissions. Published by stone of evidence-­ based medicine.1 An AGREE II is a generic tool that applies to all BMJ. average of 5639 new articles were indexed per disciplines of medicine, and no modifica- For numbered affiliations see month under the subject heading ‘Surgery’ tion or extension of the framework has been end of article. in the National Library of Medicine over the proposed, described or developed for specific Correspondence to past decade.2 Given this fact, keeping abreast clinical branches such as surgery. Dr George A Antoniou; of the latest evidence is a strenous task for The tool is composed of 23 items organ- ​antoniou.​ga@​hotmail.​com healthcare practitioners. Clinical practice ised in seven thematic domains: scope and Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 1 Open access purpose, stakeholder involvement, rigour of develop- ment, clarity of presentation, applicability and editorial independence. It concludes with an overall assessment and a statement of whether the guideline is considered of sufficient quality to be used or recommended in clinical practice (online supplementary appendix). Need for an AGREE II extension Our research group has acted as methodological and content coordinators of landmark surgical guidelines and have served as members of surgical guideline develop- ment groups.9–14 Even though members of our group, in their role as guideline developers, have made every effort to comply with the highest methodological standards, as indicated by adherence to Grading of Recommendations Assessment, Development and Evaluation (GRADE) and AGREE II methodologies,15 16 we noticed that compliance with all aspects of several parameters of the AGREE II instrument was not possible. For example, the item ‘The potential resource impli- cations of applying the recommendations have been considered’ may be difficult to be universally addressed. Figure 1 Development steps of the guideline assessment Cost-­effectiveness studies are scarce in the surgical liter- project with the ultimate objective to develop an AGREE ature and relevant evidence typically varies in different II extension document for surgical guidelines. AGREE II, settings.17 Since surgical expertise varies across countries Appraisal of Guidelines for Research and Evaluation. and institutions, there is a need for the instrument to consistently apply to different healthcare settings. Surgical interventions are complex, and details on the interven- project is a result of a partnership between an interna- tions/comparators are imperative for the target users to tional team of surgical research experts and two of the be able to assess the external validity of the guidelines. AGREE research team leads (IDF and MB). The AGREE Specialists from different specialties and allied health research team is currently under a membership renova- professionals with a wide range of expertise are involved tion process, and, therefore, neither of the authors can in the treatment of surgical patients, which makes their speak on behalf of the entire AGREE group. However, involvement in guideline development paramount. We both AGREE research team leads state that AGREE has have hypothesised that the original AGREE II document supported the project from its inception. Furthermore, may not be applicable to clinical practice guidelines in they have agreed to support dissemination activities by surgery, which often represent complex and multifaceted making this new tool available in the AGREE website interventions. (https://​agreetrust.​org). This protocol complies with the Guidance for Devel- Objective opers of Health Research Reporting Guidelines.21 There are a few guideline reporting documents in other fields of medicine18 19; however, a scoping literature Patient and public involvement review by our group has not identified any document to No patients were involved in the development of this inform guideline development and reporting in the field protocol. of surgery. Our aim was to develop an extension of the AGREE II instrument that is specific for surgery through GAP I: literature review and exploratory analyses an evidence-­informed and consensus-­based approach. We have previously performed a structured review to identify clinical practice guidelines in the field of surgery published over a 10-­year period.22 We have assessed the METHODS AND ANALYSIS methodological and reporting quality of the selected We have formed an international multidisciplinary and guidelines using the original AGREE II criteria. Domain interdisciplinary collaborative research working group scores (calculated by summing up all the scores of the that consists of surgeons, guideline developers, evidence individual items in a domain and by scaling the total as synthesis experts, GRADE methodologists,20 biostatisti- a percentage of the maximum possible score for that cians and a lead member of the AGREE collaboration. domain)16 ranged between 0% and 56%, suggesting This is a tripartite project named Guideline Assessment generally inadequate and highly variable guideline Project (GAP): Filling the GAP in Surgical Guidelines. quality. The median overall score was 4 out of a maximum A summary of the project is outlined in figure 1. The of 7, and 40% of guidelines were not considered suitable 2 Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 Open access for use based on their quality as assessed using the AGREE Box 1 Stakeholders to participate in a web-­based Delphi II instrument. process In exploratory analyses, we have found guidelines produced by surgical organisations with a high (≥1 guide- General surgeon line per year) output (OR 3.79, 95% CI 1.01 to 12.66), Urologist and those produced by surgical organisations with a Thoracic surgeon guideline committee (OR 4.15, 95% CI 1.47 to 11.77) Vascular surgeon have higher odds of reaching sufficient quality and being Paediatric surgeon recommended for use.22 Journal editor National authority representative GAP II: statistically calibrating the AGREE II instrument NICE representative Guideline developer/representative from Europe The second part of this project was focused on statis- Guideline developer/representative from North America tical calibration of the AGREE II instrument. We have Guideline developer/representative from Asia used quality appraisal data from GAP I and employed a Guideline developer/representative from middle-­income country series of statistical methods to explore reliability, internal Healthcare provider representative consistency and unidimensionality of the AGREE II Representative from GRADE instrument when it is applied in surgical guidelines. We Guideline implementer investigated the internal consistency that refers to the Patient representative extent to which all items of the instrument measure the WHO representative same hypothetical construct. We explored if and how test European Commission representative items are intercorrelated. Large intercorrelations among GIN representative test items are indicative of the items measuring the same EQUATOR representative construct. Using reliability analysis, Kendall’s tau statis- tics, factor analysis and the item response theory, we EQUATOR, Enhancing the Quality and Transparency of health Research; explored whether items of each AGREE II domain are GIN: Guidelines International Network; GRADE, Grading of Recommendation Assessment, Development and Evaluation; NICE, National Institute of Health and intercorrelated and are, therefore, indicators of the same Care Excellence. construct. Statistical modelling showed that excluding five items from the original tool (items 1, 2, 5, 7 and 8) and rearranging the remaining items into four domains instead of six would enhance the instrument. We have ►► The strategic steering group is responsible for over- finally drafted a modified AGREE II document for guide- seeing the project. lines in surgery, on the basis of the outcomes of statistical ►► The methods group coordinates the methodology of models. the project. ►► The evidence review group will review the literature GAP III: AGREE II extension for surgical guidelines for evidence on candidate new items to be included The third part of the project aims to use the information in the extension document. from the previous GAP projects and other published ►► The evidence synopsis group will summarise evidence information on the topic to develop the extension docu- for presentation to Delphi participants. ment using a structured Delphi process involving relevant The group attended a 1-­ day meeting to discuss the stakeholders. findings of previous work, define the methodology and The multidisciplinary Delphi panel will include surgical study design, and identify potential stakeholder groups to specialists, journal editors, guideline development bodies, comprise the Delphi panel. GRADE representatives and patient representatives. Under consideration of the evidence, stakeholders will be Delphi process asked to provide their input through a Delphi process, The Delphi panel will consist of key stakeholders, which will inform the preparation of an AGREE II exten- including representatives from different surgical disci- sion for surgical guidelines. plines (general surgery, urology, thoracic surgery, vascular surgery and paediatric surgery), guideline devel- Participants opers from different continents and representatives of The executive group consists of surgeons (ML-­C, SRM, guideline development organisations. The public will be GS, GAA, NKF and SAA), members of surgical quality involved by participation of patient representatives from and research boards (ML-­C, NKF and SAA), guideline the European Patients Forum (box 1). We will develop a developers (ML-­C, IDF, MB, GS, NKF and SAA), evidence web-­based survey tool to facilitate Delphi exercises. Find- synthesis experts (IDF, GAA, DM and SAA), GRADE ings of previous work (GAP I and GAP II) and further methodologists (ML-­C and SAA),20 biostatisticians (DM evidence that will be identified through a scoping litera- and ST) and two leads of the AGREE Group (IDF and ture review will be summarised and presented to Delphi MB). It is further divided into four working groups with participants. Summary information will also be available distinct functions and responsibilities: on the project website (https://​gap-​project.​org). Online Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 3 Open access links to full documents for detailed review of the evidence based on criteria of relevance (whether they contribute will be provided. to the development or testing of the initial theories)24 The first round will include open-­ended questions to and appropriateness for addressing the research ques- identify candidate items for inclusion in the extension tions.25 26 document. Responses will be grouped and summarised Studies will be entered into A ​ TLAS.​ti and coded to by the methods group before the second round is identify the specific features relevant to development and commenced. reporting of surgical guidelines. Themes will be discussed The second round will include closed-­ended questions by the research team using an iterative and speculative in a 5-­point Likert scale to assess participants’ opinions process.26 Adjudication and triangulation will be applied and level of agreement on including candidate items or to refine theories which can be used across the studies to excluding existing items from the extension document. understand findings. As per protocol, 1/2 indicates strong/moderate disagree- Furthermore, we will invite users of social media ment; 3 indicates no opinion; and 4/5 indicates moderate/ through the project account on Twitter (@GAProject2) strong agreement. Candidate items will have been identi- and through communication streams of the sponsoring fied through GAP I and GAP II and the scoping literature bodies (Facebook, Twitter and email newsletters) to review. We will discard low-­scoring items (ie, those with nominate parameters of importance in the development a median score of 1/2 on the Likert scale) and use the and reporting of guidelines in surgery, and will group and shortlisted items in a third Delphi round. We will repeat summarise their responses. Evidence identified from the the process until an agreement of 80% (4/5 on the Likert aforementioned pathways, along with information from scale) is reached among Delphi participants. GAP I and GAP II, will be summarised and taken into The Delphi panel’s contribution will be acknowledged account when developing the extension document. by group authorship in subsequent publications of the extension document, the elaboration document and Consensus meeting supporting tools. Following the Delphi process, the executive group will meet to discuss the findings and compose the first draft Qualitative research synthesis of the extension document. We will present new items We will perform qualitative evidence synthesis to iden- that will be identified through the Delphi exercise and tify factors of conceptual importance to the quality of the qualitative synthesis, and discuss their plausibility and evidence in surgery. The overarching question will be possible inclusion in the instrument. Similarly, items to be How do clinical practice guidelines in surgery differ excluded with the respective rationale will be discussed. from non-­surgical guidelines? Specific thematic questions The group will finalise the extension document by will be addressed: ordering and allocating items into domains. 1. Which are the concepts that make surgical guidelines The executive group will hold a further meeting with different from guidelines or summary evidence in oth- the advisory group, which is composed of journal editors er medical fields? and representatives of surgical associations to discuss 2. Which are potential items that may be of sufficient im- dissemination and implementation processes of the portance to be included in an AGREE II extension for developed extension instrument. surgical guidelines? 3. Which are the items of the original AGREE II that Pilot testing and assessment of internal validity might not be relevant to surgical guidelines? The extension instrument will be pilot-­ tested by two 4. How should items of the original AGREE II instrument members of the executive group. One member will apply be modified to be more relevant to an AGREE II exten- the instrument on the surgical guidelines, which were sion for surgical guidelines? identified by the structured search process as described in We will conduct a scoping search of PubMed, Embase GAP I22 and on additional guidelines that will be identi- and Google Scholar. In keeping with realist review guide- fied by extending the search to the present date. A second lines,23 there will be no restrictions on the types of study member will independently follow the same process in a design eligible for inclusion. We will consider editorials, randomly selected sample of 15 guidelines. The biosta- letters to the editor, commentaries, opinions and any type tistical team will assess the internal validity by applying of publication that captures the breadth discussions about the statistical models of GAP II. Any difficulties encoun- development of surgical guidelines. Information will be tered with the use of the instrument will be documented used to identify characteristics that specifically apply to and addressed. Results of the statistical assessment will be surgical guidelines. appraised against statistical findings of the appraisal of The realist review will aim to develop an explanatory the original AGREE II instrument (GAP II). understanding of development and reporting of surgical guidelines, how surgical guidelines differ from non-­ AGREE II extension statement surgical ones and how AGREE II can be modified to reflect The extension statement along with an explanation and the specific aspects of surgical guidelines. According to elaboration (E&E) document will be composed by the the realist synthesis methodology, studies will be assessed executive group. The E&E document will detail the use of 4 Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 Open access the extension instrument in developing and reporting a The outcome of this project will be the first AGREE II new surgical guideline and appraising an existing surgical extension document. Reporting Tool for Practice Guide- guideline. lines in Healthcare (RIGHT) is another reporting instru- ment for clinical practice guidelines.37 We are aware of a AGREE II extension checklist planned RIGHT extension for public versions of guide- A checklist including the AGREE II extension items will lines and a RIGHT extension for adapted practice guide- be developed with the aim of this checklist to be used lines.38 39 An extension document of RIGHT for surgical by guideline developers (to summarise development guidelines would be justified as well. However, in view of and reporting parameters), guideline users (to appraise the evidenced gap in methodological quality of surgical quality), peer reviewers and journal editors (to assess guidelines,22 we considered more appropriate to elabo- adequacy of reporting parameters).27 rate on AGREE II, as it addresses guideline development, reporting and appraisal. Feedback and criticism We will invite constructive feedback on the instrument Strengths and limitations through the dedicated website (https://​gap-​project.​org), This tripartite project is the first to employ statistical and we will consider comments in letters to the editor models to inform the validity of an extension, modifica- and via the social media. An ad hoc team will collect and tion or update document on guidelines reporting. We summarise the feedback received in 3 monthly intervals will correlate statistical findings with conceptual consid- for the first year after publication, and the executive erations of the need for adjustments/extension of the group will discuss and address this information in web-­ AGREE II instrument. The project methods group has based meetings. adopted recommendations on developing research Monitoring, update and future steps reporting guidelines, proposed by a collaborative team The executive group will monitor the use of the exten- who have developed a significant number of such guide- sion document and appraise its applicability in surgical lines.21 The holistic approach to developing an extension guidelines for a reasonable period of time after dissemi- document for clinical practice guidelines in surgery is nation and will publish their findings. Following consid- reflected in the diverse scientific, cultural and geograph- eration of the outcomes, feedback, criticism, suggestions ical backgrounds of experts in the field involved in the and new evidence in the field, we will discuss the need for project. Similarly, the Delphi panel will include stake- an update. The development of further extension instru- holders and members from a variety of backgrounds, ments for national surgical guidelines and guidelines in including clinicians/surgeons, methodologists, guide- distinct surgical or other interventional disciplines will be line developers, policymakers and the public (patient considered following discussions with key stakeholders. representatives). A face-­to-­face meeting of Delphi participants, instead Implications for practice and research of a full web-­based Delphi process, might be more effi- Clinical practice guidelines directly impact clinical prac- cacious in developing the extension document allowing tice and healthcare delivery and, as such, development direct exchange of opinions, information and ideas. We must follow rigorous methodological and reporting will encourage a full participation and exchange of infor- standards. The AGREE II instrument has been designed mation by developing a user-­ friendly and interaction-­ as a generic tool for development and appraisal of clin- allowing web-­ based platform. Furthermore, we will ical practice guidelines.6 It is not intended to substitute incentivise participation and engagement of potential established detailed guidance on guideline development Delphi members by proposing group authorship and principles, processes and procedures, such as the GRADE participation in future associated projects. approach.15 It has addressed a vital need to summarise and detail essential development steps and reporting Ethics and dissemination parameters for high-­quality guidelines. Furthermore, as Research Ethics Committee and Health Research an appraisal instrument, it may be used by healthcare prac- Authority approval was waived since this is a professional titioners, policymakers and other stakeholders to inform staff study only and no duty of care lies with the National decisions regarding the use of an existing guideline. Health Service to any of the participants. We will request In addition, AGREE II has been shown to be a valu- electronic informed consent from Delphi participants and able tool for assessment of guideline quality in several the responses of the Delphi members will be anonymised. clinical disciplines and evidence fields.22 28–36 Such We have obtained conflict of interest forms of all exec- summaries alert the scientific community to the need utive group members and will request electronic and/ for improving specific aspects of clinical practice guide- or written informed consent by Delphi participants and lines (corresponding to the instrument domains) or the members of the advisory group. We will deal with potential overall quality of guidelines. Our previous research has conflicts of interest by reassigning functions or replacing highlighted the need for improvement of the quality of participants who pose interest conflict. surgical guidelines.22 An AGREE II extension for surgical We will submit the final paper with the extension docu- guidelines is expected to meet this need. ment to be considered for publication in the UEG Journal Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 5 Open access and Surgical Endoscopy, as defined in the respective prede- of the work are appropriately investigated and resolved. ST: analysis of data, velopment agreements. We will negotiate simultaneous interpretation of data, revising the work critically for important intellectual data, final approval for the work to be published, agreement to be accountable for all publications in other surgical journals for widest dissemi- aspects of the work in ensuring that questions related to the accuracy or integrity of nation as recommended by the Guidance for Developers any part of the work are appropriately investigated and resolved. ML-­C: acquisition of Health Research Reporting Guidelines.21 of data, analysis of data, interpretation of data, revising the work critically for We will make the extension document available in a important intellectual data, final approval for the work to be published, agreement to be accountable for all aspects of the work in ensuring that questions related to dedicated website with links to the original publications. the accuracy or integrity of any part of the work are appropriately investigated and We will encourage surgical organisations with guideline resolved. IDF: conception and design, interpretation of data, drafting the work, final development activities to use the instrument. We will approval for the work to be published, agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any further pursue dissemination through the websites of the part of the work are appropriately investigated and resolved. MB: conception and funding bodies and channels of social media of major design, interpretation of data, revising the work critically for important intellectual stakeholers, such as the Guideline International Network, data, final approval for the work to be published, agreement to be accountable GRADE and EQUATOR. for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. SRM: Through direct contact, we will advise international interpretation of data, revising the work critically for important intellectual data, surgical and guideline development organisations and final approval for the work to be published, agreement to be accountable for all policymakers to endorse the extension instrument. aspects of the work in ensuring that questions related to the accuracy or integrity of Furthermore, editors of surgical journals will be advised any part of the work are appropriately investigated and resolved. GS: interpretation of data, revising the work critically for important intellectual data, final approval for to provide an extension instrument checklist that authors the work to be published, agreement to be accountable for all aspects of the work of clinical practice guidelines should submit, along with in ensuring that questions related to the accuracy or integrity of any part of the the original manuscript. work are appropriately investigated and resolved. NF: interpretation of data, revising The GAP III study aims to address the need for improve- the work critically for important intellectual data, final approval for the work to be published, agreement to be accountable for all aspects of the work in ensuring ment of the methodology, reporting and appraisal of that questions related to the accuracy or integrity of any part of the work are surgical guidelines. An extension document specifically appropriately investigated and resolved.SA: conception and design, acquisition of designed for clinical practice guidelines in surgery will data, analysis of data, interpretation of data, drafting the work, final approval for the further improve the value, use and applicability of the work to be published, agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work AGREE II instrument in the surgical field with the ulti- are appropriately investigated and resolved. mate goal of enhancing patient care, experience and Funding This project received funding from the United European Gastroenterology outcomes. and the European Association for Endoscopic Surgery. The funding bodies had no influence on the development of this protocol. Author affiliations 1 Competing interests None declared. Department of Vascular and Endovascular Surgery, The Royal Oldham Hospital, Pennine Acute Hospitals NHS Trust, Northern Care Alliance NHS Group, Manchester, Patient consent for publication Not required. UK Provenance and peer review Not commissioned; externally peer reviewed. 2 Division of Cardiovascular Sciences, School of Medical Sciences, University of Open access This is an open access article distributed in accordance with the Manchester, Manchester, UK 3 Creative Commons Attribution Non Commercial (CC BY-­NC 4.0) license, which Department of Primary Education, School of Education, University of Ioannina, permits others to distribute, remix, adapt, build upon this work non-­commercially, Ioannina, Greece 4 and license their derivative works on different terms, provided the original work is Faculté de Médecine, Université Paris Descartes, Paris, France properly cited, appropriate credit is given, any changes made indicated, and the use 5 Abdominal Wall Surgery Unit, Department of General Surgery, Hospital Universitari is non-­commercial. See: http://​creativecommons.​org/​licenses/​by-​nc/​4.​0/. Vall d'Hebron, Universitat Autònoma de Barcelona, Barcelona, Spain 6 Department of Health Research Methods Evidence and Impact, McMaster ORCID iD University, Hamilton, Ontario, Canada George A Antoniou http://​orcid.​org/​0000-​0001-​8209-​0406 7 Department of Pediatrics, Universidad de Antioquia, Medellin, Colombia 8 Department of Surgery and Cancer, Imperial College London, London, UK 9 Department of Surgery, University Hospital "La Sapienza", Rome, Italy 10 Department of General Surgery, Yeovil District Hospital NHS Foundation Trust, Yeovil, UK REFERENCES 11 Medical School, European University Cyprus, Nicosia, Cyprus 1 Djulbegovic B, Guyatt GH. Progress in evidence-­based medicine: a 12 Department of Surgery, Mediterranean Hospital of Cyprus, Limassol, Cyprus quarter century on. Lancet 2017;390:415–23. 2 Pubmed. Available: https://​pubmed.​ncbi.​nlm.​nih.​gov/ [Accessed 2 Oct 2017]. Twitter Dimitris Mavridis @dimi_mavridis, Sofia Tsokani @STsokani, Manuel 3 Graham R, Mancher M, Wolman DM. Clinical practice guidelines we López-­Cano @ManuelLpezCano1, Iván D Flórez @IvanD_Florez, Melissa Brouwers can trust, 2011. @melissa_SEPH_UO, Sheraz R Markar @MarkarSheraz, Nader K Francis @nader_ 4 Altman DG, Simera I, Hoey J, et al. EQUATOR: reporting guidelines nkfrancis and Stavros A Antoniou @sa_antoniou for health research. Lancet 2008;371:1149–50. 5 Equator. Equator resource centre. Available: https://www.​equator-​ Acknowledgements The authors wish to thank Mrs. Meropi Gioumidou for the network.​org/ tireless contribution and the excellent administrative support to the project. 6 Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing Contributors GAA: conception and design, interpretation of data, drafting the work, Guideline development, reporting and evaluation in health care. J final approval for the work to be published, agreement to be accountable for all Clin Epidemiol 2010;63:1308–11. 7 World Health Organization. WHO Handbook for Guideline aspects of the work in ensuring that questions related to the accuracy or integrity development, 2012. Available: http://​apps.​who.​int/​iris/​bitstream/​ of any part of the work are appropriately investigated and resolved. DM: conception 10665/​75146/​1/​9789241548441_​eng.​pdf [Accessed 2 Oct 2017]. and design, analysis of data, interpretation of data, drafting the work, final approval 8 NICE. The guidelines manual, Guidance and guidelines. Available: for the work to be published, agreement to be accountable for all aspects of the https://www.​nice.​org.​uk/​process/​pmg6/​chapter/​reviewing-​the-​ work in ensuring that questions related to the accuracy or integrity of any part evidence [Accessed 2 Oct 2017]. 6 Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 Open access 9 Antoniou SA, Agresta F, Garcia Alamino JM, et al. European Hernia 25 Pawson R, Greenhalgh T, Harvey G, et al. Realist review--a Society guidelines on prevention and treatment of parastomal new method of systematic review designed for complex policy hernias. Hernia 2018;22:183–98. interventions. J Health Serv Res Policy 2005;10 Suppl 1:21–34. 10 Muysoms FE, Antoniou SA, Bury K, et al. European Hernia Society 26 Wong G, Greenhalgh T, Pawson R. Internet-­Based medical guidelines on the closure of abdominal wall incisions. Hernia education: a realist review of what works, for whom and in what 2015;19:1–24. circumstances. BMC Med Educ 2010;10:12. 11 Gorter RR, Eker HH, Gorter-­Stam MAW, et al. Diagnosis and 27 Brouwers MC, Kerkvliet K, Spithoff K, et al. The AGREE reporting management of acute appendicitis. EAES consensus development checklist: a tool to improve reporting of clinical practice guidelines. conference 2015. Surg Endosc 2016;30:4668–90. BMJ 2016;352:i1152. 12 López-­Cano M, García-­Alamino JM, Antoniou SA, et al. EHS clinical 28 Chen Z, Hong Y, Liu N, et al. Quality of critical care clinical practice guidelines on the management of the abdominal wall in the context guidelines: assessment with agree II instrument. J Clin Anesth of the open or burst abdomen. Hernia 2018;22:921–39. 2018;51:40–7. 13 Morales-­Conde S, Peeters A, Meyer YM, et al. European association 29 Bazzano AN, Green E, Madison A, et al. Assessment of the quality for endoscopic surgery (EAES) consensus statement on single-­ and content of national and international guidelines on hypertensive incision endoscopic surgery. Surg Endosc 2019;33:996–1019. disorders of pregnancy using the agree II instrument. BMJ Open 14 Francis NK, Sylla P, Abou-­Khalil M, et al. EAES and SAGES 2016;6:e009189. 2018 consensus conference on acute diverticulitis management: 30 Armstrong JJ, Rodrigues IB, Wasiuta T, et al. Quality assessment evidence-­based recommendations for clinical practice. Surg Endosc of osteoporosis clinical practice guidelines for physical activity 2019;33:2726–41. and safe movement: an AGREE II appraisal. Arch Osteoporos 15 Schünemann H, Brożek J, Guyatt G. GRADE Handbook for grading 2016;11:6. quality of evidence and strength of recommendations. The GRADE 31 Sabharwal S, Patel NK, Gauher S, et al. High methodologic quality Working Group, 2013. but poor applicability: assessment of the AAOS guidelines using the 16 AGREE Next Steps Consortium. The AGREE II Instrument [Electronic AGREE II instrument. Clin Orthop Relat Res 2014;472:1982-8. version] 2009. 32 Huang T-­W, Lai J-­H, Wu M-­Y, et al. Systematic review of clinical 17 Prinja S, Nandi A, Horton S, et al. Costs, Effectiveness, and Cost-­ practice guidelines in the diagnosis and management of thyroid Effectiveness of Selected Surgical Procedures and Platforms. In: nodules and cancer. BMC Med 2013;11:191. Essential surgery: disease control priorities. Volume 1. Third Edition, 33 Wong JJ, Côté P, Sutton DA, et al. Clinical practice guidelines for 2015. the noninvasive management of low back pain: a systematic review 18 Jünger S, Payne SA, Brine J, et al. Guidance on conducting by the Ontario protocol for traffic injury management (OPTIMa) and reporting Delphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. collaboration. Eur J Pain 2017;21:201–16. Palliat Med 2017;31:684–706. 34 Acuna SA, Huang JW, Scott AL, et al. Cancer screening 19 Choi J, Choi T-­Y, Jun JH, et al. Preferred reporting items for the recommendations for solid organ transplant recipients: a systematic development of evidence-­based clinical practice guidelines in review of clinical practice guidelines. Am J Transplant 2017. traditional medicine (PRIDE-­CPG-­TM): explanation and elaboration. 35 van Dijk AH, de Reuver PR, Besselink MG, et al. Assessment of Eur J Integr Med 2016;8:905–15. available evidence in the management of gallbladder and bile 20 Norris SL, Meerpohl JJ, Akl EA, et al. The skills and experience of duct stones: a systematic review of international guidelines. HPB GRADE methodologists can be assessed with a simple tool. J Clin 2017;19:297–309. Epidemiol 2016;79:150–8. 36 Zeng M, Yi Q, Zeng L, et al. Quality of therapeutic drug monitoring 21 Moher D, Schulz KF, Simera I, et al. Guidance for developers of (TDM) guidelines is suboptimal: an evaluation using the AGREE II health research reporting guidelines. PLoS Med 2010;7:e1000217. instrument. J Clin Epidemiol 2019. 22 Antoniou SA, Tsokani S, Mavridis D, et al. Guideline assessment 37 Chen Y, Yang K, Marušic A, et al. A reporting tool for practice project: filling the gap in surgical guidelines. Ann Surg guidelines in health care: the right statement. Ann Intern Med 2019;269:642–51. 2017;166:128–32. 23 Wong G, Greenhalgh T, Westhorp G, et al. Development of 38 Wang X, Zhou Q, Chen Y, et al. Protocol of reporting items for methodological guidance, publication standards and training public versions of guidelines: the reporting tool for practice materials for realist and meta-­narrative reviews: the RAMESES guidelines in health Care-­public versions of guidelines. BMJ Open (realist and Meta-­narrative evidence syntheses – evolving standards) 2019;9:e023147. project. Health Serv Deliv Res 2014;2:1–252. 39 Song Y, Darzi A, Ballesteros M, et al. Extending the RIGHT statement 24 Pawson R. Evidence-­Based policy. London: SAGE Publications Ltd, for reporting adapted practice guidelines in healthcare: the RIGHT-­ 2006. Ad@pt Checklist protocol. BMJ Open 2019;9:e031767. Antoniou GA, et al. BMJ Open 2020;10:e037107. doi:10.1136/bmjopen-2020-037107 7