Open access Protocol BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. Defining decision thresholds for judgments on health benefits and harms using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence to Decision (EtD) frameworks: a protocol for a randomised methodological study (GRADE-­THRESHOLD) Gian Paolo Morgano,1,2 Lawrence Mbuagbaw ‍ ‍,1 Nancy Santesso,1,2 Feng Xie ‍ ‍,1 Jan L Brozek,1,2 Uwe Siebert,3,4,5 Antonio Bognanni,1 Wojtek Wiercioch,1,2 Thomas Piggott,1,2 Andrea J Darzi ‍ ‍,1,2 Elie A Akl ‍ ‍,6 Ilse M Verstijnen,7 Elena Parmelli ‍ ‍,8 Zuleika Saz-­Parkinson,8 Pablo Alonso-­Coello ‍ ‍,9 Holger J Schünemann1,2,10,11 To cite: Morgano GP, Protected by copyright. Mbuagbaw L, Santesso N, et al. Defining decision thresholds for judgments on health ABSTRACT Strengths and limitations of this study benefits and harms using the Introduction The Grading of Recommendations Grading of Recommendations Assessment, Development and Evaluation (GRADE) and ► The calculation of the decision thresholds will be Assessment, Development and similar Evidence to Decision (EtD) frameworks require its Evaluation (GRADE) Evidence based on empirical data. users to judge how substantial the effects of interventions to Decision (EtD) frameworks: ► We will use structured case scenarios to present are on desirable and undesirable people-­important health a protocol for a randomised survey participants with the information relevant to outcomes. However, decision thresholds (DTs) that could methodological study (GRADE-­ make their judgments. help understand the magnitude of intervention effects and THRESHOLD). BMJ Open ► We will employ a randomisation process to en- 2022;12:e053246. doi:10.1136/ serve as reference for interpretation of findings are not yet sure that case scenarios will be equally distributed bmjopen-2021-053246 available. across survey participants. The objective of this study is an approach to derive and ► We acknowledge that the survey requires effort and ► Prepublication history and use DTs for EtD judgments about the magnitude of health that this could impact test–retest reliability and ap- additional supplemental material benefits and harms. We hypothesise that approximate DTs plicability of the survey results which we overcome for this paper are available online. To view these files, could have the ability to discriminate between the existing in part by conducting a large trial. please visit the journal online four categories of EtD judgments (Trivial, Small, Moderate, (http://dx.doi.org/10.1136/​ Large), support panels of decision-­makers in their work, bmjopen-2021-053246). and promote consistency and transparency in judgments. study at the beginning of the trial. This statement reads: Methods and analysis We will conduct a methodological You are being invited to participate in a research project Received 12 May 2021 randomised controlled trial to collect the data that allow which aims to identify indicative DTs that could assist Accepted 07 February 2022 deriving the DTs. We will invite clinicians, epidemiologists, users of the GRADE EtD frameworks in making judgments. decision scientists, health research methodologists, Your input will be used in determining these indicative experts in Health Technology Assessment (HTA), members thresholds. By completing this survey, you provide consent © Author(s) (or their of guideline development groups and the public to that the anonymised data collected will be used for the employer(s)) 2022. Re-­use participate in the trial. Then, we will investigate the research study and to be summarised in aggregate in permitted under CC BY-­NC. No validity of our DTs by measuring the agreement between publication and electronic tools. commercial re-­use. See rights judgments that were made in the past by guideline panels PROTOCOL registration number NCT05237635. and permissions. Published by and the judgments that our DTs approach would suggest if BMJ. applied on the same guideline data. For numbered affiliations see Ethics and dissemination The Hamilton Integrated INTRODUCTION end of article. As advocated by the National Academy of Research Ethics Board reviewed this study as a quality Correspondence to improvement study and determined that it requires no Medicine of the United States (formerly the Dr Holger J Schünemann; further consent. Survey participants will be required to Institute of Medicine), the assessment of the ​schuneh@​mcmaster.​ca read a consent statement in order to participate in this benefits and harms of alternative care options Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246 1 Open access BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. (ie, interventions, actions) is an essential component a common understanding and promote consistency and of any decision-­ making process underlying guideline transparency in judgments. recommendations.1 This assessment should be explicit and include considerations around the probability, Conceptual approach magnitude and importance of health-­ related benefits In the proposed DTs approach, we will consider that and health-­related harms, and other desirable and unde- judgments on how substantial anticipated effects sirable consequences of the recommendation or deci- (health benefits and harms) are should be influenced sion.2 The Grading of Recommendations Assessment, by: (1) the size of the intervention’s effects on each Development and Evaluation (GRADE) Working Group outcome (eg, the probability of people who experi- has developed the Evidence to Decision (EtD) frame- ence benefit or harm); and (2) the value assigned works to help guideline developers use the evidence in to those outcomes by the people who are affected.5 a structured and transparent way and to ensure that they Under this assumption, we will collect data about the consider all the criteria relevant to their decisions.3 4 The association between the dyad composed of size of GRADE and other EtD frameworks require decision-­ intervention’s effects and value of the outcome on makers to evaluate explicitly the benefits and harms of one hand, and judgments on the magnitude of the alternative care options through separate judgments anticipated effects on the other. In accordance with based on the two following questions: ‘How substantial are the EtD frameworks, judgments on desirable effects the desirable anticipated effects (health benefits)?’, ‘How substan- and on undesirable effects will be collected separately tial are the undesirable anticipated effects (health harms)?’. The and should not account for any potential tradeoff guidance from the GRADE Working Group includes between benefits and harms. We will use this data to expressing and facilitating these judgments by assigning estimate the DTs and provide a conceptual framework the health benefits or health harms of some intervention for their interpretation and use (see online supple- under evaluation to one of the following four categories: mental file 1). ‘Trivial or None’, ‘Small’, ‘Moderate’ and ‘Large’.3 4 To be useful, however, this simplification requires that EtD METHODS Protected by copyright. users have a similar understanding of what magnitude of health benefits or health harms belong into which This study will consist of two parts. In the first part, we category and are consistent in their judgments. A similar will conduct a methodological randomised controlled common understanding is also important between those trial to collect the data that will be used to derive the assigning a category and those interpreting the meaning DTs. Second, we will investigate the validity of our DTs by of a category that is communicated to them (ie, ‘imag- measuring the agreement between judgments that were ining’ how substantial is an effect based on the category). made in the past by guideline panels and the judgments This can be achieved only when people make similar that our DTs approach would suggest if applied on the judgments. To direct EtD users on how to make these same guideline data. judgments appropriately, the GRADE Working Group Randomizsed controlled trial has produced guidance articles that include the descrip- The following description of methods and analysis of this tion of the underpinning concepts and examples of judg- trial follows the latest guidance by the Standard Protocol ments based on clinical scenarios.4 5 Despite the popular Items: Recommendations for Interventional Trials.13 We use of thresholds to support decision-­making in various registered this protocol in the Protocols Registration and fields of healthcare research,6–8 and its adoption by the Results System (​ clinicaltrials.​ gov, protocol registration GRADE approach,9 10 use of decision thresholds (DTs) number: NCT05237635). for EtD judgments about health benefits and harms is not yet established. For continuous outcomes, EtD users can Design and setting revert to statistical notions such as Cohen’s standardised Study participants will be recruited to complete a effect sizes or the minimal important difference to inter- randomised electronic survey (see online supplemental pret the magnitude of effects.11 12 However, empirical data file 2) designed to elicit ratings on the magnitude of the supporting judgments on health benefits and harms for potential health effects (benefits or harms) of interven- dichotomous outcomes are not yet available for the EtDs. tions. Ratings on health benefits and health harms will be collected separately. We will organise the survey into Objectives three sections: introduction and example, ratings and The objective of this study is to derive DTs for EtD judg- questions about respondent demographics. Ratings will ments on the magnitude of health benefits and harms. We be based on five outcomes having a different impact on hypothesise that DTs could discriminate between the four health (death, major ischaemic stroke, pulmonary embo- categories for EtD judgments. Explicit DTs, providing lism of moderate severity, diarrhoea of moderate severity an indication for which could be the appropriate judge- and mild nausea/vomiting) presented through descrip- ment for a given scenario, might have the potential to tive case scenarios. Each case scenario will include: support panels of decision-­makers in their work, facilitate (1) a GRADE Summary of Finding table14 providing 2 Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246 Open access BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. information about the PICO (Population, Interven- the lower bound for 'Small' will be considered as 'Trivial tion, Comparator, Outcome), the relative and absolute or None’, and any estimate above the upper bound of anticipated effects of the intervention and the certainty 'Moderate' will be considered as 'Large'. in the evidence; (2) a Health Outcome Descriptor15 describing key attributes of the outcome under consid- Outcomes eration including symptoms, time horizon, testing and The primary endpoints of this trial are the three DTs treatment and consequences; and (3) a measure of the (T1=DTTrivial/Small, T2=DTSmall/Moderate, T3=DT- impact on health of the outcome (also known as ‘value’ Moderate/Large) that would allow discriminating of the outcome or ‘health utility’ in health economics). between EtD judgments of ‘Trivial or None’ and ‘Small’, This measure will be expressed on a scale from 0 (being ‘Small’ and ‘Moderate’ and ‘Moderate’ and Large’, dead) to 1 (perfect health) which means that outcomes respectively. with a higher value are valued closer to perfect health as compared with outcomes with a lower value. For each Randomizsation outcome, we will include a case scenario descriptive of Randomisation will ensure that case scenarios will be desirable health effects and another one descriptive of equally distributed across survey participants to get undesirable health effects, for a total of 10 case scenarios balanced judgments on outcomes. It will reduce poten- across five outcomes. These scenarios differ in the descrip- tial confounding due to order effects and possible differ- tion of the severity of the outcome and the consequences ences between case scenarios (eg, clarity). Randomisation to represent clearly different values. will also avoid selection bias that could arise if allowing participants to select the case scenarios more familiar to Participants them. Selection criteria The target population of the survey will include clini- Sample size calculation cians, epidemiologists, decision scientists, health research We based our sample size calculation on the data methodologists, experts in HTA and members of guide- collected during pilot testing (n=15 participants). line working groups, but it will be open to the public too. Based on this data, we computed the mean thresholds Protected by copyright. Prior knowledge of the GRADE approach and experi- T1, T2 and T3 for each outcome separately and esti- ence with the EtD frameworks will not be required for mated that we need to recruit 1406 survey respondents participation. to demonstrate a difference of 15% of the mean with non-­overlapping 95% CIs. These computations were Patient and public involvement statement done using Winpepi.16 There was no direct dedicated patient or public involve- ment but patients and the public will participate in the Statistical methods survey and can provide feedback. Calculation of thresholds from survey ratings Recruitment We will use the ranges of ARD for judgments of We will distribute the survey through colleagues, the 'Small' and 'Moderate' collected through the survey to research group’s e-­ mail lists including that of the calculate the thresholds associated with each rating. Cochrane Collaboration, Guidelines International The thresholds will be derived through the product Network, guideline developers and of the Global between each ARD indicated as range boundary and Evidence Synthesis Initiative. Twitter, LinkedIn and the difference in value from perfect health (1 — other social medial platforms will also be used for outcome’s value) for the outcome associated with broader distribution. We will continue recruitment that rating (see online supplemental file 2). We will for this trial until reaching our anticipated sample size calculate the DTs as the weighted mean of the corre- (see below) or until 31 December 2022 as it is unlikely sponding thresholds derived from survey ratings. that we will meet the sample size through additional We will use a weighted mean to account for multiple recruitment efforts beyond then. ratings from the same survey respondent. Intervention and comparison Primary analysis Participants will be randomised to a set of four case We will use frequencies and percentages to describe scenarios, written in lay language, that will be used as the characteristics of survey respondents. For each intervention (or comparison) in this trial. For each case DT, we will calculate mean, SD and 95% CIs. We will scenario, we will ask survey participants to consider the conduct an analysis of variance (ANOVA) to determine intervention’s effects and the value of the outcome and if there are any differences between the thresholds rate how substantial the described health benefits or (T1≠T2≠T3). If we identify a difference, since each health harms are. We will also ask them to indicate the participant will contribute data to each threshold, we lower and upper bound for the ranges of magnitudes of will employ a post-­hoc paired sample t-­test to assess absolute risk difference (ARD) that they associate with the which of the DTs are different that is, (T1≠T2; T2≠T3; judgments of 'Small' and 'Moderate'. Any estimate below T1≠T3). Our a-­ priori hypothesis is that there will Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246 3 Open access BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. be a difference between the DTs and no difference collected either electronically or by voice recordings and between the magnitude of DTs for benefits and harms. discussed during study meetings. Furthermore, to test the feasibility of the study, we recruited 75 participants from Sensitivity analyses the target population. Participants were able to complete Subgroup analysis the exercise in the majority of cases. Only 7 out of 75 did We will conduct explorative subgroup analyses based on not complete the survey after they signed up. Participants participants’ characteristics (training in epidemiology, contributed a total of 295 ratings with only 17 out of 312 familiarity with the EtD frameworks, previous participa- expected ratings missing indicating that the approach tion in guideline development groups, language used). to obtaining DTs is feasible. This is true for people of Our a-­priori hypotheses is that, in each of the identified varying backgrounds and educational levels. The findings subgroups, there will be a difference between the DTs and based on the preliminary analysis of the data support our no difference between the magnitude of DTs for benefits hypothesis that DTs can help discriminate between the and harms. judgments (see online supplemental file 3). Furthermore, we will use periodic interim results to inform judgments Incoherent ratings and outliers by guideline groups that develop recommendations but We expected that, given the complexity of the topic, some will not use these to draw final conclusions about the trial responses might not be internally coherent or represent results until it is stopped formally by reaching the calcu- outliers. We define a threshold as incoherent if T1>T2 or lated sample size or on 31 December 2022. No additional T2>T3. We define thresholds as outliers if they fall more data are available. than three IQRs below the first quartile or above the third quartile. We will verify if the primary analysis would differ if incoherent thresholds or data outliers are excluded. The a-­priori hypothesis for the sensitivity analyses will be ETHICS AND DISSEMINATION the same as for the primary analysis. After review, the Hamilton Integrated Research Ethics Board (HiREB) determined that as a quality improve- Order effects ment project, this study was exempt from formal ethics Protected by copyright. We will conduct an ANOVA analysis to assess for poten- review as per TCPS2 (2014) Article 2.5. We will inform tial order effects. We will examine whether participants respondents of this decision and the anonymous nature randomised to a case scenario for a low-­value outcome of the study. Survey participants will be required to read a (outcome value <0.5) in the first case-­scenario provided consent statement in order to participate in this study at different thresholds as compared with participants who the beginning of the trial. This statement reads: You are were randomised to a high-­ value outcome first. Simi- being invited to participate in a research project which larly, we will examine whether participants who provided aims to identify indicative DTs that could assist users of a judgement of ‘Small’ in the first iteration provided the GRADE EtD frameworks in making judgments. Your different thresholds as compared with participants who input will be used in determining these indicative thresh- provided a judgement of ‘Large’ in the first iteration. Our olds. By completing this survey, you provide consent a-­priori hypothesis is that of no differences if comparing that the anonymised data collected will be used for the each DT between these groups. research study and to be summarised in aggregate in publication and electronic tools. Retrospective comparison of judgments The results of this randomized trial will be published in To investigate the validity of our DTs, we will purposively a peer-­reviewed journal. We also aim to present the results select judgments from existing guidelines developed using in national and international conferences. the EtD frameworks and measure the agreement between judgments made by guideline panels and the judgments that our DTs approach would suggest. We will consider for inclusion guidelines reporting the value assigned to DISCUSSION outcomes during the decision-­making process. We will We believe that DTs for judgments on desirable and use frequencies and percentages to describe the agree- undesirable health effects can be useful to decision-­ ment. We will employ SPSS V.26 (IBM Corp, Armonk, makers using the EtD frameworks. Guideline panels New York, USA) to conduct all statistical analyses. We will using the GRADE EtDs often ask what are ‘Trivial or use the Bonferroni correction for multiple testing in all None’, ‘Small’, ‘Moderate’ and ‘Large’ effects. The secondary analyses.17 proposed DTs approach could provide an answer based on empirical data and be used to initiate and Pilot testing and assessment of feasibility promote discussion. Furthermore, it is simple to apply, To ensure usability and clarity of the survey across respon- and requires only to calculate the product between dents having different background or expertise, we ARD and the reduction in value associated with the piloted the survey with study co-­investigators as well as outcome. This endeavour will expand the research on complementary representatives of the target population the use of DTs within the GRADE methodology and (n=15). Comments on three iterations of the survey were could be integrated into GRADEpro. 4 Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246 Open access BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. Our work with Hultcrantz et al10 suggests that clinical DTs Contributors HJS is the principal investigator and conceived of the study and can be used to allow appropriate ratings of the certainty together with GPM, NS, FX and JLB designed and established this research project. LM, IMV, TP, EP, ZS-­P, AB and US piloted the survey and provided methodological of the evidence, but there is no empirical data. Further- input. GPM was responsible for the ethics application and with HJS for registration more, it focuses on the construct of certainty of evidence of the protocol on ​clinicaltrials.​gov. HJS, GPM, AB and LM designed the statistical and targets different degrees of contextualisation, while analysis. GPM, AB, WW, AJD and HJS are responsible for recruitment. HJS, WW we address judgments on the magnitude of effects and and GPM are responsible for the coordination of the study. GPM and HJS drafted the manuscript. NS, FX, AB and JLB reviewed early drafts. EAA and PA-­C provided made by users of the EtD frameworks. The joint consider- methodological input in the study design. All listed authors participated in the ation of the estimate of effect and outcome’s importance writing and revision of the manuscript and approved its final version. has been already adopted in another effort of the GRADE Funding The authors have not declared a specific grant for this research from any Working Group. In a GRADE concept paper,18 Alper et al funding agency in the public, commercial or not-­for-­profit sectors. aim to define the certainty in the net benefit and suggest Competing interests HJS is the co-­chair of the GRADE working group. Decision calculating the net effect of an intervention by combining thresholds will be used in the GRADEpro app and for other projects. Currently no importance-­ adjusted effect estimates calculated from financial interests. HJS and JLB are codevelopers of the GRADEpro app. US is an unpaid member of Working Group for the German Clinical S3 Guideline Prevention different outcomes. While this strategy is appealing and of Cervical Cancer; Committee for Cancer Screening of the Austrian Federal Ministry would allow us to apply our research to EtD judgments on of Health; Oncology Advisory Council of the Federal Ministry of Health, Austria. the trade-­off between benefits and harms, further research Patient consent for publication Not applicable. is needed to establish if the estimates to be combined are Provenance and peer review Not commissioned; externally peer reviewed. independent and not correlated with each other. Other Supplemental material This content has been supplied by the author(s). It has quantitative approaches to assess the benefits, harms and not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been net benefit associated with treatments are available in the peer-­reviewed. Any opinions or recommendations discussed are solely those literature,19 but none aims to characterise the magnitude of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and of effects into categories (ie, ‘Trivial or None’, ‘Small’, responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability ‘Moderate’, ‘Large’) as needed to make judgments using of the translations (including but not limited to local regulations, clinical guidelines, the EtD frameworks. Utilitarian frameworks are common terminology, drug names and drug dosages), and is not responsible for any error in health economic research, where health-­utilities elic- and/or omissions arising from translation and adaptation or otherwise. ited from target populations are used to inform model- Open access This is an open access article distributed in accordance with the Protected by copyright. ling techniques such as cost-­effectiveness analysis based Creative Commons Attribution Non Commercial (CC BY-­NC 4.0) license, which on quality-­adjusted-­life-­years.20 21 However, our trial will permits others to distribute, remix, adapt, build upon this work non-­commercially, and license their derivative works on different terms, provided the original work is not be free of limitations. Generalisability of the findings properly cited, appropriate credit is given, any changes made indicated, and the use may be limited by the use of the case scenarios we chose is non-­commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. and the limited number of effect sizes we include in the ORCID iDs trial. Generalisability may also be limited by the type of Lawrence Mbuagbaw http://orcid.org/0000-0001-5855-5461 participants we will be able to recruit. Therefore, we plan, Feng Xie http://orcid.org/0000-0003-3454-6266 following the completion of this trial, to conduct further Andrea J Darzi http://orcid.org/0000-0002-2498-1697 research with additional case scenarios and different Elie A Akl http://orcid.org/0000-0002-3444-8618 Elena Parmelli http://orcid.org/0000-0002-3730-8530 target populations. Pablo Alonso-­Coello http://orcid.org/0000-0002-8001-8504 Author affiliations 1 Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada REFERENCES 2 Michael G. DeGroote Cochrane Canada & McMaster GRADE Centres, McMaster 1 Institute of Medicine Committee on Standards for Developing University, Hamilton, Ontario, Canada Trustworthy Clinical Practice G. Clinical practice guidelines we can 3 Institute of Public Health, Medical Decision Making and Health Technology trust. Washington (DC): National Academies Press (US), 2011. 2 Schünemann HJ. Guidelines 2.0: do no net harm-­the future of Assessment, Department of Public Health, Health Services Research and practice Guideline development in asthma and other diseases. Curr Technology Assessment, UMIT – University for Health Sciences, Medical Informatics Allergy Asthma Rep 2011;11:261–8. and Technology, Hall in Tirol, Austria 3 Atkins D, Eccles M, Flottorp S, et al. Systems for grading the quality 4 of evidence and the strength of recommendations I: critical appraisal Division of Health Technology Assessment and Bioinformatics, ONCOTYROL - Center for Personalized Cancer Medicine, Innsbruck, Austria of existing approaches the grade Working group. BMC Health Serv 5 Res 2004;4:38. Center for Health Decision Science/Department of Health Policy and 4 Alonso-­Coello P, Schünemann HJ, Moberg J, et al. Grade evidence Management, Harvard TH Chan School of Public Health and MGH-­ITA/Department to decision (ETD) frameworks: a systematic and transparent of Radiology, Harvard Medical School, Harvard University, Boston, Massachusetts, approach to making well informed healthcare choices. 1: USA introduction. BMJ 2016;353:i2016. 6 Department of Internal Medicine, American University of Beirut, Beirut, Lebanon 5 Alonso-­Coello P, Oxman AD, Moberg J, et al. Grade evidence to 7 Dutch National Health Care Institute, Diemen, The Netherlands decision (ETD) frameworks: a systematic and transparent approach 8 to making well informed healthcare choices. 2: clinical practice European Commission Joint Research Centre, Ispra, Italy guidelines. BMJ 2016;353:i2089. 9 Iberoamerican Cochrane Centre - Department of Clinical Epidemiology and 6 Marcucci M, Sinclair JC. A generalised model for individualising a Public Health, Biomedical Research Institute Sant Pau (IIB Sant Pau), Barcelona, treatment recommendation based on group-­level evidence from Spain randomised clinical trials. BMJ Open 2013;3:e003143. 10 Department of Medicine, McMaster University, Hamilton, Ontario, Canada 7 McCabe C, Claxton K, Culyer AJ. The NICE cost-­effectiveness 11 Dipartimento di Scienze Biomediche, Humanitas University, Milan, Italy threshold: what it is and what that means. Pharmacoeconomics 2008;26:733–44. 8 Pauker SG, Kassirer JP. Therapeutic decision making: a cost-­benefit Twitter Feng Xie @fengxie_mac analysis. N Engl J Med 1975;293:229–34. Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246 5 Open access BMJ Open: first published as 10.1136/bmjopen-2021-053246 on 10 March 2022. Downloaded from http://bmjopen.bmj.com/ on November 30, 2023 at Sant Paul Hospital Consortia. 9 Hultcrantz M, Mustafa RA, Leeflang MMG, et al. Defining ranges for 16 Abramson JH. WINPEPI updated: computer programs for certainty ratings of diagnostic accuracy: a grade concept paper. J epidemiologists, and their teaching potential. Epidemiol Perspect Clin Epidemiol 2020;117:138–48. Innov 2011;8:1. 10 Hultcrantz M, Rind D, Akl EA, et al. The grade Working group 17 Abdi H. Holm’s sequential Bonferroni procedure. In: Salkin N, clarifies the construct of certainty of evidence. J Clin Epidemiol ed. Encyclopedia of research design. Thousand Oaks, California, 2017;87:4–13. 11 Faraone SV. Interpreting estimates of treatment effects: implications 2010: 1–8. for managed care. P T 2008;33:700–11. 18 Alper BS, Oettgen P, Kunnamo I, et al. Defining certainty of net 12 Johnston BC, Ebrahim S, Carrasco-­Labra A, et al. Minimally benefit: a grade concept paper. BMJ Open important difference estimates and methods: a protocol. BMJ Open 2019;9:e027445. 2015;5:e007953. 19 Puhan MA, Singh S, Weiss CO, et al. A framework for organizing 13 Chan A-­W, Tetzlaff JM, Gøtzsche PC, et al. Spirit 2013 explanation and selecting quantitative approaches for benefit-­harm assessment. and elaboration: guidance for protocols of clinical trials. BMJ BMC Med Res Methodol 2012;12:173. 2013;346:e7586. 20 Siebert U. When should decision-­analytic modeling be used in the 14 Guyatt GH, Thorlund K, Oxman AD, et al. Grade guidelines: 13. economic evaluation of health care? Eur J Health Econ preparing summary of findings tables and evidence profiles-­ continuous outcomes. J Clin Epidemiol 2013;66:173–83. 2003;4:143–50. 15 Baldeh T, Saz-­Parkinson Z, Muti P, et al. Development and use of 21 Weinstein MC, Stason WB. Foundations of cost-­effectiveness health outcome descriptors: a Guideline development case study. analysis for health and medical practices. N Engl J Med Health Qual Life Outcomes 2020;18:167. 1977;296:716–21. Protected by copyright. 6 Morgano GP, et al. BMJ Open 2022;12:e053246. doi:10.1136/bmjopen-2021-053246