Elvira et al. BMC Health Services Research (2021) 21:814 https://doi.org/10.1186/s12913-021-06784-8 RESEARCH ARTICLE Open Access Description of the use of multicriteria to support pricing and reimbursement decisions by European health technology assessment bodies David Elvira1 , Mercè Obach2 and Caridad Pontes1,2* Abstract Background: Heterogeneity in drug access throughout Europe may be influenced by differences in drug- assessment strategies. The EUnetHTA’s assessment core model (EUnetHTA-core) and the EVIDEM’s multicriteria framework are reference methodologies in this context, the latter including a wider compromise between non- contextual and contextual criteria. Compliance of 37 European Health Technology Assessment bodies (HTAb) with EUnetHTA-core has been reported, but the use of EVIDEM by this HTAb is still unknown. Methods: To describe the uptake and use of multicriteria approaches to evaluate drug value by European HTAb using EVIDEM as reference framework, a multicriteria framework was obtained based on EVIDEM model. The criteria used for drug appraisal by HTAb was extracted from the EUnetHTA report, and completed through search of websites, publications and HTAb reports. Use of EVIDEM assessment model in 37 European HTAb has been described semi-quantitatively and summarized using an alignment heatmap. Results: Aligned, medium or misaligned profiles were seen for 24,3%, 51,4% and 24,3% of HTAb when matching to EVIDEM dimensions and criteria was considered. HTAb with explicit responsibilities in providing specific advice on reimbursement showed more aligned profiles on contextual and non-contextual dimensions. Conclusions: EUnetHTA’s core model is limited in assessing medicines while EVIDEM’s framework provides contextual dimension used by some HTAb in Europe that can be escalated to other agencies. Most of the 37 European HTAb have room to broaden their contextual assessment tools, especially when social and medical perception of need requires to be explicit to support payer’s decision on reimbursement. Keywords: Health technology assessment, Multicriteria assessment methods, Reimbursement systems * Correspondence: caridad.pontes@uab.cat 1 Departament de Farmacologia, de Terapèutica i de Toxicologia, Universitat Autònoma de Barcelona, Sabadell (Barcelona), Spain 2 Servei Català de la Salut, Gerència del Medicament, Barcelona, Spain © The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Elvira et al. BMC Health Services Research (2021) 21:814 Page 2 of 12 Introduction terms of medical and economic costs (opportunity costs One of the major cost drivers in the European health- of using alternative more efficient treatments available care systems is the pharmaceutical ‘innovation’; even for patients). Several initiatives have been developed in considered more relevant than demographics [1]. At the Europe to address these challenges of funding premium same time, it is also recognized as one of the main con- priced products related to clear medical unmet needs tributors to the improvement of the population health but with limited evidence [10]. New access management status [2]. models of these drugs have been promoted across Eur- According to the most recent study from the ope recently, especially for advance therapies, orphan Organization for Economic Co-operation and Develop- drugs and medicines for cancer, and including innova- ment (OECD) [3], pharmaceutical expenditure accounts tive access schemes as value-based pricing, conditional for a percentage that range between 11.4% (UK) and reimbursement schemes or risk sharing approaches [11]. 19.1% (Spain) of total healthcare expenditure across the Despite the smooth increase of these new access five largest European drug markets (France, Germany, schemes, the number of outcome-based solutions is still Italy, Spain, and the UK). Specifically, the oncological very limited being the lack of a systematic and harmo- and hematological drugs are leading the budget impact nized value assessment methodology one of the main related to pharmaceutical innovation. The impact is limitations [12]. driven by the expansion of multiple new indications nor- Beyond the general awareness among healthcare author- mally based on a molecular definition that restricts the ities to ensure “value for money”, or the link between price population to be treated and the drug ends up being and social or clinical value of the pharmaceutical designated as orphan-like medicines [4]. As estimated by innovation [13], the reimbursement process and value as- a recent study [5, 6], the healthcare expenditures on can- sessment of drugs is still an open debate in Europe [14]. cer in the European Union member states represented Several methods have been developed to assess the value roughly 6% of total healthcare expenditures. The steady of drugs and set meaningful prices affordable to health- increase of oncology costs is aligned with the disease in- care systems [15]. These methods are normally based on creasing incidence, the progressive reduction of mortal- the clinical benefits of the drugs and partially on value- ity as well as high prices, in contrast with the less robust based pricing (e.g. cost-effectiveness analysis). However, evidence data on outcomes [7]. there is neither a consensus nor a European A recent study [5] estimated that 40% of the new or- harmonization related to drug-pricing systems and, based phan drugs authorized in Europe are related to rare neo- on a comparative international policy analysis, value-based plastic disorders, and compare to non-oncologic approaches to determine the prices of innovative products indications, the authorization is received at more ad- are diverse [16]: including the implicit clinical value of the vance stages of the clinical development and recognizing quality-adjusted life-years (QALYS), mainly used in UK, a higher potential clinical benefit. From 2009 to 2013, Sweden or Australia, or the value classification based on only 35% the 68 oncology indications approved by the innovation scales (used in France, Italy, Germany, Austria, European Medicines Agency (EMA) showed a significant Canada or Japan) [17]. Normally new drugs classified as prolongation of survival and only 10% showed an im- an innovative medicine are reimbursed at a higher price provement in quality of life at the time of market ap- than the current therapeutic alternatives; although the proval. The magnitude of the benefit on overall survival amount, type and methodology to set the premium is nor- ranged from 1.0 to 5.8 months (median 2.7 months). In mally veiled by the healthcare authorities [7]. the subsequent post marketing period (3,3 years later) In Europe, the European Network for Health Technol- there was evidence for extension of life in 7% of the pre- ogy Assessment (EUnetHTA) was set up in 2006 and in- vious authorizations and reported benefit on quality of cludes all EU Member States to provide strategic guidance life in 11% of the cases [8]. and policy orientation on the assessment of health tech- Occasionally, when the drug can cover clinical unmet nologies (including drugs), by developing policy papers needs with poor prognosis, the regulators trend to and discussing areas of potential collaboration. During the accept less and poorer evidence and include especial ap- last decade the network has focused the efforts on the provals, such as conditional approval related to further development of common methodologies, piloting and of adequate risk benefit rate in real world, after producing joint early dialogues and Health Technology commercialization, or approval under exceptional cir- Assessment (HTA) joint assessment reports, as well as cumstances when this may not be achieved, in order to developing and maintaining common tools [18]. One of ensure an earlier access to market. As described recently the most relevant tools developed by the network is the [9] the potential benefit of patients’ early access to new HTA Core Model for Rapid Relative Effectiveness medicines in areas of high unmet medical need, and Assessment (REA) [19]. The Model is a methodological based on initial data only, have relevant implications in framework for the collaborative production and sharing of Elvira et al. BMC Health Services Research (2021) 21:814 Page 3 of 12 HTA information that defines the content elements to be with EVIDEM framework of HTAs as an additional way considered in an HTA and it enables standardized assess- to explore potential reasons of assessment discrepancies. ment reporting across Europe. Because of the objective of Despite the compliance of 37 European HTAb with the framework is to share commonly required elements of using the supportive criteria for decision making pro- information, only information that is considered both im- posed in the EUnetHTA-core framework has been previ- portant and transferable is collected. The model brings a ously reported [18], whether these HTAb do also standardized framework that allows a common compari- comply with the wider EVIDEM multicriteria is son of the drivers that lead pricing and reimbursement de- unknown. cisions among different European authorities. Thus, the main aim of this study is to describe the up- HTA Network approach is focused on technical as- take and use of multicriteria approaches to appraise drug pects while methods to support alignment of deci- value by 37 European HTAb, using EUnetHTA and EVI- sions with the compassionate impetus of healthcare DEM as reference frameworks. systems is lacking [20]. In many countries, healthcare authorities are including a broader approach to assess the pharmaceutical products (especially in therapeutic Methods areas like oncology and rare diseases) [21]. EVIDEM A quantitative validation of the degree of alignment with [22] (Evidence and Value Impact on Decision Making) the EUnetHTA’s standard framework of 37 European was developed based on an analysis of the founda- HTAb from 28 countries was done, based on a previous tions of healthcare systems, the reasoning underlying qualitative analysis conducted by the European Commis- decisions and fair processes, and has become a refer- sion [18] and an additional thorough search of websites, ence for multicriteria decision approaches in this set- publications and reports of HTAb. The criteria used for ting. It includes the concept of reflective multicriteria appraisal by the different HTAb were identified and clas- assuming decision-makers are guided by a generic sified, and the matching with the criteria described in interpretative frame rooted in the baseline values of the EVIDEM methodological framework were described the healthcare systems, drawing on several domains of semi-quantitatively using a heatmap of alignment. knowledge including healthcare ethics, evidenced- The items reported included those criteria in the HTA based medicine, health economics or health technol- Core Model, namely: Relative Effectiveness Assessment ogy assessment approaches. A multicriteria analysis (REA) of pharmaceuticals, EUnetHTA methodological provides an effective approach to increase the legitim- guidelines [25] and procedure descriptions [26, 27]. Also, acy of decisions. Beyond a tool, reflective multicriteria criteria related to the types of technologies assessed, the pioneered by EVIDEM is geared to transform the administrative level (national, regional, institutional) and vision of the value of healthcare interventions and the formal background (legislation, formal agreement, how they might contribute to relevant, equitable and internal guideline) of certain methodological require- sustainable healthcare systems. EVIDEM can be used ments were also used. to compare various healthcare interventions and An updated version of EVIDEM framework (v.10) was prioritize its implementation using a performance analyzed in order to assess how the dimensions and cri- matrix underpinned in the several dimensions and teria included in the EUnetHTA methodological frame- criteria defined by the framework [20]. work fitted within the EVIDEM’s methodological EVIDEM criteria overlap with EUnetHTA-core except framework. for 4 non-contextual and 3 social criteria, which are The EVIDEM framework includes 13 non-contextual absent or partially included in the EUnetHTA frame- dimensions and 6 contextual dimensions (Table 1). The work. Inversely, 2 EUnetHTA criteria are absent in the non-contextual dimensions (EVIDEM core-model) in- EVIDEM framework (Table 1). clude normative aspects combined with the description Although multicriteria EVIDEM approach is now ap- of the technical knowledge available. Contextual dimensions plied by several healthcare authorities [23], especially tailor the framework to the context of decision-making. when the social and medical perception of need requires An HTAb heatmap was developed, where heatmap a more holistic assessment framework to support the categories were generated for each EVIDEM’s dimension payer’s decision, a formal and systematic comparison of using as a source the mentioned criteria in the EUnetH- EUnetHTA’s and EVIDEM’s methodological frameworks TA’s report [18], webs and reports available from the and whether European health technology assessment different HTAb analyzed (supplementary file). The con- bodies (HTAb) are aligned with the EVIDEM method- tribution (weight) of each mentioned criterion to the ology standards is lacking [24]. Since EUnetHTA and final heatmap’s score by dimension was equal and pro- EVIDEM frameworks differ in a substantial number of portioned to the number of criteria by dimension de- criteria, it is of interest to know the extent of compliance scribed in Table 1. Only when the mentioned criteria Elvira et al. BMC Health Services Research (2021) 21:814 Page 4 of 12 Table 1 EVIDEM and EUnetHTA criteria correspondence EVIDEM CRITERIA EUnetHTA CRITERIA NON-CONTEXTUAL CRITERIA Disease severity • Effect of disease on life-expectancy Methodology requirements for the clinical assessment compared • Effect of disease on morbidity (includes disability and to the HTA Core Model for REA - SEVERITY DEFINITION function) • Effect of disease on patients’ quality of life A description of the health problem and current use of technology are included in assessments • Effect of disease on caregivers’ quality of life Size of affected • Prevalence Methodology requirements for the clinical assessment compared population • Incidence to the HTA Core Model for REA - POPULATION A description of the health problem and current use of technology are included in assessments Unmet needs • Unmet needs in efficacy A description of the health problem and current use of • Unmet needs in safety technology are included in assessments • Unmet needs in patient reported outcomes Systematic search strategies applied to evidences (HEALTH • Patient demand PROBLEM - CURRENT TECHNOLOGY USE) Comparative • Magnitude of health gain The comparator is supported by evidence on its efficacy profile effectiveness • Percentage of the target population expected to for the respective clinical indication/population achieve the anticipated health gain Assessments analyze clinical effectiveness / efficacy (added • Onset and duration of health gain therapeutic value) • Sub-criteria for the measure of efficacy specific to the therapeutic area Systematic search strategies applied to evidences (EFFICACY- EFFECTIVENESS) Comparative safety/ • Adverse events The comparator is supported by evidence on its safety profile for tolerability • Serious adverse events the respective clinical indication/population • Fatal adverse events • Short-term safety Assessments analyze safety • Long-term safety Systematic search strategies applied to evidences (SAFETY) • Tolerability Comparative patient- • Improvement in health-related quality of life QALYs applied perceived health • Impact on autonomy Assessments analyze patient aspects • Impact on dignity • Convenience / ease of use / mode & setting of Assessments include a separate ethical analysis administration Systematic search strategies applied to evidences (PATIENT ASPECTS) Type of preventive • Eradication, prevention, reduction in disease Not available benefit transmission, reduction in the prevalence of risk factors). Public health perspective. Type of therapeutic • Symptom relief, prolonging life, cure Assessments include a description of the health problem and benefit current use of technology Comparative cost • Net cost of intervention Assessments analyze cost, budget impact or include economic consequences – cost • Acquisition cost evaluation of intervention • Implementation/ maintenance cost Comparative cost • Impact on primary care expenditures Assessments analyze cost, budget impact or include economic consequences – other • Impact on hospital care expenditures evaluation medical costs • Impact on long-term care expenditures Comparative cost • Impact on productivity Assessments analyze social aspects consequences – non- • Financial impact on patients medical costs • Financial impact on caregivers • Costs to the wider social care system Quality of evidence • Validity (study design, agreement among studies) Sources of evidence included as relevant clinical evidence for • Relevance (population, disease stage, outcomes) the clinical assessment (1- randomized controlled; 2- • Completeness of reporting (uncertainty, conflicting Nonrandomized prospective; 3- Other observational; 4- Expert results across studies, limited number of studies) Opinion). • Type of evidence Methodology requirements for the clinical assessment compared to the HTA Core Model for REA Formal tools or algorithms for evidence grading applied The GRADE approach in routine use Plan for how evidence will be synthesized (e.g. evidence tables, Elvira et al. BMC Health Services Research (2021) 21:814 Page 5 of 12 Table 1 EVIDEM and EUnetHTA criteria correspondence (Continued) EVIDEM CRITERIA EUnetHTA CRITERIA meta-analysis, qualitative synthesis) Tables and forms are standardized for evidence synthesis and analysis Evidence analysis include surrogate endpoints, composite endpoints, PROs, HRQoL measures, indirect comparisons, meta- analysis, relevant group sub-population, key deficiencies in avail- able data, transferability issues, summary of findings Sources of evidence on the technology: A. scientific journal publications, B. grey literature (e.g. published reports), C. unpublished data, D. register data, E. administrative data, F. manufacturer data Confidential data from manufacturers accepted Expert consensus/ Current consensus of experts on what constitutes state- Not available clinical practice of-the-art practices (guidelines guidelines CONTEXTUAL CRITERIA Mandate and scope Alignment with healthcare plans/systems Circumstances where HTA reports are provided of the healthcare system Population priorities • Current priorities of health system (e.g. low Assessments analyze social aspects and access socioeconomic status; specific age groups) • Special populations (e.g. ethnicity) • Remote communities • Rare diseases • Specific therapeutic areas Common goal and • Stakeholder pressures Assessments analyze social aspects specific interests • Stakeholders barriers • Conflict of interest Environmental impact • Environmental impact of production Not available • Environmental impact of use • Environmental impact of implementation • Environmental impact of production • Environmental impact of use • Environmental impact of implementation System capacity and • Organizational requirements (e.g., process, premises, Assessments include a separate ethical analysis appropriate use of equipment) intervention • Skill requirements Assessments analyze legal aspects • Legislative requirements Assessments analyze organizational aspects • Surveillance requirements • Risk of inappropriate use • Institutional limitations to uptake Political/historical/ • Political priorities and context Assessments include a separate ethical analysis cultural context • Cultural acceptability • Precedence (congruence with previous and future decisions) • Impact on innovation & research • Impact on partnership & collaboration among healthcare stakeholders Source: reference [20]. GRADE Grading of Recommendations, Assessment, Development and Evaluations, HTA Health Technology Assessment, HRQoL Health Related Quality of Life, PROs Patient Reported Outcomes, QALY Quality Adjusted Life Years, REA Relative Effectiveness Assessment were not fully aligned with the EVIDEM’s criteria, the 1 and 2, and conditional formatting was used to auto- mention was weighted by 50% of contribution: matically color code each cell using Microsoft Excel (Windows Office 365) so that graded colors were used h X  X i Heat Score ¼ #criteria mentioned by dimension = #total criteria by dimension 100 with green coding for highest alignment (100) and red for lowest alignment (0). Values outside the interquartile Descriptive statistics (mean, standard deviation, range were used to assess alignment with the EVIDEM’s percentiles) were used to summarize the data and 95% model [28]. HTAbs with and average heat score above confidence interval for each dimension and HTAb Figs. the 75th percentile were considered “Aligned” with the Elvira et al. BMC Health Services Research (2021) 21:814 Page 6 of 12 Fig. 1 EVIDEM alignment score by dimension EVIDEM model, and those below 25th percentile were dimensions (such as population priorities, common goal, considered “Misaligned”. The rest were classified as environmental impact, system capacity or political/his- “Medium” in terms of EVIDEM model’s alignment. torical/cultural context) are systematically rated low (Table 2). Results All HTAb address consistently the health problem and Most of the non-contextual criteria of EVIDEM are current use of technology, technical characteristics, clin- overlapped with the core model of EUnetHTA, except ical effectiveness and safety criteria, which are included for the type of prevention benefits, non-medical com- in the EUnetHTA core model. Choices on comparator, parative cost consequences, systematic use of expert methodology of comparison, endpoints and methods of consensus and use of clinical guidelines to define state- evidence search and synthesis, are consistently aligned. of-the-art, which are not or partially included on the On the contrary, non-clinical domains, assessment ap- EUnetHTA’s framework (Table 1). Regarding contextual proaches, methodology, modelling algorithms and data criteria, the assessment of the system capacity and ap- are consistently dis-aligned (Table 3). propriate use of intervention is the most aligned criteria None of the local HTAb had high heat scores with between both frameworks, followed by the political/his- regards to the use of contextual criteria (Table 2). Con- torical/cultural context assessment, the mandate and sidering alignment to EVIDEM-driven assessment scope of the healthcare system, the special population framework, three patterns of HTAs emerged: “Aligned”, priorities and equity on access criteria. Other social cri- “Medium” and “Misaligned” (Table 4). teria (stakeholders management, conflict of interest as- Nine agencies in Bulgary, Hungary, Italy, Malta, sessment or environmental impact assessment) are not Slovakia, Spain, Sweden and UK showed an “Aligned” reflected in the EUnetHTA’s framework. A systematic profile (average heat score above the 75th percentile) general description of the assessed technology and the with a consistent alignment on non-contextual dimen- request of clarification of the assessment process (guide- sions and significantly high alignment scores on polit- lines and legislation) are key aspects considered by the ical/historical/cultural context, system capacity and EUnetHTA analysis that are not explicitly included in appropriate use of the intervention. the EVIDEM framework. Most HTAb (19/37; 51%) showed a “Medium” align- Most of the non-contextual dimensions (such as dis- ment profile. Alignment rates for non-contextual criteria ease severity, size of affected population, unmet needs, were mainly high (e.g. patient perceived health and qual- comparative effectiveness, comparative safety/tolerability ity of evidence dimensions) in these HTAb, and also or type of therapeutic benefit) show consistently high other contextual dimensions (such as the mandate and rates among the HTAb (mean above 85% and standard scope of the healthcare system, system capacity and ap- deviation below 16%); other non-contextual dimensions propriate use of the intervention) were rated high. On (type of preventive benefit, comparative non-medical the contrary, population priorities and access dimension costs, expert consensus) and relevant contextual systematically rated below 50%, except for AEMPS. Elvira et al. BMC Health Services Research (2021) 21:814 Page 7 of 12 Fig. 2 (See legend on next page.) Elvira et al. BMC Health Services Research (2021) 21:814 Page 8 of 12 (See figure on previous page.) Fig. 2 EVIDEM alignment score by HTAb. HTAb: Health Technology Assessment body. SESCS: Servicio de Evaluación del Servicio Canario de Salud; SBU: Swedish Agency for Health Technology Assessment and Assessment of Social Services; HVB: Hauptverband der Österreichischen Sozialversicherungsträger; KCE: Belgian Health Care Knowledge Centre; INAMI-RIZIV: National Institute for Health and Disability Insurance; NCPHA: National Center of Public Health and Analyses; SUKL: State Institute for Drug Control; FIMEA: Finnish Medicines Agency; HAS: Haute Autorité de Santé; IQWiG: Institute for Quality and Efficiency in Health Care; OGYÉI: National Institute of Pharmacy and Nutrition; HIQA: Health Information and Quality Authority; NCPE: National Centre for Pharmacoeconomics; AIFA: Italian Medicines Agency; UCSC: Università Cattolica del Sacro Cuore; ZIN: Zorginstituut Nederland; AOTMiT: Agencja Oceny Technologii Medycznych i Taryfikacji; INFARMED: National Authority of Medicines and Health Products; UHIF: Union Health Insurance Fund; AEMPS: Agencia Española de Medicamentos y Productos Sanitarios; AETSA: Agencia de Evaluación de Tecnologías Sanitarias de Andalucía; AQUAS: Agència de Qualitat i Avaluació Sanitàries de Catalunya; TLV: Dental and Pharmaceutical Benefits Agency; NICE: National Institute for Health and Care Excellence; SMC: Scottish Medicines Consortium; AWTTC: All Wales Therapeutics and Toxicology Centre; NIPH: Norwegian Institute of Public Health; NoMA: Norwegian Medicines Agency; LBI-HTA: Ludwig Boltzmann Institute of Health Technology Assessment; AAZ: Agency for Quality and Accreditation in Health Care and Social Welfare; UoT: University of Tartu; G-BA: Gemeinsamer Bundesausschuss; RER: Regione Emilia-Romagna; NVD: The National Health Service; MOH: Ministry of Health Malta; MOH: Ministry of Health Slovakia; JAZMP: Agency for Medicinal Products and Medical Devices In 9/37 (24%) HTAb the profile was considered and quality of the evidence. Considering the non- “Misaligned”, with low scores on alignment (average contextual perspective, the German G-BA and the NIPH score below 25th percentile) in dimensions such as in Norway show high scores focused and limited to the patients perceived health methods, cost-consequence technical comparison of alternatives (effectiveness, safety analysis (cost of intervention and other medical costs) and quality of evidence assessment). From the Table 2 HTAb heatmap of coincidence with EVIDEM framework HTA Health Technology Assessment, HRQoL Health Related Quality of Life, PROs Patient Reported Outcomes, QALY Quality Adjusted Life Years, REA Relative Effectiveness Assessment Color code using Microsoft Excel (Windows Office 365). Graded colors were used with green coding for highest alignment (100) and red for lowest alignment (0) Elvira et al. BMC Health Services Research (2021) 21:814 Page 9 of 12 Table 3 EVIDEM heat score by dimension Criteria Mean Standard Low 95% CL Upper 95% CL 25th 75th Deviation Mean Mean Percentile Percentile Disease severity 87.2 17.3 50.0 100.0 75.0 100.0 Size of affected population 88.5 16.2 50.0 100.0 75.0 100.0 Unmet needs 95.9 12.5 50.0 100.0 100.0 100.0 Comparative effectiveness 99.5 2.7 83.3 100.0 100.0 100.0 Comparative safety/tolerability 92.8 14.5 50.0 100.0 100.0 100.0 Comparative patient-perceived health / PRO 51.7 21.9 12.5 100.0 37.5 62.5 Type of preventive benefit 0.0 0.0 0.0 0.0 0.0 0.0 Type of therapeutic benefit 94.6 15.7 50.0 100.0 100.0 100.0 Comparative– cost of intervention 74.3 32.5 0.0 100.0 50.0 100.0 Comparative – other medical costs 74.3 32.5 0.0 100.0 50.0 100.0 Comparative – non-medical costs 6.8 5.3 0.0 20.0 0.0 10.0 Quality of evidence 64.0 23.4 22.2 100.0 44.4 83.3 Expert consensus/clinical practice guidelines 0.0 0.0 0.0 0.0 0.0 0.0 Contextual criteria 0.0 0.0 0.0 0.0 0.0 0.0 Mandate and scope of the healthcare system 51.1 13.9 30.0 60.0 30.0 60.0 Population priorities and access 23.5 14.4 0.0 60.0 30.0 30.0 Common goal and specific interests 7.8 5.8 0.0 20.0 0.0 10.0 Environmental impact 0.0 0.0 0.0 0.0 0.0 0.0 System capacity & appropriate use of 25.9 16.3 0.0 53.3 13.3 40.0 intervention Political/historical/cultural context 19.5 17.6 0.0 60.0 0.0 30.0 GLOBAL 47.9 5.8 31.2 60.5 44.7 51.25 contextual perspective, all the HTAb of this group rated description and current use of technology; are consist- low on the mandate and scope of the healthcare system, ently addressed by all HTAb. As previously reported population priorities on access, system capacity, appro- [18] the institutions go only partially beyond these cri- priate use of the interventions and political/historical/ teria and it is normally dependent on the topic of assess- cultural context. ment. For those European HTAb directly advising on HTAb with explicit responsibilities in providing spe- price and reimbursement decisions, the reported criteria cific advice on pricing and reimbursement (normally re- used to support their decisions show a more balanced gional agencies in countries with more than one HTAb alignment between both methodological approaches. in place, such as Belgian KCE, German IQWIG, Irish That conclusion could explain why in many cases, the HIQA, Italian UCSC, Portuguese INFARMA, Slovakian subnational HTAb in those countries with multiple UHIF, Spanish SESCS or Swedish SBU) showed higher agencies, are the ones showing a balanced profile among and similar scores on contextual and non-contextual contextual and non-contextual dimensions. dimensions. EVIDEM provides a generic interpretive frame (MCDA – Multi-Criteria Decision Analysis – reflective grid) that Discussion and conclusions can be used to elicit individual values and facilitate delib- The alignment between EVIDEM and EUnetHTA meth- erations through a common structure that includes inter- odological frameworks is consistently high, especially pretive scores (quantitative criteria), qualitative impacts when assessing domains related to health problem de- (qualitative criteria) as well as narrative comments (all cri- scription, current use of the technology, technical char- teria) [21]. EVIDEM framework was designed to minimize acteristics, clinical effectiveness, and safety. However, the limitations of the deliberation process by ensuring other non-contextual dimensions of the EVIDEM frame- that: generic assessment criteria (either quantitative or work and the EUnetHTA core model are consistently qualitative) are included; evidence relevant to each criter- misaligned. ion is made available through an efficient synthesis meth- The main EUnetHTA core model criteria, such as clin- odology; and face validity is checked at each step of the ical effectiveness, safety conditions, health problem process (weights, scores and corresponding narratives, Elvira et al. BMC Health Services Research (2021) 21:814 Page 10 of 12 Table 4 EVIDEM heat score by HTAb HTAb Mean Standard Deviation Low 95% CL Mean Upper 95% CL Mean Degree of Alignment with EVIDEM model HVB 46.4 44.3 26.5 66.3 Medium LBI 43.0 37.8 26.0 60.0 Misaligned KCE 50.9 40.2 32.9 69.0 Medium INAMI 48.8 39.0 31.2 66.3 Medium NCPHA 52.4 42.1 33.5 71.3 Aligned AAZ 48.3 40.0 30.3 66.2 Medium SUKL 43.3 44.0 23.5 63.0 Misaligned UoT 47.8 46.8 26.7 68.8 Medium FIMEA 50.0 40.2 31.9 68.1 Medium HAS 49.3 39.2 31.6 66.9 Medium G-BA 31.2 36.8 14.6 47.7 Misaligned IQWIG 49.0 44.0 29.2 68.8 Medium OGYEI 55.9 43.3 36.4 75.4 Aligned HIQA 50.1 40.5 31.9 68.3 Medium NCPE 44.7 45.0 24.4 64.9 Misaligned AIFA 44.7 44.5 24.7 64.7 Medium RER 36.2 45.1 15.9 56.5 Misaligned UCSC 53.3 42.6 34.2 72.5 Aligned NVD 41.9 39.5 24.2 59.7 Misaligned MOH 52.1 42.3 33.1 71.1 Misaligned ZIN 49.3 38.9 31.9 66.8 Medium AOTMiT 47.0 41.0 28.6 65.5 Medium INFARMED 45.3 35.0 29.5 61.0 Medium MOH 34.4 44.1 14.6 54.2 Aligned UHIF 53.9 41.4 35.3 72.5 Aligned JAZMP 43.1 38.3 25.9 60.3 Misaligned AEMPS 52.5 38.4 35.2 69.7 Aligned AETSA 49.8 38.9 32.4 67.3 Medium SESCS 60.5 43.2 41.1 80.0 Aligned AQUAS 51.3 42.5 32.1 70.4 Medium SBU 53.0 40.9 34.6 71.4 Aligned TLV 46.4 45.2 26.1 66.7 Medium NICE 49.2 42.6 30.0 68.3 Medium SMC 52.4 44.8 32.3 72.5 Aligned AWTTC 50.6 44.3 30.7 70.5 Medium NIPH 42.3 36.5 25.9 58.8 Misaligned NOMA 51.2 44.6 31.2 71.2 Medium GLOBAL 47.9 5.8 46.0 49.7 Medium HTAb Health Technology Assessment body, SESCS Servicio de Evaluación del Servicio Canario de Salud, SBU Swedish Agency for Health Technology Assessment and Assessment of Social Services, HVB Hauptverband der Österreichischen Sozialversicherungsträger, KCE Belgian Health Care Knowledge Centre, INAMI-RIZIV National Institute for Health and Disability Insurance, NCPHA National Center of Public Health and Analyses, SUKL State Institute for Drug Control, FIMEA Finnish Medicines Agency, HAS Haute Autorité de Santé, IQWiG Institute for Quality and Efficiency in Health Care, OGYÉI National Institute of Pharmacy and Nutrition, HIQA Health Information and Quality Authority, NCPE National Centre for Pharmacoeconomics, AIFA Italian Medicines Agency, UCSC Università Cattolica del Sacro Cuore, ZIN Zorginstituut Nederland, AOTMiT Agencja Oceny Technologii Medycznych i Taryfikacji, INFARMED National Authority of Medicines and Health Products, UHIF Union Health Insurance Fund, AEMPS Agencia Española de Medicamentos y Productos Sanitarios, AETSA Agencia de Evaluación de Tecnologías Sanitarias de Andalucía, AQUAS Agència de Qualitat i Avaluació Sanitàries de Catalunya, TLV Dental and Pharmaceutical Benefits Agency, NICE National Institute for Health and Care Excellence, SMC Scottish Medicines Consortium, AWTTC All Wales Therapeutics and Toxicology Centre, NIPH Norwegian Institute of Public Health, NoMA Norwegian Medicines Agency, LBI-HTA Ludwig Boltzmann Institute of Health Technology Assessment, AAZ Agency for Quality and Accreditation in Health Care and Social Welfare, UoT University of Tartu, G-BA Gemeinsamer Bundesausschuss, RER Regione Emilia-Romagna, NVD The National Health Service, MOH Ministry of Health Malta, MOH Ministry of Health Slovakia, JAZMP Agency for Medicinal Products and Medical Devices Elvira et al. BMC Health Services Research (2021) 21:814 Page 11 of 12 aggregated measures). EVIDEM framework is sufficiently diseases: Practical implications of the unpolitical declaration. Lancet. 2013; flexible to be adapted to the local assessment context, al- 381(9867):680–9. 3. OECD. Pharmaceutical spending (indicator); 2018. https://doi.org/10.1787/ though it also requires consistency in the identification of 998febf6-en. Accessed 5 July 2018 a set of criteria, scoring scale and weights when assessing 4. Godman B, Bucsics A, Vella Bonanno P, Oortwijn W, Rothe CC, Ferrario A, a broad range of competing interventions in a specific et al. Barriers for access to new medicines: searching for the balance between rising costs and limited budgets. Front Public Health. 2018;6:328. local context [29, 30]. https://doi.org/10.3389/fpubh.2018.00328. A holistic approach is required to consistently assess 5. Kawalec P, Sagan A, Pilc A. The correlation between HTA recommendations the social and medical needs to support payer’s decision and reimbursement status of orphan drugs in Europe. Orphan J Rare Dis. 2016;11(1):122. https://doi.org/10.1186/s13023-016-0501-4. on prices and reimbursement conditions of certain 6. Jönsson B, Hofmarcher T, Lindgren P, Wilking N. The cost and burden of drugs, such as disruptive innovations or orphan drugs, cancer in the European Union 1995–2014. Eur J Cancer. 2016;66:162–70. broadening the need of using EVIDEM-like contextual https://doi.org/10.1016/j.ejca.2016.06.022. 7. Cohen D. Cancer drugs: high price, uncertain value. BMJ. 2017;359:j4543. assessment tools by European HTAb. 8. Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs Abbreviations approved by European medicines agency: retrospective cohort study of EMA: European Medicines Agency; EUnetHTA: European Network for Health drug approvals 2009-13. BMJ. 2017;359:j4530. Technology Assessment; EVIDEM: Evidence and Value Impact on Decision 9. Pontes C. et al,. Time to review authorisation and funding for new cancer Making; HTA: Health Technology Assessment; HTAb: Health Technology medicines in Europe? Inferences from the case of Olaratumab. Appl Health Assessment Bodies; MCDA: Multi-Criteria Decision Analysis; Econ Health Policy 2019. https://doi.org/10.1007/s40258-019-00527-x. OECD: Organization for Economic Co-operation and Development; 10. Godman B, Malmstrom RE, Diogene E, Gray A, Jayathissa S, Timoney A, et al. QUALYS: Quality-Adjusted Life-Years; REA: Relative Effectiveness Assessment Are new models needed to optimize the utilization of new medicines to sustain healthcare systems? Expert Rev Clin Pharmacol. 2015;8(1):77–94. Supplementary Information https://doi.org/10.1586/17512433.2015.990380. The online version contains supplementary material available at https://doi. 11. Pauwels K, Huys I, Vogler S, Casteels M, Simoens S. Managed entry org/10.1186/s12913-021-06784-8. agreements for oncology drugs: lessons from the European experience to inform the future. Front Pharmacol. 2017;8:171. 12. Piatkiewicz TJ, Traulsen JM, Holm-Larsen T. Risk-sharing agreements in the Additional file 1. EU: a systematic review of major trends. Pharmacoecon Open. 2018;2(2): 109–23. https://doi.org/10.1007/s41669-017-0044-1. 13. Tuffaha H, Aitken J, Chambers S, Scuffham P. A framework to prioritise Acknowledgements health research proposals for funding: integrating value for money. Appl Not applicable. Health Econ Health Policy. 2019;17(6):761–70. 14. Uyl-de Groot CA, Löwenberg B. Sustainability and affordability of cancer Authors’ contributions drugs: a novel pricing model. Nat Rev Clin Oncol. 2018;15(7):405–6. https:// All authors contributed to the study conception and design. Material doi.org/10.1038/s41571-018-0027-x. preparation, data collection and analysis were performed by David Elvira and 15. Chandra A, Shafrin J, Dhawan R. Utility of cancer value frameworks for Mercè Obach. The first draft of the manuscript was written by David Elvira patients, payers, and physicians. JAMA. 2016;315(19):2069–70. https://doi. and all authors commented on previous versions of the manuscript. All org/10.1001/jama.2016.4915. authors read and approved the final manuscript. 16. Wilsdon T, Attridge J, Fiz E. Assessing the value of biopharmaceitical innovation in key therapy areas in middle-income countries; 2013. Funding 17. Hidalgo-Vega A. Evaluación, financiación y regulación de los medicamentos No funding was received for the conduct of the study. innovadores en los países desarrollados; 2018. 18. Kristensen FB. Mapping of HTA methodologies in EU and Norway. In: Availability of data and materials European Commission, Science & Policy; 2017. All data generated or analysed during this study are included in this 19. Kristensen F, Lampe K, Wild C, Cerbo M, Goettsch W, Becla L. The HTA core published article and its supplementary information files (supplementary model®—10 years of developing an international framework to share file.xls). multidimensional value assessment. Value Health. 2017;20(2):244–50. 20. Goetghebeur MM, et al. Can reflective multicriteria be the new paradigm Declarations for healthcare decision-making? The EVIDEM journey. Cost Eff Resour Alloc. 2018;16(Suppl 1):54. Ethics approval and consent to participate 21. Friedmann C, Levy P, Hensel P, Hiligsmann M. Using multi-criteria decision Not applicable. analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoecon Outcomes Res. 2017;18(2):135–46. https://doi.org/10.1080/14 Consent for publication 737167.2018.1414603. Not applicable. 22. Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson L, Rindress D. Evidence and Value: Impact on DEcisionMaking - The EVIDEM framework Competing interests and potential applications. BMC Health Serv Res. 2008;8:270. David Elvira is a full-time employee of Sanofi. Caridad Pontes and Mercè 23. Guarga L, Badia X, Obach M, Fontanet M, Prat A, Vallano A, et al. Obach are public employees with no other conflicts of interest. Implementing reflective multicriteria decision analysis (MCDA) to assess orphan drugs value in the Catalan Health Service (CatSalut). Orphan J Rare Received: 11 February 2021 Accepted: 12 July 2021 Dis. 2019;14(1):1–9. 24. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUNetHTA core model in Lombardia: the VTS framework. References Int J Technol Assess Health Care. 2014;30(1):105–12. https://doi.org/10.1017/ 1. Newhouse JP. Medical care costs: how much welfare loss? J Econ Perspect. S0266462313000639. 1992;6(3):3–21. https://doi.org/10.1257/jep.6.3.3. 25. EUnetHTA. Methods for health economic evaluations-A guideline based on 2. Hogerzeil H, Liberman J, Wirtz V, Kishore S, Selvaraj S, Kiddell-Monroe R, current practices in Europe. www.eunethta.eu/wp-content/uploads/2018/ et al. Promotion of access to essential medicines for non-communicable 03/Methods_for_health_economic_evaluations.pdf. Accessed 21 Apr 2020. Elvira et al. BMC Health Services Research (2021) 21:814 Page 12 of 12 26. Guyatt G, Oxman A, Akl E, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. 27. Jaeschke R, Guyatt G, Dellinger P, Schünemann H, Levy M, Kunz R, et al. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ. 2008;337(7665):327–30. 28. Vicent B, Wiitala W, Luginbill K, Molling D, Hofer T, Ryan A, et al. Template matching for benchmarking hospital performance in the veterans affairs healthcare system. Medicine. 2019;98(20):1–9. 29. Youngkong S, Tromp N, Chitama D. The EVIDEM framework and its usefulness for priority setting across a broad range of health interventions. Cost Effect Res Alloc. 2011;9(1):8. https://doi.org/10.1186/1478-7547-9-8. 30. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making—An introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19:1–13. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.