Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 https://doi.org/10.1186/s13023-019-1121-6 RESEARCH Open Access Implementing reflective multicriteria decision analysis (MCDA) to assess orphan drugs value in the Catalan Health Service (CatSalut) Laura Guarga1,2, Xavier Badia2, Mercè Obach1, Manel Fontanet1,3, Alba Prat1, Atonio Vallano1, Josep Torrent1 and Caridad Pontes1,3* Abstract Background: Orphan medicines show some characteristics that hinder the evaluation of their clinical added value. The often low level of evidence available for orphan drugs, together with a high budget impact and an incremental cost-effectiveness ratio many times higher than drugs used for non-orphan diseases, represent challenges in their appraisal and effective access to clinical use. In order to explore how to handle these hurdles, the Catalan Health Service (CatSalut) began an initiative on a multidimensional assessment of drugs value during the appraisal process. Reflective multicriteria decision analysis (MCDA) using analytical methods was chosen, since it may help to standardise and contextualize all the relevant data related with the drug that could contribute to a decision. The aim of the study was to determine whether the implementation of reflective MCDA methodology could support the decision-making process about orphan medicines in the context of CatSalut. Methods: The assessment and decision-making process for orphan drugs in the Programa d’Harmonització Farmacoterapeutica (PHF) of CatSalut was prioritized to test the implementation of the reflective MCDA both a qualitative and quantitatively. A staged approach was used with the following main steps: selection and structuration of quantitative criteria (Core Model) and qualitative criteria (Contextual Tool), framework scoring and assessment of three orphan drug case studies. This proof-of-concept would grant a continued refinement of the methodology and, if and when validated, its potential integration to other therapeutic areas of the PHF. Results: The final framework was composed by 10 quantitative criteria (Core Model) and 4 qualitative criteria (Contextual Tool) according to the PHF goals being the most important criteria “disease severity”, “unmet need”, “comparative effectiveness” and “comparative safety /tolerability”. The matrix developed for the case studies served as a guide for the selection of the essential information that the decision-makers were expected to include in a framework. The reflective discussion was considered the most relevant phase of the approach to support inputs for health decision-making processes reflecting both drug value and place in therapy. (Continued on next page) * Correspondence: cpontes@catsalut.cat 1 Àrea del Medicament, Servei Català de la Salut (CatSalut), Travessera de les Corts, 131-159, 08028 Barcelona, Spain 3 Departament de Farmacologia, de Terapèutica i de Toxicologia, Universitat Autònoma de Barcelona, Av. de Can Domènech, 737, 08193 Cerdanyola del Vallès (Barcelona), Spain Full list of author information is available at the end of the article © The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 2 of 9 (Continued from previous page) Conclusions: The study showed that reflective MCDA methodology could be implemented to complement the decision-making process in CatSalut, as an aid to determine the clinical added value for orphan medicines. MCDA provided transparency and a structured discussion during the committee meetings, thus increasing transparency and predictability of the relevant items supporting the agreements adopted on orphan drugs access. Keywords: Multi-criteria decision analysis, Catalan healthcare, Orphan drugs, Decision-making Introduction CatSalut was compared with the reflective MCDA meth- In Spain, the decision for price and reimbursement of odology and was adapted to the local context. Finally, it therapeutic innovations in the basic services portfolio of was concluded that reflective MCDA could be a useful the public health care system is a national responsibility. and feasible tool to complement the current evaluation However, the other health competences are mainly methods of CatSalut, contributing to standardisation dependent on the autonomous communities who are and pragmatism, providing a method to tackle ethical responsible for the budget allowance, prioritization and dilemmas and facilitating discussions related to decision- development of measures to ensure efficient access of making [7]. This initiative was included in the Health new drugs considering the characteristics of their own Plan of Catalonia for 2016–2020 [2]. population [1]. Thus, the aim of the present study was to assess the In Catalonia, the Catalan Health Care System (Servei feasibility of the implementation of a reflective MCDA Català de la Salut, CatSalut) is the regional health care methodology to support orphan drug decision-making institution responsible for ensuring public access and process to the Catalan context. rights for health delivery. CatSalut runs a specific program for drug evaluation and decision-making, Methods Programa d’Harmonització Farmacoterapeutica (PHF) A qualitative study was carried out with a staged ap- aimed to prioritize innovative drugs according to their proach. The following main steps were used: 1) selection added value and considering the needs and budgeting and structuration of quantitative criteria (MCDA core priorities, while to guaranteeing equity in the access to Model) and qualitative criteria (Contextual Tool), 2) treatments throughout Catalonia. In addition to establish weighting of the quantitative criteria of the framework the relative effectiveness, PHF also specifies the clinical and 3) testing through assessment of three orphan drug criteria recommendations and the place in therapy of case studies. the new drug in relation to available therapies [2]. A dedicated committee composed by 18 members (6 Selection and structuration of criteria for orphan drugs physicians, 6 healthcare service managers, 5 pharmacists framework and 1 patient representative) was specifically devoted to An updated specific MCDA framework for orphan decision-making regarding of Orphan Drugs at the time medicines evaluation was developed considering the of the present exercise. suggested quantitative and qualitative criteria of the pilot To address the PHF goals, CatSalut began an initiative framework previously developed (December 2015), [7] and in 2015 focused on a multidimensional assessment of additional criteria identified through the following sources: drugs value [3]. Among all the multicriteria approaches, the reflective multicriteria decision analysis (MCDA) – Updated version of EVIDEM framework (v.4): New was chosen since, through analytical methods, helps to considerations of the EVIDEM framework which is contextualize all the relevant data related with the continually updated by the EVIDEM collaboration product and which could contribute to a decision [4]. group to support pragmatic and reflective healthcare Reflective MCDA framework (EVIDEM) is composed by decision-making [8] a set of criteria derived from the ethical imperatives of – Systematic Literature Review (SLR): Criteria from a healthcare and their relative importance under a trans- SLR conducted by the Instituto Carlos III (ISCIII). parent and standardised process which promote the The study identifies and analyses the information reflection of the stakeholders and facilitate the sharing about the reimbursement criteria and of diverse perspectives [5, 6]. pharmaceutical policies used for orphan drugs The first step was focused on adapting and assessing decision-making applied in the main countries of the the value of a decision reflective MCDA framework for European Union, Australia and Canada. Their search orphan drugs evaluation from CatSalut perspective. The strategy covered literature published between to standard evaluation and decision-making procedure of 2004 and 2015 [3]. Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 3 of 9 The abovementioned resources were analysed in order orphan drugs matrix for each drug assessment, consider- to assess the feasibility of their adaptation for the inclu- ing the Catalan healthcare context. sion in the current PHF. The analysis was performed by two of the authors (XB and LG). Data analysis The final version of the framework was discussed and Data were collected individually, transferred to a validated indicating whether a criterion should or should common database and analysed with Microsoft Excel. A not be systematically considered when appraising a descriptive analysis of the value of each criterion was orphan drug in the setting of the PHF by 5 members of conducted separately. Non-comparative criteria were PHF (physicians, pharmacists and health service man- rated in a 0 (worst) to 5 (best) scale and comparative cri- agers). Definition of criteria is similar to EVIDEM teria (efficacy, safety, PRO (Patient Reported Outcomes) framework focused on orphan drugs [8]. and cost) was rated in a scale from − 5 to 5. For each criterion, the mean, standard deviation (SD) and range Weighting of orphan drugs framework (value system of minimum and maximum scores were calculated. The elicitation) value contribution (Vx) of each quantitative criterion The weighting of the quantitative criteria of the final was then calculated as the product of its normalised orphan drugs framework was done by a subgroup of scoring (Wx, ∑ Wx = 1) and standardised score (non representatives of the committee of CatSalut in charge comparative criteria: Sx = score/5 or comparative criteria: of decision-making processes (clinicians and hospital Sx = score/10). The overall MCDA value estimate (V) of pharmacist during a workshop performed on July 2017 each orphan drugs was calculated based on a linear in Barcelona). This subgroup was selected to enable the additive model as a sum the value contributions (Vx) of properly field testing of the orphan drug framework and all (n) criteria of the quantitative criteria [8, 10]: was trained on reflective MCDA methodology previously to participate in the panel. X n X n A direct rating scale (five-point weighting technique) VE ¼ VCx ¼ ðWx  Sx Þ x¼1 x¼1 was used where each participant gave a relative weight per criterion using a nonhierarchical simple 5-point scale (1 = lowest relative importance, 5 = highest relative importance) [8, 9]. Results Selection and structuration of the criteria for orphan drugs framework Data analysis The pilot framework developed in the previous study [7] All data analysis was carried out in Microsoft Excel. considers a set of criteria structured into two distinct Criteria weights were normalized to sum up to 1 for sections: MCDA Core Model, composed of 13 quantita- each participant: for the 5-point rating scale method, tive criteria selected for the assessment of the drug and each weight was divided by the sum of weights across all a MCDA Contextual tool, composed of 4 qualitative criteria; for the point allocation method, criteria ratings criteria that consider the context surrounding decision- were multiplied by domain weight and rescaled to range making (Table 1). from 0 to 1 [8]. In addition, seven new criteria were analysed by the PHF members in order to decide their introduction or Case studies – appraisal of three orphan drugs modification in the updated version of the framework Three orphan drugs approved in Europe were chosen to (Table 2). test the suitability of the proposed MCDA framework to All the criteria from Table 2 were accounted to assess capture all the relevant dimensions and criteria used for if they could complement the orphan drugs framework. discussion. The medicines selected were chosen from Some considerations were provided by PHF members medicines already scheduled for assessment by the PHF, regarding the goals and content of the program: and considering the fact that the 3 drugs differed in terms of size of target population, availability of treat- – Size of affected population: The ethical basis of this ment alternatives, quantity and quality of the supporting criterion reflects an aspect of the utilitarianism evidence and economical impact, among others, The principle (greatest good for greatest number). available evidence was summarized for each orphan drug Orphan drugs target small populations. Then, this in the evidence matrix, following the reflective MCDA principle, albeit important, need to be considered as methodology [8]. a low priority, in order to minimise differences During the workshop performed on July 2017, the par- between rare and ultra-rare diseases. It was agreed ticipating PHF members rated individually the three to exclude the criteria. Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 4 of 9 Table 1 Criteria considered in the pilot framework Criteria Type of criteria Source Disease severity Quantitative Previous study [7] Size of affected population Quantitative Previous study [7] Unmet needs Quantitative Previous study [7] Comparative effectiveness Quantitative Previous study [7] Comparative safety/tolerability Quantitative Previous study [7] Comparative patients perceived health (PRO) Quantitative Previous study [7] Type of preventive benefit Quantitative Previous study [7] Type of therapeutic benefit Quantitative Previous study [7] Cost of orphan drug Quantitative Previous study [7] Other medical costs Quantitative Previous study [7] Non-medical costs Quantitative Previous study [7] Quality of evidence Quantitative Previous study [7] Expert consensus / clinical practice guidelines Quantitative Previous study [7] Population priorities and access Qualitative Previous study [7] Common goal and specific interests Qualitative Previous study [7] Opportunity costs and affordability Qualitative Previous study [7] System capacity and appropriate use of intervention Qualitative Previous study [7] PRO Patient Reported Outcomes – Preventive benefit: the vast majority of rare diseases – Budget Impact: Relevant criteria for drug assessment are of genetic origin and interventions are therefore in PHF process. It was agreed to include the criteria symptomatic or curative rather than preventive. It to the domain “feasibility contextual criteria” was concerted to exclude the criteria. (Contextual tool). Furthermore, its inclusion – Therapeutic benefit: Based on its definition, the supports the exclusion of the criteria “size of the nature of the clinical benefit provided for patient- population”, which would be covered with the level intervention (i.e, symptom relief, prolongation financial/budgeting exercise. of the life, healing), the criterion was maintained in – Rarity: The concept of common disease, rare or the framework. However, it will be redefined so that it ultra -rare, is closely related with the number of captures the therapeutic benefit in a broad and far- patients affected. To be consistent with the reaching way, including aspects such as reduction of exclusion of “size of affected population” criteria it risk factors, reduction in disease transmission, etc. was agreed not to include this criterion. – Non-medical costs: Although non-medical expenses – The rule of rescue: The criterion describes the are highly relevant to rare diseases, the lack of infor- perceived duty to save endangered life whenever mation and the difficulty to quantify them makes possible although the drug is not indicated or them difficult to consider. It was agreed to exclude cost-effective. It was agreed to exclude the the criteria. criteria. Table 2 Additional criteria considered for orphan drugs framework Criteria Type of criteria (quantitative/qualitative) Source Size of affected population Quantitative Previous study [7] Preventive benefit Quantitative Previous study [7] Therapeutic benefit Quantitative Previous study [7] Non-medical costs Quantitative Previous study [7] Budget Impact Quantitative or qualitative EVIDEM framework v.4 [8] and SLR [3] Rarity Quantitative or qualitative SLR [3] The rule of rescue Quantitative or qualitative SLR [3] SLR Systematic Literature Review Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 5 of 9 The final orphan drugs framework validated by partici- perception of their impact on mortality and pants is shown as Table 3. morbidity. Alpha – 1 antitrypsin was not considered to have an added value versus standard of care in Weighting of orphan drugs framework (value system regards to efficacy, safety and quality of life elicitation) (QoL). Regarding the type of benefit, alpha – 1 According to the results collected from the 5-point antitrypsin was considered as not providing an weighting scale (Fig. 1), the more important criteria for added therapeutic benefit, considering orphan drugs were “disease severity and unmet needs” improvements on clinical variables or imaging (4,7 points ± 0,5) followed by “comparative effectiveness” abnormalities. Comparative treatment cost and (4,6 points ± 0,5) and “comparative safety/ tolerability” other medical costs were assessed with a neutral (4,4 points ± 0,7). The least important criteria were “ex- score. Related to its inclusion in the international pert consensus /clinical practise guidelines” (2,4 points ± guidelines and quality of evidence for alpha – 1 0,7) and “quality of evidence” (2,8 points ± 1,2). antitrypsin was considered moderate, because of uncertainty in the methodology and results of the Case studies – appraisal of 3 orphan drugs clinical trial publication. The next step involved in the implementation approach – Gaucher disease was considered a very severe of the framework in the drug review process was disease reflecting a high estimated mortality and appraising three orphan drugs: alpha 1-antitrypsin for co-morbidities because of impairment of relevant alpha1-antitrypsin deficiency, eliglustat for Gaucher organs such as liver, spleen, bone marrow and disease and tolvaptan for autosomal dominant polycystic bones. Some enzyme replacement therapies have kidney disease. been approved for Gaucher disease: imiglucerase, The matrix developed for the case studies served as a velaglucerase and miglustat. Eliglustat is an oral guide for the relevant information that decision-makers treatment that was considered similar to the expect to include in a framework. previously approved drugs in regards to the The overall scores achieved for each orphan drug, therapeutic efficacy, but with the advantage of oral including the contribution of the individual criteria route despite a worst safety profile, although scores to the total are depicted in Figs. 2, 3, 4 and 5. possibly better than the other oral alternative miglustat. The QoL outcomes were seen as – Disease severity of the alpha – 1 antitrypsin translating this fact in a benefit for the patients. deficiency was considered high, reflecting the Treatment cost per patient and other medical costs Table 3 MCDA Core Model (criteria appraised quantitatively) and MCDA Contextual Tool (criteria appraised qualitatively) MCDA Core Model Domains Criteria Disease Impact Disease severity Unmet needs Comparative outcomes of orphan drugs Improvement of efficacy/effectiveness Improvement of safety/ tolerability Improvement of patient perceived health/PRO Type of therapeutic benefit Economic consequences of intervention Annual patient cost of treatment Other medical costs Knowledge about intervention Quality of evidence Expert consensus/clinical practice guidelines MCDA Contextual Tool Domains Criteria Normative contextual criteria Population priorities and access (principle of equity) Common goal and specific interests Feasibility contextual criteria System capacity and appropriate use of orphan drugs Opportunity costs and affordability (budget impact) MCDA Multiple Criteria Decision Analysis Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 6 of 9 Fig. 1 Mean (SD) of level of importance rated by participants to final orphan drugs framework. SD: standard deviation; PRO: Patient Reported Outcomes Fig. 3 Value of contribution of each criteria: alpha – 1 antitrypsin vs BSC. PRO: Patient Reported Outcomes were assessed with slightly positive score to eliglustat. Quality of evidence was considered low. evidence was considered very low due to the poor Eliglustat was not included in the spansish external validity or applicability. The ERA-EDTA recommendation about Gaucher disease [11] (European Renal Association and European Dialysis management since by the time of the guideline and Transplant Association) recommended tolvap- publication eliglustat was not yet approved by the tan with some clinical criteria of use. European Medicines Agency (EMA). – Autosomal dominant polycystic kidney disease was The overall MCDA value estimates (V) of each orphan perceived as a very severe disease with an drug were 0.16, 0.23 and 0.22, for alpha-1 antitrypsin, unfavourable prognostic based on reduced survival eliglustat and tolvaptan, respectively. The utility of the of these patients, and the lack of effective overall MCDA value estimates was discussed, and the alternatives. It was observed that the efficacy of PHF members considered that MCDA contributed tolvaptan was moderately better than placebo most to decision making process when criteria were although it is accompanied with a worst safety described separately, rather than described through a profile. There was no published data related to QoL, summary score. The PHF members considered that a for this reason the score was considered neutral. structured approach to multiple criteria allowed Regarding the type of benefit tolvaptan was judged focusing discussion into most relevant domains and to provide a moderate therapeutic benefit. criteria, avoided potential linking between numeric Comparative treatment cost was assessed with a thresholds and decisions, and thus added most value negative score to tolvaptan and other medical costs from the multidimensional approach to their process with neutral score given the lack of available of decision-making. evidence related to these criteria. Quality of Discussion The outcome of this study provides valuable insight on the implementation of the multicriteria orphan drug Fig. 2 Comparison of value contribution of each criterion: alpha – 1 Fig. 4 Value of contribution of each criteria: eliglustat vs imiglucerase. antitrypsin, tolvaptan and eliglustat. PRO: Patient Reported Outcomes PRO: Patient Reported Outcomes Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 7 of 9 elements and making them transparent and supportive of the reflection during discussion across the range of dimensions in the overall assessment of a new drug, ensuring a systematic and thorough procedure. The ap- proach helps to focus the discussion, providing a common structure which is similar across products. Also allows to identify points with diverging opinions from the multidisciplinary decision-making committee. Another strength of MCDA is that it is suited to in- corporate the interpretation of the evidence from differ- ent perspectives (reasoning the scoring and weights value), and the methodology has the advantage of Fig. 5 Value of contribution of each criteria: tolvaptan vs placebo. PRO: Patient Reported Outcomes providing sufficient flexibility to incorporate relevant items tailored to singular cases and/or situations that may be incorporated rapidly to PHF process within the established framework. Although the MCDA was assessment in the CatSalut context. MCDA would be adapted to the process of the PHF, the adaption is feasible to support the assessment of reflective value of tailored to our process, and may be non-generalizable to new orphan drugs, and may be useful to structure and other purposes or contexts. However, similar processes conduct a deliberate the discussion during the PHF can be made to adapt MCDA in other settings. committee meetings. There have been attempts in Europe to implement Evaluation and decision-making for orphan drugs is a reflective MCDA approaches to support managed challenge, due to the inherent difficulties to generate a deliberation through considerations, avoiding the way high-quality evidence of value in small populations. For predominant group members can take over decision- this reason, the management of orphan drugs often making [12]. implies areas of substantial uncertainty. Such scenario Friedmann et al. [12], reviewed the existing evidence was suitable for testing whether the reflective multicri- regarding the use of MCDA in the assessment of orphan teria tool might be able to provide perspective and facili- drugs worldwide. A total of seven articles suggest that tating decision-making through a systematic, consistent MCDA is increasingly being used in the context orphan and integrated approach to several dimensions of assess- drug assessment. MCDA demonstrate to be a flexible as- ment. MCDA allows visualizing the criteria that may be sessment with the potential to assisting decision-making not explicitly informed during the decisions reached in regarding reimbursement for orphan drugs [12]. The medicines evaluation committees meetings, such as main findings of the 7 articles reviewed by Friedmann et unmet needs, severity of the disease and QoL, as well as al. report the use of the EVIDEM framework in several other contextual variables which could contribute to studies, with criteria concerning disease-specific, health reach a final decision considering the complexity of the intervention-specific and contextual factors. In our study drug, the potentially treated population and/or the “disease severity” and “clinical efficacy/effectiveness” complexity of its environment. were considered the most relevant criteria for the assess- Reflective MCDA can be used to quantify the import- ment of orphan drugs, similarly than reported by other ance of the different criteria of assessment and the institutions [12]. Additionally, our results considered the relevance assigned to specific items. Furthermore, it was criterion “unmet clinical need” as very relevant. In con- regarded that quantitative modelling had to be sup- trast, in previously published studies “Other comparative ported by a qualitative descriptive approach, similar to a costs” and “treatment innovation” were the criteria con- process of consensus. Reflective MCDA allowed to iden- sidered as less important [12], while in our exercise the tify the range of results for the dimensions and for each less relevant criteria were “quality of evidence” and “expert assessment criteria, as well as the relevance assigned to consensus/clinical practise guidelines”. specific results. Theses ranges and results were used to In Italy, the Lombardy health directorate has used a prepare qualitative discussion in a structured qualitative reflective multicriteria approach combined with the approach that is designed to support the identification European network for health technology assessment and communication of the most relevant criteria. (EUnetHTA) core model to appraise health technologies MCDA in this case would not be used as a formula to and make health funding coverage decisions since 2012. come to a decision, but rather as a tool to complement The interaction with stakeholders was facilitated by a the values and viewpoints of committee members, transparent process and demonstration of reasonable- allowing visualization and traceability of key decision ness. The process was evolving to further increase Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 8 of 9 legitimacy by involving a wider array of stakeholders approach, contextualising the relevant data for each drug such as citizens or patients [4, 13]. and facilitating evidence and ethical-based discussions The implementation of a reflective MCDA method- among all the different stakeholders involved in evalu- ology to support orphan drug decision-making follows ation and decision-making purposes. one of the nine recommendations set up by the Abbreviations European Working Group for Value Assessment and Catsalut: Catalan Health Service (Servei Català de la Salut); EMA: European Fundings Processes in Rare Diseases (ORPH-VAL), Medicines Agency; ERA-EDTA: European Renal Association and European which propose that orphan drug assessment should Dialysis and Transplant Association; EUnetHTA: European network for health technology assessment; MCDA: Multiple Criteria Decision Analysis; ORPH- consider all relevant elements of product value in an VAL: European Working Group for Value Assessment and Fundings Processes appropriate multi-dimensional framework [14]. in Rare Diseases; PHF: Pharmacotherapeutic Harmonization Program The current study has some limitations. The constitu- (Programa d’Harmonització Farmacoterapeutica); PRO: Patient Reported Outcome; QoL: Quality of life; SD: Standard deviation; SLR: Systematic tion of only a subgroup representative of committee to Literature Review enable the properly test of the tool could imply that not all the different perspectives were taken into account Acknowledgements during the discussion (health economists, bioethical and We acknowledge the contribution of committee participants and their collaboration in the project. The MCDA study has been conducted by patient representatives didn’t participate in the study). Omakase Consulting with the collaboration of CatSalut. Another limitation of the study was that the scenario adopted just focused on orphan drug assessment, which Authors’ contributions LG participated in the design and coordination of the study, analysed and for their particular challenges, led to exclusion of some adapted the additional criteria from literature, supervised the evidence criteria relevant for common diseases. The scope of matrix, support the reflective discussions and drafted the manuscript. XB CatSalut committee is to take decisions on both rare designed the study, supervised the analysis and adaptation of additional criteria from literature, lead the reflective discussions and review the and common diseases showing that the framework needs manuscript. MO and MF participated in the discussion and validation of to be expanded from orphan drugs to non-orphan drugs additional criteria, developed the three evidence matrix, scored and medicine products. weighted the framework and helped to draft the manuscript. AP and AV participated in the discussion and validation of additional criteria, scored and Additionally, another limitation was the description of weighted the framework and helped to draft the manuscript. JT participated the concept and definitions of each criterion which in the design of the study, participated in the discussion and validation of could have been more explicit to reduce the variability additional criteria, scored and weighted the framework and reviewed the manuscript. CP participated in the design and coordination of the study, of the interpretation by the decision-makers. Such participated in the discussion and validation of additional criteria, scored and variability might have conditioned different results for weighted the framework and reviewed the manuscript. All authors read and the scoring and weighting phases. approved the final manuscript. Finally, it is important to highlight that the synthesis Funding of the evidence to develop the evidence matrix could CatSalut funded the development of this study. minimise the advantages of the tool, which are focused on the identification of the variety of points of views and Availability of data and materials promotion of a multidimensional reflective discussion. Not applicable. Ethics approval and consent to participate Conclusion Not applicable. This study shows that MCDA methodology could be- come a systematic procedure to complement the overall Consent for publication Not applicable. process of assessment of the added clinical value of innovative medicines by CatSalut. Competing interests There is a growing understanding that the assessment The authors declare that they have no competing interests. based only on traditional criteria such as clinical efficacy, Author details safety and cost, does not allow the value of a new drug to 1 Àrea del Medicament, Servei Català de la Salut (CatSalut), Travessera de les be fully captured. Considering the goal to promote the Corts, 131-159, 08028 Barcelona, Spain. 2Omakase Consulting S.L., Entença street 332334 Floor 6, door 4, 08029 Barcelona, Spain. 3Departament de quality of drug prescription, MCDA allows a deep explor- Farmacologia, de Terapèutica i de Toxicologia, Universitat Autònoma de ation of a set of quantitative and qualitative inputs, includ- Barcelona, Av. de Can Domènech, 737, 08193 Cerdanyola del Vallès ing reflective discussions and individual comments (Barcelona), Spain. facilitating the evaluation of innovative drug, positioning Received: 29 December 2018 Accepted: 7 June 2019 medicines within therapeutic algorithms and allowing equitable resource allocation in the healthcare sector. By integration of MCDA to the PHF procedures, References 1. Ministerio de Sanidad Servicios Sociales e Igualdad . [cited 2017 May 1] MCDA could support the technical evaluation report Available from: National Health System. Spain. https://www.msssi.gob.es/ providing a structured, standardised and transparent organizacion/ministerio/home.htm. Guarga et al. Orphanet Journal of Rare Diseases (2019) 14:157 Page 9 of 9 2. Generalitat de Catalunya. Pla de Salut de Catalunya 2016-2020. [cited 2017 May 1] Available from: http://salutweb.gencat.cat/ca/el_departament/Pla_ salut/pla-de-salut-2016-2020/. 3. Agencia de Evaluación de Tecnologías Sanitarias (AETS). Instituto de Salud Carlos III - Ministerio de Economía y Competitividad. Criterios de financiación y reembolso de los medicamentos huérfanos. 2016. 4. Goetghebeur M, Castro-Jaramillo H, Baltussen R, et al. The art of medicine the art of priority setting. Lancet. 2017;389:17. 5. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Mak. 2012;32(2):376–88. 6. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria analysis. Cost Eff Resour Alloc. 2006;4:14. 7. Gilabert-Perramon A, Torrent-Farnell J, Catalan A, Prat A, Fontanet M, Puig- Peiró R, et al. Drug evaluation and decision making in catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (MCDA) for orphan drugs. Int J Technol Assess Health Care. 2017;33(1):111–20. 8. EVIDEM Framework (v 4.0) [Internet]. [cited 2017 May 1]. Available from: https://www.evidem.org. 9. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Heal J Int Soc Pharmacoeconomics Outcomes Res. 2016;19(1):1–13. 10. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19(2):125–37. 11. Fundación Española para el estudio y terapéutica de la enfermedad de Gaucher y otras enfermedades lisosomales (FEETEG) [Internet]. Available from: http://www.feeteg.org/. 12. Friedmann C, Levy P, Hensel P, Hiligsmann M. Using multi-criteria decision analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoecon Outcomes Res. 2017;18(2):135–46. Available from: https:// www.tandfonline.com/doi/full/10.1080/14737167.2018.1414603. 13. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUNetHTA core model in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30(1):105–12. 14. Annemans L, Aymé S, Le Cam Y, Facey K, Gunther P, Nicod E, Reni M, Roux J-L, Schlander M, Taylor D, Tomino C, Torrent-Farnell J, Upadhyaya S, Hutchings A, Le Dez L. Recommendations from the European working Group for Value Assessment and Funding Processes in rare diseases (ORPH-VAL). Orphanet J Rare Dis. 2017;12:50. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.