PLOS ONE RESEARCH ARTICLE Clinical benefit of cancer drugs approved in Switzerland 2010–2019 Roman Adam ID1, Ariadna Tibau2, Consolación Molto Valiente2, Boštjan Šeruga3, Alberto Ocaña4, Eitan Amir ID5, Arnoud J. Templeton ID1,6,7* 1 Faculty of Medicine, University of Basel, Basel, Switzerland, 2 Oncology Department, Departament de Medicina de la Universitat Autònoma de Barcelona, Hospital de la Santa Creu i Sant Pau, Institut d’Investigació Biomèdica Sant Pau, Barcelona, Spain, 3 Institute of Oncology Ljubljana and Faculty of Medicine, Department of Medical Oncology, University of Ljubljana, Ljubljana, Slovenia, 4 Experimental a1111111111 Therapeutics Unit, Medical Oncology Department, Hospital Clı́nico Universitario San Carlos and IdISSC, Madrid, Spain, 5 Division of Medical Oncology and Hematology, Department of Medicine, Princess Margaret a1111111111 Cancer Center and the University of Toronto, Toronto, ON, Canada, 6 Department of Medical Oncology, a1111111111 St. Claraspital, Basel, Switzerland, 7 St. Clara Research Ltd., Basel, Switzerland a1111111111 a1111111111 * arnoud.templeton@claraspital.ch Abstract OPEN ACCESS Background Citation: Adam R, Tibau A, Molto Valiente C, It is unknown to what extent cancer drugs approved in Switzerland by the Swissmedic fulfil Šeruga B, Ocaña A, Amir E, et al. (2022) Clinical benefit of cancer drugs approved in Switzerland criteria of clinical benefit according to the European Society of Medical Oncology Magnitude 2010–2019. PLoS ONE 17(6): e0268545. https:// of Clinical Benefit Scale version 1.1 (ESMO-MCBS), the American Society of Clinical Oncol- doi.org/10.1371/journal.pone.0268545 ogy Value Framework version 2 (ASCO-VF) and the Swiss OLUtool v2 (OLUtool). Editor: Wen-Wei Sung, Chung Shan Medical University, TAIWAN Patients and methods Received: August 31, 2021 An electronic search identified studies that led to marketing authorisations in Switzerland Accepted: May 3, 2022 2010–2019. Studies were evaluated according to ESMO-MCBS, ASCO-VF and OLUtool. Published: June 10, 2022 Substantial benefit for ESMO-MCBS, was defined as a grade A or B for (neo)adjuvant intent and 4 or 5 for palliative intent. For ASCO-VF and OLUtool clinical benefit was defined as Peer Review History: PLOS recognizes the benefits of transparency in the peer review score �45 and A or B, respectively. Concordance between the frameworks was calculated process; therefore, we enable the publication of with Cohen’s Kappa (κ). Factors associated with clinical benefit were evaluated by logistic all of the content of peer review and author regression. responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0268545 Results Copyright: © 2022 Adam et al. This is an open In the study period, 48 drugs were approved for 92 evaluable indications, based on 100 access article distributed under the terms of the studies. Ratings for ESMO-MCBS, ASCO-VF and OLUtool could be performed for 100, 86, Creative Commons Attribution License, which and 97 studies, respectively. Overall, 39 (39%), 44 (51%), 45 (46%) of the studies showed permits unrestricted use, distribution, and substantial clinical benefit according to ESMO-MCBS v1.1, ASCO-VF, OLUtool criteria, reproduction in any medium, provided the original author and source are credited. respectively. There was fair concordance between ESMO-MCBS and ASCO-VF in the palli- ative setting (κ = 0.31, P = 0.004) and moderate concordance between ESMO-MCBS and Data Availability Statement: All relevant data are within the paper and its Supporting Information OLUtool (κ = 0.41, P<0.001). There was no significant concordance between ASCO-VF and files. OLUtool (κ = 0.18, P = 0.12). Factors associated with substantial clinical benefit in multivari- Funding: The authors received no specific funding able analysis were HRQoL benefit reported as secondary outcome for ESMO-MCBS and for this work. the ASCO-VF and blinded studies for OLUtool. PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 1 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Competing interests: I have read the journal’s Conclusions policy and the authors of this manuscript have declared the following competing interests (and At the time of approval, only around half of the trials supporting marketing authorisation of leave it to the judgement of the editor whether this recently approved cancer drugs in Switzerland meet the criteria for substantial clinical bene- is of relevant for the submitted work): Eitan Amir: fit when evaluated with ESMO-MCBS, ASCO-VF or OLUtool. There was at best only moder- reports personal fees for expert testimony from Genentech/Roche and an advisory role for Sandoz, ate concordance between the grading systems. Novartis and Exact Sciences. Ariadna Tibau: reports personal fee for travel grant from Pfizer and honoraria from Eisai, Roche and Novartis outside Background the submitted work. Arnoud J. Templeton: advisory board/consultancy: Astellas, MSD, BMS Over time, the number of cancer drugs approved by the Food and Drug Administration (institution), Janssen (institution), Sanofi (FDA) has increased [1]. This has been attributed to advances in cancer drug research but also (institution), Roche (institution); honoraria: to faster approvals which are based often on preliminary data, and intermediate primary end- Astellas, Sanofi; conference/travel support: Bayer, points instead of overall survival (OS) [2,3]. While it is desirable to make treatments available Sanofi, Janssen, Ipsen, Roche. This does not alter on the market in a shorter time, this should not happen at the expense of effectiveness or safety our adherence to PLOS ONE policies on sharing data and materials. All remaining authors have of the approved substances and must therefore be approached with caution. There are con- declared no conflict of interest. cerns that faster drug approval may be associated with lower drug efficacy and worse patient safety [4,5]. Possibly also partly due to the high amount of approvals based on intermediate endpoints [6,7] whose correlation to long-term outcomes like OS and Health related quality of life (HRQoL) was low in studies [8–10]. Effective treatments should prolong survival time and/or improve HRQoL. Previous studies have argued that certain cancer drugs approved by the FDA [11] and the European Medicines Agency (EMA) [12] show questionable clinical benefit. In Switzerland, approval for new drugs is granted by the Swiss national authorisation and supervisory authority Swissmedic. In the authorisation process of medical products Swissmedic takes into account pharmacological and clinical data on efficacy and safety as well as HRQoL. The price of a medicinal product is not assessed by Swissmedic and is determined after authorisation either by the authorisation holder or, in the case of an application for health insurance coverage, by the Swiss Federal Office of Public Health. There are different validated tools to assess the clinical benefit of a cancer drugs: Commonly used tools are the European Society for Medical Oncology–Magnitude of Clinical Benefit Scale version 1.1 (ESMO-MCBS v1.1) [13], the American Society of Clinical Oncology—Value Framework version 2 (ASCO VF v2) [14], and in Switzerland the OLUtool Onko version 2.0 (OLUtool v2) [15]. These tools were developed with different goals. The ESMO-MCBS was created to quantify the effectiveness of new cancer treatments [16], the ASCO-VF was devel- oped to facilitate physicians and patients to assess the expected clinical benefit of a cancer treatment to help them in their decision making process [17], and the OLUtool is used for the case-by-case decision on the reimbursement of treatment costs in the off-label use of cancer drugs [15]. Creating a validated tool to quantify the clinical benefit of cancer treatments is complex and takes into account different aspects on the efficacy, toxicity and HRQoL. The different selec- tion and weighting of these data can lead to differences in the results of the application of the frameworks. Previous studies that assessed the concordance between the ESMO-MCBS and the ASCO-VF have shown varying results ranging from fair [18] to substantial [19] concordance. The aim of our study was to investigate to what extent the cancer drugs approved in Swit- zerland between January 1 2010 and December 31 2019 fulfil the criteria for a substantial clini- cal benefit when evaluated with the ESMO-MCBS v1.1, the ASCO-VF v2 and the OLUtool v2. Furthermore, we investigated whether there was concordance between the grading tools and evaluated factors associated with clinical benefit. PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 2 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Methods Data sources, study selection and data extraction Drugs used for treatment of solid tumours which received first marketing authorisation between January 2010 and December 2019 were identified from the official Swissmedic Jour- nals and the website swissmedicinfo.ch [20]. Subsequently, we searched for pivotal studies sup- porting the approved drug. Next, studies which reported an endpoint evaluable by ESMO-MCBS v1.1, ASCO-VF v2, or OLUtool Onko v2, namely overall survival (OS), progres- sion-free survival (PFS), disease-free survival (DFS), relapse-free survival (RFS), and objective response rate (ORR) were identified. Updated data of pivotal studies or reports on toxicity or QoL outcomes were considered in the analysis only if they were published before the date of Swiss marketing approval of a respective drug for a given indication. Drug names, indications and approval dates were recorded from the official website and the following data were subsequently extracted from publications: name of first author, publi- cation date, journal, publication type (abstract / full paper), primary cancer site, number of patients evaluated, study design (randomized vs. single-arm), blinding (double-blind vs. open- label), phase of study (phase I, II, III), drug class, treatment setting (curative vs. palliative), pri- mary and secondary study endpoints (OS, PFS, DFS, RFS, ORR), data on toxicity and on HRQoL, cross-over (yes vs. no), subgroup analysis (yes vs. no), survival curve plateau (yes vs. no) and the need for companion diagnostics. If the indication corresponded to a specific subgroup evaluated in a study, the data for the subgroup were evaluated if they were reported separately in the study. If the indication was based on different studies with different selection criteria (e.g. mutation status, prior therapies) or studies of different treatment lines, all these studies were considered. If the indication was expanded to include other subgroups (e.g. addition of a further mutation class or new line or treatment, e.g. additional approval as first-line) the study data supporting the expansion were evaluated. In cases where different studies with identical selection criteria supported approval, the study with the biggest sample size was selected. If different subgroups were reported in a trial, we selected the subgroup which best matched the indication. Data synthesis, scoring All identified studies were evaluated using the ESMO-MCBS v1.1, the ASCO-VF v2 and OLU- tool v2 according to published guidelines [13,14,21]. For the grading of the studies, only statis- tically significant data with a p-value < 0.05 were considered. Additionally, HRQoL outcomes had to be based on a validated questionnaire and show a clinically relevant improvement. Sub- stantial clinical benefit was defined as 5 or 4 and A or B in the ESMO-MCBS in the palliative and curative setting, respectively. For ASCO-VF v2 a conservative threshold for substantial benefit of 45 points was defined as previously reported by Cherny et al. [19]. We performed a sensitivity analysis for agreement in which we only compared studies that were evaluated for the same efficacy endpoints. To evaluate if the previously reported threshold of 45 points for a substantial clinical benefit for the ASCO-VF v2 also fits our data we generated a receiver-operating characteristic (ROC) curve to find the optimal threshold for substantial clinical benefit for our data and calculated the concordances in this analysis. Statistical analysis Study data are reported as median, ranges and point estimates of time to event endpoints, where appropriate. The agreement between the different frameworks was calculated using PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 3 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Cohen’s Kappa (for dichotomous outcomes) and Spearman’s ranks correlation (for continu- ous outcomes). Interpretation of correlations and concordance was performed according to previous studies [22,23]. Trends over time were calculated by linear regression and their p-val- ues were reported. We used box plots to show the ratings of the ASCO-VF v2 scores in relation to the ESMO-MCBS v.1.1 scores and the OLUtool v2 grades visually. Independent predictors (shown in S4 Table) were calculated using univariable and multivariable logistic regression and stated as Odds ratio (OR) with corresponding 95% confidence intervals (CI) and p-values. Only variables with a p-value < 0.1 in the univariable analysis were included in multivariable analyses. SPSS version 25 (IBM Corp, Armonk NY) was used for all analyses. All statistical tests were two sided and significance level was defined as p-value < 0.05. No correction for multiple statistical testing was applied. Results Drugs approved During the study period from 2010 to 2019, Swissmedic approved 48 new cancer drugs for 101 indications in solid tumours. Overall, 100 studies supporting the approvals of 92 indications of 45 new cancer drugs were found (Fig 1). Study characteristics are presented in Table 1. Most studies were in the palliative setting and most of them were randomized phase 3 trials. Although the HRQoL was evaluated in around half of the studies it was only reported and thus evaluable as a secondary outcome in around one third of the studies. Substantial clinical benefit Evaluations with ESMO-MCBS v.1.1, ASCO-VF v2 and OLUtool v2 could be performed for 92, 78, and 90 indications supported by 100, 86, and 97 studies, respectively. At the time of approval, 39 (39%), 44 (51%), and 45 (46%) of all evaluated studies met the criteria for substan- tial clinical benefit according to ESMO-MCBS v1.1, ASCO-VF v2 and OLUtool v2, respec- tively (Fig 2 and S1 Table). Fig 1. Identification of new active substances, indications for solid tumours and their supporting pivotal trials published between January 2010 and December 2019 which showed an endpoint evaluable with the ESMO-MCBS v1.1, the ASCO-VF v2 and the OLUtool v2. https://doi.org/10.1371/journal.pone.0268545.g001 PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 4 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Table 1. Characteristics of pivotal trials evaluable with the ESMO-MCBS v1.1, the ASCO-VF v2 and the OLUtool v2. Framework ESMO-MCBS v1.1 ASCO-VF v2 OLUtool v2 Characteristics N (%) N (%) N (%) Studies available 100 (100) 86 (100) 97 (100) Randomized Patients (patients included in the study) Median 548 599 553 Range 50–4805 133–4805 50–4805 Analysed Patients Median 444 556 493 Range 50–4804 79–4804 50–4804 Tumour type Lung cancer 23 (23) 17 (20) 23 (24) Melanoma 17 (17) 16 (19) 15 (16) Breast cancer 17 (17) 15 (17) 16 (17) Gastrointestinal (including hepatocellular) 10 (10) 8 (9) 10 (10) Prostate cancer 7 (7) 7 (8) 7 (7) Ovarian cancer 6 (6) 6 (7) 6 (6) Sarcoma (including GIST) 5 (5) 5 (6) 5 (5) Renal cancer 5 (5) 5 (6) 5 (5) Urothelial 3 (3) 2 (2) 3 (3) Other 7 (7) 5 (6) 7 (7) Setting Curative (neoadjuvant/adjuvant) 7 (7) 6 (7) 6 (6) Palliative 93 (93) 80 (93) 91 (94) Line of treatment Neoadjuvant/Adjuvant 7 (7) 6 (7) 6 (6) First line 43 (43) 38 (44) 42 (43) Second line 43 (43) 35 (41) 42 (43) Third line 7 (7) 7 (8) 7 (7) Study design Randomized 87 (87) 86 (100) 84 (87) Single-arm 13 (13) 0 13 (13) Phase of study Phase 1 2 (2) 0 2 (2) Phase 2 17 (17) 5 (6) 16 (17) Phase 3 81 (81) 81 (94) 79 (81) Blinding Open-label 51 (51) 37 (43) 48 (49) Double-blind 49 (49) 49 (57) 49 (51) Crossover Allowed 19 (25) 19 (26) 19 (26) Not allowed 56 (75) 55 (74) 54 (74) Time-to-event as primary endpoint Yes 84 (84) 84 (98) 83 (86) No 16 (16) 2 (2) 14 (14) Number of primary endpoints 1 80 (80) 67 (78) 78 (80) >1 20 (20) 19 (22) 19 (20) Primary endpoint (Continued ) PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 5 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Table 1. (Continued) Framework ESMO-MCBS v1.1 ASCO-VF v2 OLUtool v2 Overall survival 41 (41) 41 (48) 40 (41) Progression or disease free survival 54 (54) 54 (63) 54 (56) Objective response rate 16 (16) 3 (4) 15 (16) Health related quality of life reported Yes 52 (52) 48 (56) 52 (54) No 48 (48) 38 (44) 45 (46) Health related quality of life reported as secondary outcome Yes 32 (32) 32 (37) 32 (33) No 68 (68) 54 (63) 65 (67) Improvement in HRQoL reported as secondary outcome Yes 15 (15) 15 (17) 15 (15) No 85 (85) 71 (83) 82 (85) Companion diagnostics Yes 41 (41) 30 (35) 40 (41) No 59 (59) 56 (65) 57 (59) Experimental drug Small molecule (TKI, PARPi, etc.) 37 (37) 29 (34) 37 (38) Immune checkpoint inhibitor monotherapy 22 (22) 19 (22) 21 (22) CDK4/6 inhibitor plus endocrine therapy 7 (7) 7 (8) 7 (7) Immune checkpoint inhibitor combination 7 (7) 6 (7) 7 (7) Endocrine therapy 6 (6) 6 (7) 6 (6) Non-checkpoint inhibitor antibody combination 6 (6) 5 (6) 5 (5) Small molecule combination 5 (5) 4 (5) 5 (5) Chemotherapy monotherapy 4 (4) 4 (5) 4 (4) Other 6 (6) 6(7) 5 (5) Abbreviations: N: number; GIST: Gastrointestinal stromal tumour; HRQoL: Health related Quality of life; TKI: Tyrosine kinase inhibitor; PARPi: Poly (ADP-ribose) polymerase inhibitor; TKI: Tyrosine kinase inhibitor; PARPi: Poly (ADP-ribose) polymerase inhibitor; CDK4/6: Cycline dependent kinase 4/6. https://doi.org/10.1371/journal.pone.0268545.t001 Concordance and correlation of grading systems The number of concordant studies, concordances and correlations between the grades of the studies evaluated with the different frameworks are presented in Table 2. Overall, there was Fig 2. Number of studies which fulfilled the criteria for substantial clinical benefit for the respective frameworks. A. All studies; B: studies in the palliative setting. https://doi.org/10.1371/journal.pone.0268545.g002 PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 6 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Table 2. Concordance and correlation between the different frameworks. ESMO-MCBS v1.1/ASCO-VF v2 ESMO-MCBS v1.1/OLUtool v2 ASCO-VF v2/OLUtool v2 all studies palliative curative all studies (N palliative curative all studies (N palliative curative (N = 86) setting (N = 80) setting = 97) setting (N = setting (N = = 84) setting (N = 78) setting (N = (N = 6) 91) 6) 6) Number of 53 (62%) 52 (65%) 1 (17%) 69 (71%) 65 (71%) 4 (67%) 49 (58%) 46 (59%) 3 (50%) concordant studies Spearman’s rho 0.42 (P<0.001) 0.58 (P<0.001) 0.40 (P<0.001) Cohen’s Kappa 0.23 0.31 (P = 0.004) 0.41 0.41 (P<0.001) 0.16 0.18 (P = 0.121) (P = 0.029) (P<0.001) (P = 0.133) Abbreviations: ESMO-MCBS v1.1: European Society for Medical Oncology—Magnitude of Clinical Benefit Scale version 1.1; ASCO-VF v2: American Society of Clinical Oncology—Value Framework version 2; OLUtool v2: OLUtool version 2; P: p-value. https://doi.org/10.1371/journal.pone.0268545.t002 fair to moderate concordance between the different frameworks and was highest for the com- parison of ESMO-MCBS v1.1 with the OLUtool v2 (kappa 0.41, p < 0.001). Higher scores in the ASCO-VF v2 were associated with higher grades from ESMO-MCBS v1.1 and OLUtool v2 (Fig 3) and showed fair correlation between the tools (Table 2). ROC curve analyses suggested that 47 points and 56 points has the highest discriminatory ability in the studies analysed for the comparison with the ESMO-MCBS v1.1 and the OLUtool v2, respectively (S2 Table). Con- sidering studies in the palliative setting only (N = 93) the optimal cut-offs were 49 and 56, respectively. In sensitivity analyses concordance was similar when comparing the overall study selection and the palliative setting and when considering studies with the same endpoints only (S3 Table). Trends over time The proportion of studies supporting drug approval in Switzerland with results indicating sub- stantial clinical benefit has largely remained unchanged over the study period (Fig 4). Also, when only considering studies in the palliative setting there was no statistically significant trend over time for the rate of studies with substantial clinical benefit evaluated with the ESMO-MCBS (P trend = 0.13) or the ASCO-VF (P trend = 0.72). Of note, for OLUtool v2 the number of trials meeting substantial clinical benefit significantly increased over time (2010– 2012: 31%, vs. 2017–2019: 53%; P trend = 0.032) suggesting that more recently approved indica- tions more frequently fulfil the criteria for a substantial clinical benefit. Predictors for substantial clinical benefit In univariable analyses the benefit of HRQoL as secondary outcome was associated with a higher likelihood of substantial clinical benefit for ESMO-MCBS v1.1 and ASCO-VF v2 (S4 Table). In the ESMO-MCBS v1.1 and OLUtool v2 phase 3 studies had a higher likelihood for a substantial clinical benefit while this was not the case with ASCO VF v2. Studies for treatments with a curative intent and studies in which crossover was allowed were also predictive of a sub- stantial clinical benefit when evaluated with the ESMO-MCBS v1.1 and for the OLUtool blinded studies showed higher odds for a substantial clinical benefit. The association between HRQoL benefit for the ESMO-MCBS v1.1 and ASCO-VF v2 and blinded studies for the OLU- tool v2 mentioned above were maintained in the multivariable analysis (Table 3). Other associ- ations were no longer significant. PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 7 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 8 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Fig 3. Scores evaluated with the ASCO-VF-v2 according to their ESMO-MCBS v1.1 scores (A/C) or OLUtool v2 grades (B/ C). https://doi.org/10.1371/journal.pone.0268545.g003 Discussion We evaluated the pivotal trials supporting approval of cancer drugs in Switzerland during the last decade using the ESMO-MCBS v1.1, ASCO-VF v2 and the OLUtool v2 to evaluate clinical benefit according to these well-characterized frameworks. We found that only around half of the included trials showed a substantial clinical benefit at the time of approval. These findings are in line with analyses of studies supporting cancer drug approvals by the FDA, the EMA and the Canadian Agency for Drugs and Technologies in Health [11,12,24]. This result also suggests that the Swiss regulatory body Swissmedic evaluates the effectiveness of the treat- ments differently than the frameworks examined in our study, likely emphasizing formal sta- tistical positivity of studies irrespective of the clinical impact of the primary endpoint. We also evaluated the concordance of the different frameworks and found discrepancies in the grades which likely originate in the different methodical approaches and selection of the analysed data as discussed by Cherny and colleagues [19]. One of the major differences between the frameworks is the use of the point estimate of the hazard ratio rather than the lower-limit of the 95% confidence interval in the ASCO-VF v2 and the OLUtool v2, as com- pared to the ESMO-MCBS v1.1, respectively. Second, the frameworks also differ in the way how the toxicity of the treatments is considered. While the ESMO-MCBS v1.1 and the OLU- tool v2 consider the absolute proportion of patients with high-grade adverse events in their rat- ings, the ASCO-VF v2 bases its toxicity adjustment on the relative occurrence of adverse events of all grades. In addition, in the OLUtool v2 and the ESMO-MCBS v1.1, only one bonus point is possible for adjustment, whereas for the ASCO-VF v2 bonus points are often awarded for reduction of toxicity and for improvement of HRQoL, factors that often go hand in hand in a clinical trial. The frameworks also differ in how they credit the tail of the curve and it seems more difficult to obtain a bonus for a tail of the curve with ESMO-MCBS v1.1 and OLU- tool v2 than with the ASCO-VF v2. In general, this means that outcomes that are used for adjustments (such as toxicity and HRQoL) might have a cumulative effect with ASCO-VF v2 in the non-curative setting whereas in ESMO-MCBS v1.1 and the OLUtool v2 this is not the case. Differences between the ratings are likely to also have resulted from studies utilizing differ- ent efficacy endpoints. Additionally, the threshold of 45 or greater points for ASCO-VF v2 may not have been optimal for our study selection. This is shown by the sensitivity analyses, where studies for which the same endpoint was evaluated in both compared frameworks and for which the optimal threshold was calculated with the ROC curve showed higher concor- dance. Indeed, when using the established thresholds of grade A or B with OLUtool v2 the optimal cut for substantial clinical benefit when using the ASCO-VF v2 was 56 points. How- ever, this cut-off resulted from the comparison with OLUtool v2 which was originally created to help with the decision of cost coverage in the off-label setting and thus uses more rigorous criteria than can be expected for registration trials, likely with a higher threshold. Apart from the grades for the OLUtool v2, there was no significant changes in scores over time. However, there is an encouraging trend that more recent registration trials evaluate HRQoL more often as a secondary outcome than this was done in older registration trials. Some of these data on HRQoL were not yet published at the time of approval and were there- fore not included in our analysis. However, considering that HRQoL was an independent pre- dictor for substantial clinical benefit in ESMO-MCBS v1.1 and ASCO-VF v2, it is likely that the ratings of the pivotal trials would be higher when including these HRQoL data. PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 9 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 10 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland Fig 4. Proportion of of studies meeting the criteria for a substantial clinical benefit from 2010 to 2019. A: ESMO-MCBS 1.1; B: ASCO-VF v2; C: OLUtool v2. https://doi.org/10.1371/journal.pone.0268545.g004 Table 3. Predictors for substantial clinical benefit on multivariable logistic regression for all studies. ESMO-MCBS v1.1 (all studies) ASCO-VF v2 (all studies) OLUtool v2 (all studies) Multivariable analysis OR (95% CI) P OR (95% CI) P OR (95% CI) P Phase 3 (vs. phase 1, 2) 4.65 (2.03–124.86) 0.235 3.47 (0.86–13.98) 0.080 Cross-over allowed (versus not) 3.23 (1.05–9.09) 0.056 Line of treatment Neoadjuvant/adjuvant 8.15 (1.26–94.52) 0.142 First line (vs. further line) 2.06 (0.83–5.12) 0.121 Blinded study (vs. open label) 0.69 (0.27–1.80) 0.451 3.30 (1.30–8.40) 0.012 HRQoL benefit as secondary outcome 6.33 (2.24–32.98) 0.012 4.35 (1.04–18.15) 0.044 (yes vs. no) Approved since 2017 (vs. 2010–2016) 0.47 (0.19–1.16) 0.100 Abbreviations: ESMO-MCBS v1.1: European Society for Medical Oncology—Magnitude of Clinical Benefit Scale Version 1.1; ASCO-VF v2: American Society of Clinical Oncology—Value Framework Version 2; OLUtool v2: OLUtool Version 2; OR: odds ratio; 95% CI: 95% confidence interval; P: p-value; vs.: versus; incl.: inclusive; HRQoL: Health related quality of life. Only variables with a p-value < 0.1 in the univariable analysis (for details see S4 Table) were included in multivariable analyses. https://doi.org/10.1371/journal.pone.0268545.t003 Our study has several limitations. First, we limited our analysis to solid tumours and excluded drugs approved to treat hematologic malignancies. Second, Swissmedic does not pro- vide the exact data on which the marketing approval is granted and since they constantly update the study data on their webpage, the data do not always correspond to the data available at the time of authorisation. In order to address this problem, we decided to search for the original data of the pivotal trials and their updates and evaluate only data that had already been pub- lished before the date of the Swiss marketing authorisation. However, it is possible that Swiss- medic had access to updated data which were not published at the time of approval and thus could not be included in our analysis. Third, our data were retrieved from published articles only. Therefore, data on HRQoL were missing frequently and data on toxicity was often based on pooled analyses. Due to the different methods of considering data on toxicity, adjustments made for toxicity in the different frameworks can differ depending on the way data is presented by the authors. Fourth, although OLUtool v2 was developed to evaluate off-label treatments, in this study we used it to rate on-label treatments where higher level evidence might be expected. In summary, at the time of approval, around half of the pivotal studies supporting cancer drug approvals during the last decade in Switzerland meet criteria for substantial clinical bene- fit as rated with various frameworks (ASCO-VF v2, ESMO-MCBS v1.1, OLUtool v2). These frameworks only have fair to moderate concordance suggesting different appreciation of end- points and magnitude of effect reported in studies. Further research is needed to establish opti- mal rating systems to determine and compare clinical benefits of new cancer drugs in clinical studies and in daily practice. Supporting information S1 Table. Grades evaluated with the ESMO-MCBS v1.1, the ASCO-VF v2 and the OLUtool v2. (DOCX) PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 11 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland S2 Table. Sensitivity analysis for the correlation and concordance between the studies with the optimal threshold calculated with ROC curve analyses for ASCO-VF v2. (DOCX) S3 Table. Sensitivity analysis for the concordance and the correlation between the studies considering studies with the same evaluated endpoints only. (DOCX) S4 Table. Predictors of substantial clinical benefit on univariable logistic regression for all studies. (DOCX) S1 File. (XLSX) Author Contributions Conceptualization: Ariadna Tibau, Arnoud J. Templeton. Data curation: Roman Adam, Consolación Molto Valiente, Arnoud J. Templeton. Formal analysis: Roman Adam, Ariadna Tibau, Consolación Molto Valiente, Boštjan Šeruga, Alberto Ocaña, Eitan Amir, Arnoud J. Templeton. Investigation: Roman Adam, Eitan Amir, Arnoud J. Templeton. Methodology: Roman Adam, Ariadna Tibau, Eitan Amir, Arnoud J. Templeton. Project administration: Arnoud J. Templeton. Supervision: Arnoud J. Templeton. Validation: Ariadna Tibau, Consolación Molto Valiente, Boštjan Šeruga, Alberto Ocaña, Eitan Amir, Arnoud J. Templeton. Visualization: Roman Adam, Arnoud J. Templeton. Writing – original draft: Roman Adam, Arnoud J. Templeton. Writing – review & editing: Ariadna Tibau, Consolación Molto Valiente, Boštjan Šeruga, Alberto Ocaña, Eitan Amir, Arnoud J. Templeton. References 1. Gyawali B, Sharma S, Booth CM. Is the number of cancer drug approvals a surrogate for regulatory suc- cess? Journal of Cancer Policy. 2019; 22: 100202. https://doi.org/10.1016/j.jcpo.2019.100202 2. Chen EY, Joshi SK, Tran A, Prasad V. Estimation of Study Time Reduction Using Surrogate End Points Rather Than Overall Survival in Oncology Clinical Trials. JAMA Intern Med. 2019; 179: 642. https://doi. org/10.1001/jamainternmed.2018.8351 PMID: 30933235 3. Ribeiro TB, Buss L, Wayant C, Nobre MRC. Comparison of FDA accelerated vs regular pathway approvals for lung cancer treatments between 2006 and 2018. PLOS ONE. 2020; 15: e0236345. https://doi.org/10.1371/journal.pone.0236345 PMID: 32706800 4. Mostaghim SR, Gagne JJ, Kesselheim AS. Safety related label changes for new drugs after approval in the US through expedited regulatory pathways: retrospective cohort study. BMJ. 2017; 358: j3837. https://doi.org/10.1136/bmj.j3837 PMID: 28882831 5. Frank C, Himmelstein DU, Woolhandler S, Bor DH, Wolfe SM, Heymann O, et al. Era Of Faster FDA Drug Approval Has Also Seen Increased Black-Box Warnings And Market Withdrawals. Health Affairs. 2014; 33: 1453–1459. https://doi.org/10.1377/hlthaff.2014.0122 PMID: 25092848 PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 12 / 13 PLOS ONE Clinical benefit of oncologic drugs approved in Switzerland 6. Kim C, Prasad V. Cancer Drugs Approved on the Basis of a Surrogate End Point and Subsequent Over- all Survival: An Analysis of 5 Years of US Food and Drug Administration Approvals. JAMA Intern Med. 2015; 175: 1992. https://doi.org/10.1001/jamainternmed.2015.5868 PMID: 26502403 7. Downing NS, Aminawung JA, Shah ND, Krumholz HM, Ross JS. Clinical Trial Evidence Supporting FDA Approval of Novel Therapeutic Agents, 2005–2012. JAMA. 2014; 311: 368. https://doi.org/10. 1001/jama.2013.282034 PMID: 24449315 8. Haslam A, Hey SP, Gill J, Prasad V. A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology. European Jour- nal of Cancer. 2019; 106: 196–211. https://doi.org/10.1016/j.ejca.2018.11.012 PMID: 30528804 9. Booth CM, Eisenhauer EA. Progression-Free Survival: Meaningful or Simply Measurable? JCO. 2012; 30: 1030–1033. https://doi.org/10.1200/JCO.2011.38.7571 PMID: 22370321 10. Kovic B, Jin X, Kennedy SA, Hylands M, Pedziwiatr M, Kuriyama A, et al. Evaluating Progression-Free Survival as a Surrogate Outcome for Health-Related Quality of Life in Oncology: A Systematic Review and Quantitative Analysis. JAMA Intern Med. 2018; 178: 1586. https://doi.org/10.1001/jamainternmed. 2018.4710 PMID: 30285081 11. Tibau A, Molto C, Ocana A, Templeton AJ, Del Carpio LP, Del Paggio JC, et al. Magnitude of Clinical Benefit of Cancer Drugs Approved by the US Food and Drug Administration. JNCI: Journal of the National Cancer Institute. 2018; 110: 486–492. https://doi.org/10.1093/jnci/djx232 PMID: 29244173 12. Grössmann N, Del Paggio JC, Wolf S, Sullivan R, Booth CM, Rosian K, et al. Five years of EMA- approved systemic cancer therapies for solid tumours—a comparison of two thresholds for meaningful clinical benefit. European Journal of Cancer. 2017; 82: 66–71. https://doi.org/10.1016/j.ejca.2017.05. 029 PMID: 28648700 13. Cherny NI, Dafni U, Bogaerts J, Latino NJ, Pentheroudakis G, Douillard J-Y, et al. ESMO-Magnitude of Clinical Benefit Scale version 1.1. Annals of Oncology. 2017; 28: 2340–2366. https://doi.org/10.1093/ annonc/mdx310 PMID: 28945867 14. Schnipper LE, Davidson NE, Wollins DS, Blayney DW, Dicker AP, Ganz PA, et al. Updating the Ameri- can Society of Clinical Oncology Value Framework: Revisions and Reflections in Response to Com- ments Received. JCO. 2016; 34: 2925–2934. https://doi.org/10.1200/JCO.2016.68.2518 PMID: 27247218 15. Fries R, Hummel Y, Kipfer B, Schafroth U, Seiler B, Zollikofer J. OLUtool als neues Einheitsmodell der Vertrauensärzte. Schweizerische Ärztezeitung. 2018; 99: 1571–1572. https://doi.org/10.4414/saez. 2018.17190 16. Cherny NI, Sullivan R, Dafni U, Kerst JM, Sobrero A, Zielinski C, et al. A standardised, generic, vali- dated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer ther- apies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Annals of Oncology. 2015; 26: 1547–1573. https://doi.org/10.1093/annonc/mdv249 PMID: 26026162 17. Schnipper LE, Davidson NE, Wollins DS, Tyne C, Blayney DW, Blum D, et al. American Society of Clini- cal Oncology Statement: A Conceptual Framework to Assess the Value of Cancer Treatment Options. JCO. 2015; 33: 2563–2577. https://doi.org/10.1200/JCO.2015.61.6706 PMID: 26101248 18. Paggio JCD, Sullivan R, Schrag D, Hopman WM, Azariah B, Pramesh CS, et al. Delivery of meaningful cancer care: a retrospective cohort study assessing cost and benefit with the ASCO and ESMO frame- works. The Lancet Oncology. 2017; 18: 887–894. https://doi.org/10.1016/S1470-2045(17)30415-1 PMID: 28583794 19. Cherny NI, de Vries EGE, Dafni U, Garrett-Mayer E, McKernin SE, Piccart M, et al. Comparative Assessment of Clinical Benefit Using the ESMO-Magnitude of Clinical Benefit Scale Version 1.1 and the ASCO Value Framework Net Health Benefit Score. JCO. 2018; 37: 336–349. https://doi.org/10. 1200/JCO.18.00729 PMID: 30707056 20. Swissmedic. Arzneimittelinformation. In: Swissmedic. Arzneimittelinformation [Internet]. 29 May 2021 [cited 29 May 2021]. Available: https://www.swissmedicinfo.ch/. 21. SGV. OLUtool Onko Version 2.0, August 2018. In: SGV- OLUtool Onbko Version 2.0 [Internet]. 29 May 2021 [cited 29 May 2021]. Available: https://www.vertrauensaerzte.ch/expertcom/71kvv/updmay18/ olutool_onko_2_0_2018_v_sep_18_deutsch.pdf. 22. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977; 33: 159–174. https://doi.org/10.2307/2529310 PMID: 843571 23. Chan YH. Biostatistics 104: correlational analysis. Singapore Med J. 2003; 44: 614–619. PMID: 14770254 24. Meyers DE, Jenei K, Chisamore TM, Gyawali B. Evaluation of the Clinical Benefit of Cancer Drugs Sub- mitted for Reimbursement Recommendation Decisions in Canada. JAMA Internal Medicine. 2021; 181: 499–508. https://doi.org/10.1001/jamainternmed.2020.8588 PMID: 33616606 PLOS ONE | https://doi.org/10.1371/journal.pone.0268545 June 10, 2022 13 / 13