Received: 4 February 2021 | Revised: 21 April 2021 | Accepted: 23 April 2021 DOI: 10.1002/cam4.4029 RESEARCH ARTICLE Fragility of randomized trials supporting cancer drug approvals stratified by approval pathway and review designations Brooke E. Wilson1,2 | Alexandra Desnoyers1 | Michelle B. Nadler1 | Ariadna Tibau3 | Eitan Amir1 1 Princess Margaret Cancer Centre, Department of Medical Oncology, Abstract University of Toronto, Toronto, ON, Background: It has been suggested that the results from fragile trials are less likely Canada 2 to translate into benefit in routine clinical practice. University of New South Wales, Kensington, NSW, Australia Methods: We searched the Food and Drug Administration (FDA) archives to identify 3 Oncology Department, Hospital drug approvals for solid organ malignancies between 2010 and 2019. We calculated de la Santa Creu i Sant Pau, Institut the Fragility Index (FI) supporting each approval, using methods to account for time-­ d’Investigació Biomèdica Sant Pau, and to-­event. We compared FI and trial and approval characteristics using Mann-­Whitney Universitat Autònoma de Barcelona, Barcelona, Catalonia, Spain U and Kruskal-­Wallis test. Using logistic regression, we examined study characteris- tics associated with withdrawal of consent or lost to follow-­up (WCLFU) exceeding Correspondence the calculated FI. Eitan Amir, Division of Medical Oncology and Hematology, Princess Results: The median FI among 125 included studies was 23 (range 1–­322). The FI Margaret Cancer Centre, 610 University was ≤10 in 35 studies (28%), 11–­20 in 21 (17%), and >20 in 69 (55%). The me- Ave, Toronto, ON M5G 2M9, Canada. Email: eitan.amir@uhn.ca dian FI/Nexp was 7.7% (range 0.1–­51.7%). The median FI was significantly lower among approvals processed through the accelerated vs regular pathway (5.5 vs 25, p = 0.001), but there was no difference in median FI/Nexp. The WCLFU exceeded FI in 42% of studies. Overall survival endpoints were more likely to have a WCLFU exceeding FI (OR 3.16, p = 0.003). WCLFU exceeding FI was also associated with a lesser magnitude of effect (median HR 0.69 vs 0.55, p < 0.001). In a sensitivity analy- sis including only studies with 1:1 randomization, 51% of studies had WCLFU >FI. Conclusion: The median FI among all trials was 23, and WCLFU exceeded FI in 42%. Comparative trials in solid tumors supporting approval through the accelerated pathway are more fragile compared to trials approved through the regular pathway, an observation likely explained by a lower sample size in the experimental arm. KEYWORDS accelerated approvals, FDA approvals, fragility index, trial robustness This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2021 The Authors. Cancer Medicine published by John Wiley & Sons Ltd. Cancer Medicine. 2021;10:5405–5414.  wileyonlinelibrary.com/journal/cam4 | 5405 5406 |    WILSON et al. 1 | IN T RO D U C T IO N fragility of trials, as demonstrated previously.13-­15 We have previously developed alternative methods for calculating the The US Food and Drug Administration (FDA) has developed FI.3 Among a subset of tumor types, estimated median FI was expedited review pathways and designations for approval 28.3 of drugs for diseases with high unmet need.1 Drugs can be The objectives of this study are three-­fold: (1) to calculate approved either through a regular or accelerated regulatory the FI of all comparative trials supporting solid tumor drug pathway. Regulatory approval may also be granted using an registration between 2010–­2019; (2) to perform a stratified expedited review designation (fast-­ tracked, breakthrough, analysis of FI by review pathway (accelerated vs regular) and priority review). These various pathways and designations by expedited review designation (fast-­track, break-­through or are summarized in Table S1. The accelerated regulatory rapid review); and (3) to examine characteristics associated pathway allows more rapid approval of medications based with studies where patient withdrawal of consent or loss to on a surrogate endpoint with a reasonable likelihood of pre- follow-­up (WCLFU) exceeds the FI. dicting clinical benefit. Most drugs processed through an ac- celerated regulatory approval must then fulfill post-­approval requirements including additional trials or safety analyses.2 2 | M ETHODS Breakthrough therapies are designed to expedite the approval process for drugs that demonstrate a substantial improvement 2.1 | Study selection and data collection over current available therapies, while Fast Track approvals are for drugs to treat serious conditions with unmet medical We searched the FDA archives16 to identify RCTs support- need. We hypothesized that accelerated approvals, break- ing drug approvals for solid organ malignancies (excluding through and fast-­track designations may allow for less robust lymphoma) between January 2010 and December 2019. Both (i.e. more fragile) clinical trials to support drug registration, initial approvals and expanded indications were included, speeding the time to market. In contrast, priority review sim- provided that the expanded indication was based on new ply implies a commitment to rapid processing of the applica- trial results. Only studies based on trials with comparative tion (within 6 months) and is unlikely to be correlated with data were included, as the FI cannot be calculated for non-­ less robust results. comparative data. For each identified RCT we extracted The Fragility Index (FI) is a metric quantifying the statis- the following: tumor site, year of approval, the number of tical robustness of randomized controlled trials (RCTs).3 The patients, randomization ratio, number of events, the hazard FI quantifies the internal reliability of clinical trials by esti- ratio (HR) for the outcome supporting approval, the regu- mating the number of events needed to change a statistically latory approval pathway (regular vs accelerated) and any significant result to non-­significant. In contrast to a p-­value rapid review designations (breakthrough, fast track, priority which relates to the probability that the observed results are review). The class of drug was grouped into immunotherapy, no different between comparison groups, the FI quantifies chemotherapy, monoclonal antibodies, targeted therapies (in- the difference in terms of the number of events required to cluding PARP inhibitors, CDK4/6 inhibitors, mTOR inhibi- change a trial from positive to negative. An FI of 10 in a tors and antibody-­drug conjugates), tyrosine kinase inhibitors study with 200 participants indicates an additional 10 events (TKIs), androgen receptor blockers and other. The number of in the intervention arm would render the study statistically patients who WCLFU was extracted from the CONSORT di- non-­significant. Recently, journals have placed increased em- agram. As WCLFU was not always reported clearly, we also phasis on reporting clinically meaningful results, encourag- extracted the number of patients who discontinued study drug ing a shift away from reliance on p-­values to determine the for any reason other than progression, death, adverse event importance of results.4 The FI provides a clinically tangible or completion of planned therapy (henceforth referred to as metric of the robustness of the p-­value, in meaningful units. early drug discontinuation) from the CONSORT diagram. Furthermore, the FI can be compared directly to the number of patients withdrawing consent or being lost to follow-­up, providing further insight into the internal validity of the trial 2.2 | Data synthesis and statistical analysis results, not otherwise captured by a p-­value. Since its first application in 2014,5 the FI has been as- We applied the FI framework developed by Walsh et al,5 mod- sessed in multiple areas of medicine.6-­11 However, in oncol- ified for time-­to-­event data. We reconstructed survival tables ogy, FI was previously calculated by dichotomizing the final from the published Kaplan-­Meier Curves using the Parmar event data without accounting for the time-­to-­event.12 In can- Toolkit17 ensuring estimates of effect size and power were cer, where the benefit of a drug is often measured by its abil- consistent with the primary analysis of the respective trials. ity to prolong life and/or delay disease progression or relapse, Then, we calculated the number of additional events in the ignoring the time-­to-­event occurrence can over-­estimate the experimental group that would result in a non-­significant WILSON et al.    | 5407 effect for the endpoint supporting drug approval. In studies 1 because the survival curve was not available). Therefore, with equal randomization, the FI in the experimental arm 125 studies were included in the remaining analysis closely approximates the FI in the control arm. However, for (Figure 1). Among the included studies in our cohort, there studies with unequal randomization, we present the FI for the were no instances identified where a single trial resulted in experimental (larger) arm. All data extractions and calcula- multiple approval indications. tions were performed by BW, and a sample of 70 studies were The median FI among all 125 included studies was 23 verified by a second author AD to ensure reliability of data (range 1–­322) (Figure 2). Characteristics of the included extraction (r = 0.99 between BW and AD). For studies with studies are presented in Table S2. The FI was ≤10 in 35 stud- dichotomous outcomes, we applied the original Walsh meth- ies (28%), 11–­20 in 21 studies (17%), and >20 in 69 studies odology.5 If the approval was based on multiple significant (55%). Of the 125 comparative studies included, 117 were endpoints, a hierarchy was applied with FI calculated pref- processed through regular approval (116 with time-­to-­event erentially for primary over secondary endpoints, and overall endpoints, 1 with ORR) while only 8 (6.4%) were processed survival (OS) over other co-­primary endpoints. If multiple through accelerated regulatory approval (5 with time-­ to-­ trials (or subgroups) were used for a given drug approval, event endpoints, 2 with ORR as the primary outcome and the trial (or subgroup) with the highest FI (i.e. most robust) 1 with pathological complete response [pCR]). The FI and was chosen. We then calculated the FI as a proportion of the the FI/Nexp for different trial and approval characteristics is experimental group size (FI/Nexp) to provide a standardized reported in Table 1. The median FI among trials processed measure between studies accounting for sample size. through accelerated approval (n = 8) was significantly lower We compared the association between FI and FI/Nexp with than the included studies processed through regular approval trial characteristics and the approval or rapid review pathway (n = 117) (5.5 vs 25, p = 0.001). The median FI/Nexp among using Mann Whitney U (2 groups) and Kruskal-­Wallis test all included studies was 7.7% (range 0.1 to 51.7%), and FI/ (>2 groups). Trends over time were assessed through log Nexp was similar between regular and accelerated approvals transformation of the FI or FI/Nexp (to normalize the data), (7.8% vs 7.3%, p = 0.60) (Table 1). followed by linear regression. The association between trial There was no difference in the proportion of studies characteristics and trials where WCLFU exceeds FI was ex- excluded when stratified by priority review designation amined using univariable logistic regression. HR were log (Table S3). However, a higher proportion of studies given transformed for statistical testing to ensure linearity of effect breakthrough designation were excluded from our analysis size. Multivariable analyses were not planned as the small (49.1% vs 21.4%), as a higher proportion were based on non-­ number of comparative trials supporting accelerated approval comparative or single arm trials (Table S3). In contrast, the did not allow for adequate fitting of a multivariable model. proportion of excluded studies given fast-­track designation We then performed two sensitivity analyses. First, we exam- was smaller (13.8% vs 34.7% not given fast-­track designa- ined trial characteristics associated with early drug discon- tion). Among the included studies, there was no significant tinuation being greater than the FI. Second, we included only difference in FI or FI/Nexp for drugs processed with any of studies with 1:1 randomization and re-­examined trial charac- the expedited review designations (Table 1). teristics associated with WCLFU exceeding FI. All analyses The median FI was significantly lower in studies where were performed using STATA version 12.0 (StataCorps LP). OS was the endpoint on which approval was based (12 Statistical significance was defined as p < 0.05. No correc- vs 32, p < 0.001), as was the FI/Nexp (11.7% vs 3.9%, tions were applied for multiple significance testing. p < 0.001). Higher FI was seen in studies where the con- trol arm was placebo or best supportive care compared to studies with an active control (33.5 vs 23, p = 0.04); how- 3 | R E S U LTS ever, when examining the association between FI/Nexp and the type of control arm this did not meet statistical signifi- We identified 179 drug approvals (42 accelerated and 137 cance (7.2% vs 9.1%, p = 0.7). The median FI was similar regular). After excluding non-­inferiority, biosimilar and non-­ between immunotherapy (20.5), chemotherapy (19), TKIs comparative studies, 127 approvals and their associated trials (24.5), and targeted agents (34) but lower in monoclonal were selected. Among those processed through the acceler- antibodies (5) and significantly higher in studies of andro- ated pathway, 34 (81%) were excluded from further analysis gen receptor blockers (90) (p < 0.001). In keeping with as they were based on single arm or non-­comparative stud- these results, FI/Nexp was highest in studies of androgen ies (Table S1). In contrast, among the 137 studies processed receptor blockers (16.5%), but lowest in chemotherapy through the regular approval pathway, only 20 (14.5%) studies (3.9%). The median FI was higher in prostate can- were excluded (18 based on study design being either non-­ cer trials (88) compared to other tumor groups (p < 0.001). inferiority, non-­comparative or biosimilar, 1 because the data This was driven by the larger sample size of prostate cancer could not be fitted adequately to the Kaplan-­Meier curve, and studies (median sample size 1195 prostate vs 519.5 for all 5408 |    WILSON et al. F I G U R E 1 Schema for study FDA Approvals for Solid inclusion. KM-­Kaplan-­Meier; Malignancies 2010-2019 N=179 PFS-­progression free survival; OS-­overall survival; RFS-­relapse free survival; DFS-­disease free survival; MFS-­metastasis Approval based on 2 or more studies (n=10) free survival; TTP-­time to progression; ORR-­objective response rate; pCR-­pathological complete response Number of Studies assessed N=189 Non-comparave trials (n=53) Alternave study FI used (n=5) Non inferiority trials (n=2) Biosimilar/Generic (n=2) 127 Approvals Unable to fit data (n=2) 125 Approvals Other me-to event endpoint PFS endpoint OS endpoint ORR/pCR (RFS, DFS, MFS and TTP) (n=57) (n=49) (n=4) (n=15) F I G U R E 2 Frequency of fragility index of trials in solid malignancy gaining food and drug administration approval between 2010–­2019 (n = 125) WILSON et al.    | 5409 TABLE 1 Median fragility index by trial characteristics Trial and Approval Characteristics Fragility Index Median FI/Nexp (%) p (n = 125) (range) p value Median ± SD value Regulatory Approval Pathway 0.0012 0.6 Regular (n = 117) 25 (1–­322) 7.8 (0.1–­51.7) Accelerated (n = 8) 5.5 (2–­22) 7.3 (1.8–­16.7) Breakthrough Therapy Designation 0.17 0.27 No (n = 77) 22 (1–­322) 7.1 (0.1–­51.7) Yes (n = 30) 30.5 (5–­108) 7.8 (2.5–­38.7) b N/A (n = 18) 21 (1–­68) 8.2 (0.4–­30.4) Fast Track Designation 0.66 0.52 No (n = 92) 23 (1–­322) 7.1 (0.1–­51.7) Yes (n = 25) 34 (2–­254) 10.4 (0.6–­34.2) d N/A (n = 8) 22 (1–­68) 9 (0.5–­30.4) Priority Review Designation 0.92 0.39 No (n = 31) 26 (1–­322) 7.1 (0.3–­34.5) Yes (n = 94) 22 (1–­254) 7.75 (0.1–­51.7) a Composite Rapid Review Designation 0.81 0.29 No Rapid Review Designation (n = 29) 25 (1–­322) 6.5 (0.3–­34.5) Any Rapid Review Designation (n = 96) 22 (1–­256) 7.9 (0.1–­51.7) Endpoint <0.001 <0.001 c Other (n = 76) 32 (2–­322) 11.7 (0.1–­51.7) OS (n = 49) 12 (1–­88) 3.9 (0.3–­14.6) Year of Approval 0.04 0.096 2010/12 (n = 26) 22 (1–­90) 8.85 (0.4–­34.2) 2013/15 (n = 34) 12 (1–­117) 6.95 (0.3–­44.8) 2016/17 (n = 25) 24 (2–­108) 7.6 (0.1–­38.7) 2018/19 (n = 40) 32 (3–­322) 9.9 (1.5–­51.7) Drug Class <0.001 <0.001 Immunotherapy (n = 28) 20.5 (3–­77) 6.6 (1.8–­21.4) Chemotherapy (n = 9) 12 (2–­54) 3.9 (0.4–­15) Monoclonal Antibodies (n = 17) 6 (1–­32) 2.1 (0.1–­11.2) Androgen Receptor Blockers (n = 9) 90 (20–­322) 16.5 (3.8–­34.5) TKI (n = 34) 24.5 (2–­117) 11.9 (0.4–­44.8) Targeted (n = 25) 34 (5–­87) 10.1 (1–­33.3) Other (n = 3) 43 (18–­60) 7 (2.9–­51.7) Malignancy Site 0.0013 0.12 Breast (n = 21) 24 (2–­63) 8 (0.1–­18.2) Lung (n = 26) 27 (3–­108) 11.75 (0.6–­38.7) Melanoma (n = 15) 25 (2–­93) 7.5 (1.9–­21.9) Prostate (n = 11) 88 (12–­322) 11 (3.2–­34.5) Other (n = 52) 15.5 (1–­117) 6.25 (0.3–­51.7) Setting of Approval 0.8 0.29 Metastatic (n = 113) 23 (1–­322) 7.8 (0.3–­51.7) Neoadjuvant/Adjuvant (n = 12) 24 (2–­77) 6.7 (0.1–­21.2) Control Group 0.04 0.07 Active Control (n = 89) 23 (1–­322) 7.2 (0.1–­51.7) Placebo or BSC alone (n = 36) 33.5 (3–­117) 9.1 (0.4–­44.8) Abbreviations: BSC, best supportive care; N/A, not available; OS, overall survival; TKI, Tyrosine kinase inhibitors. a Composite Rapid Review Designation includes Breakthrough, Fast Track and/or Priority Review. b Breakthrough began in 2016, studies published prior to 2016 were not eligible for this designation. c Other includes PFS, DFS, EFS and dichotomous endpoints. d Fast-­track listings are publicly available through the FDA website for all years from 1998 until 2020, except 2011 where the data are not available (n = 8, listed as N/A). 5410 |    WILSON et al. TABLE 2 Association between studies where WCLFU exceeds the fragility indexa and trial characteristics (n = 117) OR 95% CI p Fragility Index (median, range) 0.95 0.92–­0.97 <0.001 Reported HR (median, range) (n = 114)c 48.13 7.13–­323.50 <0.001 Number WCLFU (median, range) 1.04 1.02–­1.07 <0.001 WCLFU as percent of total sample size (median, 1.81 1.44–­2.28 <0.001 range) Fragility Index as proportion of experimental 0.83 0.76–­0.90 <0.001 group size (median, range) Sample Size (median, range) 1 0.99–­1.00 0.2 Year of Approval (n, %) 0.54b 2010/12 (n = 23) 1 2013/15 (n = 31) 1.28 0.42–­3.83 2016/17 (n = 25) 1.98 0.63–­6.26 2018/19 (n = 38) 0.72 0.24–­2.12 Number of Events (median, range) (n = 107) 1 0.99–­1.00 0.45 Drug Class (n, %) Immunotherapy (n = 26) 1 Chemotherapy (n = 9) 0.5 0.10–­2.43 0.39 Monocolonal Antibodies (n = 17) 1.8 0.52–­6.44 0.34 Androgen Receptor Blockers (n = 6) 0.2 0.02–­1.95 0.17 TKI (n = 31) 0.7 0.25–­2.06 0.54 Targeted (n = 25) 0.3 0.09–­1.04 0.06 Other (n = 3) 2 0.16–­24.9 0.6 Setting of Approval (n, %) Metastatic (n = 106) 1 Adjuvant (n = 11) 1.76 0.50–­6.13 0.38 Malignancy Site (n, %) Breast (n = 21) 1 Lung (n = 25) 0.73 0.22–­2.37 0.6 Melanoma (n = 13) 1.28 0.32–­5.13 0.72 Prostate (n = 8) 0.37 0.06–­2.25 0.28 Other (n = 50) 0.73 0.26–­2.05 0.55 Endpoint (n, %) Other (n = 73) 1 OS (n = 49) 3.16 1.46–­6.85 0.003 Difference in time to event outcome between 0.87 0.77–­0.99 0.04 intervention and control in months (median, range) (n = 94) Regulatory Approval Pathway (n, %) Regular (n = 111) 1 Accelerated (n = 6) 2.9 0.52–­16.7 0.22 Breakthrough Therapy Designation (n, %) No (n = 71) 1 Yes (n = 30) 0.79 0.33–­1.91 0.6 N/A (n = 16) 1.37 0.46–­4.05 0.57 Fast Track (n, %) (Continues) WILSON et al.    | 5411 TABLE 2 (Continued) OR 95% CI p No (n = 86) 1 Yes (n = 24) 0.79 0.31–­2.01 0.63 N/A (n = 7) 0.99 0.21–­4.71 0.99 Priority Review (n, %) No (n = 30) 1 Yes (n = 87) 1.34 0.57–­3.15 0.5 Control Group (n, %) Active Control (n = 83) 1 Placebo or BSC alone (n = 34) 1.13 0.51–­2.54 0.75 Abbreviations: BSC-­best supportive care; FU-­follow up; N/A-­not available; OR-­odds ratio; CI-­confidence interval; FI-­fragility index; OS-­overall survival; TKI-­ Tyrosine kinase inhibitors. a In 49 studies, the WCLFU>FI, while in 68 studies the WCLFUFI suggests a similar proportion of positive studies only 8 of 42 (19%) drugs processed through the accelerated in oncology would become non-­significant if event rates are regulatory pathway for solid malignancies had comparative higher among those WCLFU, especially if there were differ- data from which FI could be calculated, as compared to ences in the proportion of patients censored in the experi- 85% of those processed through the regular approval path- mental and control groups.22 As such, trials where WCLFU way. Therefore, our finding of lower FI (i.e. higher fragility) exceeds the FI may over-­estimate the benefits of treatment, among studies processed through the accelerated approval may be less likely show benefit in routine clinical practice, pathway only applies to comparative studies, and cannot be and may have inferior cost-­effectiveness. generalized to non-­comparative trials supporting accelerated In this study, FI is calculated based on the number of addi- approval. Instead, our results highlight an important limita- tional events in the experimental arm required to make the re- tion of single-­arm data; internal robustness of these trials sults insignificant. In trials with unequal randomization, the cannot be quantified easily. The differences in the propor- FI will differ as a function of experimental or control group tion of approvals supported by comparative data between size. Therefore, comparing FI to WCLFU in the total sample accelerated and regular approval pathways is itself import- may overestimate the number where WCLFU > FI in studies ant, and is in keeping with prior research demonstrating with unequal randomization. However, in sensitivity analysis that drugs processed through accelerated pathways are more excluding studies with unequal randomization, we found 51% likely to be single-­arm and utilize ORR as the primary out- of studies with WCLFU exceeding FI, suggesting that prior come.2 Together, these findings support the requirement for estimates may be conservative. post-­approval trials in drugs processed through the acceler- The performance of drugs in clinical trials generally ated pathway even if based on comparative data, to confirm exceeds results seen in real-­ world practice. Studies have benefit and demonstrate robust results. demonstrated an efficacy-­effectiveness gap for hepatocellular We also found a higher proportion of studies given break-­ carincoma,23 lung cancer,24,25 prostate cancer,26 breast and through designation were based on non-­comparative data and hematological malignancies.27 The efficacy-­ effectiveness excluded from our analysis. In contrast, the proportion of gap is often attributed to the differences that exist between pa- excluded studies was similar for priority review designation, tients in trials and in routine practice including clinically rel- and lower for those given fast-­track designation. FI was sim- evant differences in age, performance status, co-­morbidities ilar regardless of whether any expedited review designations and prior and subsequent lines of therapy. These differences were used, providing some reassurance that drugs used to can result in variability in toxicity and drug tolerability as treat solid malignancies given expedited review designations well as long-­term outcomes. Studies with low FI, where (priority review, fast track and breakthrough) are as robust small changes in the number of events renders the results in- as those processed without expedited designations, when ini- significant, may be more vulnerable to these differences be- tially supported by comparative data. However, these results tween clinical trials and real-­world practice and may result in cannot be generalized to non-­comparative studies. a higher efficacy-­effectiveness gap. Another concern is that the WCLFU was greater than the Over time, an increasing number of drugs are being FI in 42% of studies, and early drug discontinuation exceeded processed through the expedited development or review FI in 65% of studies. This is similar to the results by Walsh programs.28 At present, only those processed through accel- et al., evaluating studies in general medicine.5 Trials where erated approval have mandated post-­marketing requirements. WCLFU exceeds the FI should be interpreted with caution, And yet, there is evidence that over 50% of approvals have as uncaptured events in censored patients could render the not completed all post-­marketing requirements 3 years after results non-­significant. Prior research in general medicine obtaining approval,29 and in up to 25% the results may not be has shown that the median percentage of participants lost to disseminated publicly.30 Even among accelerated approvals follow-­up is 6%, but that the quality of reporting is inconsis- that have undergone confirmatory trials, only 20% demon- tent.20 We found that median percentage of patients WCLFU strated improvements in overall survival.31 As an example, among all studies was 2.9% (mean 4.2%), supporting the high Olaratumab was granted accelerated approval in 2016 based quality of oncology trials leading to FDA approvals. However, on phase Ib/II results demonstrating prolonged OS.32 Based patients rarely withdraw or are lost to follow-­up from clinical on our calculations, the FI of this study was 5. The FDA man- studies if they are doing well, and in advanced cancer, rates of dated confirmatory phase III trial ANNOUNCE33 failed to progression among censored patients are higher than in those demonstrate any improvements in OS and the approval was who remain on study.21 In a simulation study in general med- subsequently withdrawn. As the outcomes of trials are dif- icine, varying the assumptions regarding the event rate in pa- ficult to predict, studies with low fragility index (i.e. more tients lost to follow-­up caused 17% to 58% of positive studies fragile results) may not be preventable in the trial design to become non-­significant.20 While simulations were beyond stages. However, studies with low FI, regardless of whether the scope of this paper, our finding that 42% of studies had they are processed through regular or accelerated pathways WILSON et al.    | 5413 or whether they are given expedited review designations, arm. There was no difference in the FI for studies processed should be prioritized for confirmatory trials, especially if the through any of the rapid review pathways. In just under half approval was based on early phase data. We would encour- of studies, the number of patients WCLFU exceeded FI, age the FDA to look at other metrics of trial robustness and supporting the need for post-­marketing trials or real-­world internal validity, such as the FI, that might indicate whether analyses to ensure the benefit observed in clinical trials is a study result requires confirmation, regardless of approval robust and reproducible, regardless of approval pathway or pathway or expedited designation. Whether a low FI could be expedited designations. used to predict which studies based on early phase data are unlikely to confirm benefit in larger phase III studies remains ACKNOWLEDGMENTS to be seen, and ongoing research is needed. Dr. Brooke E. Wilson was supported as a National Breast This study has important limitations. There were a small Cancer Foundation of Australia International Fellow. Dr. number of comparative trials supporting accelerated approv- Michelle B. Nadler was supported as a Dream Hold ‘Em For als included in our analysis, and further research over time Life clinical oncology fellow. Dr. Alexandra Desnoyers was to expand this dataset and confirm these results is needed. supported as Hold’Em clinical oncology fellow. Accelerated approvals based on non-­comparative data were excluded from this analysis as the FI cannot be calculated, in- CONFLICT OF INTERESTS troducing sampling bias. Therefore, the conclusions regard- Dr. Eitan Amir reports personal fees from Genentech/Roche, ing lower trial robustness may only be applied to accelerated personal fees from Apobiologix, personal fees from Myriad approvals based on comparative data. Studies with smaller Genetics, personal fees from Agendia, outside the submit- sample size granted accelerated approval are likely to have ted work. No Conflict of Interest for Drs Brooke E. Wilson, lower FI than larger studies, if based on the same endpoint. Alexandra Desnoyers, Michelle B. Nadler and A. Tibau. By providing FI as a percentage of the total the experimen- tal sample size (FI/Nexp) we have provided standardization ETHICS STATEMENT between studies. Our results have also demonstrated that OS As all data were publicly available, no ethics approval was endpoints are more fragile than surrogate endpoints, which are sought. more commonly used in studies processed through the accel- erated regulatory pathway. Therefore, accelerated approvals DATA AVAILABILIT Y STATEMENT still have the potential for robust results when based on sur- The data that support the findings of this study are available rogate endpoints. Due to the small sample size of this study, from the corresponding author upon reasonable request. we were unable to fit a multivariable analysis. In this study, we applied the Walsh methodology adapted for time-­to-­event ORCID data using the Palmar toolkit. At least two other methodolo- Brooke E. Wilson https://orcid.org/0000-0001-6557-7298 gies have been applied in oncology to examine time-­to-­event Michelle B. Nadler https://orcid. data.14,15 While each applies similar principles, the resulting org/0000-0002-4372-3204 FI calculations may differ and research into comparative Eitan Amir https://orcid.org/0000-0002-3706-525X methodologies would be helpful. Finally, studies did not always present WCLFU data clearly, and therefore we may R E F E R E NC E S have underestimated the true number of patients WCLFU. 1. FDA. Guidance for Industry: Expedited Programs for Serious Furthermore, patients may be censored from the analysis for Conditions -­Drugs and Biologics. 2014. https://www.fda.gov/ reasons other than WCLFU, and this could also impact on the files/​drugs/​publi​shed/Exped​ited-­Progr​ams-­for-­Serio​us-­Condi​tions​ -­Drugs​-­and-­Biolo​gics.pdf. validity and robustness of results. Unfortunately, censoring 2. Beaver JA, Howie LJ, Pelosof L, et al. A 25-­year experience of US rates and reasons are often not clearly reported, and improved Food and Drug Administration accelerated approval of malignant transparency should be encouraged.22 hematology and oncology drugs and biologics: a review. JAMA Oncol. 2018;4(6):849-­856. 3. Desnoyers A, Wilson BE, Nadler MB, Amir E. Fragility index of 5 | CO NC LUS ION trials supporting approval of anti-­cancer drugs in common solid tumours. Cancer Treat Rev. 2021;94:102167. The median FI among all comparative trials supporting reg- 4. Harrington D, D’Agostino RB, Gatsonis C, et al. New guidelines for statistical reporting in the Journal. New England Journal of Medicine. ulatory approval in oncology between 2010 and 2019 was 2019;381(3):285–­286. https://doi.org/10.1056/nejme​1906559 23. Trials in solid tumors processed through the accelerated 5. Walsh M, Srinathan SK, McAuley DF, et al. The statisti- review pathway are more fragile compared to those pro- cal significance of randomized controlled trial results is fre- cessed through the regular approval pathway, an observation quently fragile: a case for a Fragility Index. J Clin Epidemiol. likely explained by a lower sample size in the experimental 2014;67(6):622-­628. 5414 |    WILSON et al. 6. Ridgeon EE, Young PJ, Bellomo R, Mucchetti M, Lembo R, 25. Cramer-­van der Welle CM, Peters BJ, Schramel FM, Klungel Landoni G. The fragility index in multicenter randomized con- OH, Groen HJ, van de Garde EM. Systematic evaluation of the trolled critical care trials. Crit Care Med. 2016;44(7):1278-­1284. efficacy–­effectiveness gap of systemic treatments in metastatic 7. Matics TJ, Khan N, Jani P, Kane JM. The fragility of statistically nonsmall cell lung cancer. Eur Respir J. 2018;52(6). significant findings in pediatric critical care randomized controlled 26. Templeton A, Vera-­Badillo F, Wang L, et al. Translating clinical trials. Pediatr Crit Care Med. 2019;20(6):e258-­e262. trials to clinical practice: outcomes of men with metastatic castra- 8. Tignanelli CJ, Napolitano LM. The fragility index in randomized tion resistant prostate cancer treated with docetaxel and prednisone clinical trials as a means of optimizing patient care. JAMA Surg. in and out of clinical trials. Ann Oncol. 2013;24(12):2972-­2977. 2019;154(1):74-­79. 27. Phillips CM, Parmar A, Guo H, et al. Assessing the efficacy-­ 9. Khan MS, Ochani RK, Shaikh A, et al. Fragility index in car- effectiveness gap for cancer therapies: a comparison of overall diovascular randomized controlled trials. Circ Cardiovasc Qual survival and toxicity between clinical trial and population-­based, Outcomes. 2019;12(12):e005755. real-­world data for contemporary parenteral cancer therapeutics. 10. Kruse BC, Vassar BM. Unbreakable? An analysis of the fragility Cancer. 2020;126(8):1717-­1726. of randomized trials that support diabetes treatment guidelines. 28. Kesselheim AS, Wang B, Franklin JM, Darrow JJ. Trends in uti- Diabetes Res Clin Pract. 2017;134:91-­105. lization of FDA expedited drug development and approval pro- 11. Shen C, Shamsudeen I, Farrokhyar F, Sabri K. Fragility of results grams, 1987–­2014: cohort study. BMJ. 2015;351. in ophthalmology randomized controlled trials: a systematic re- 29. Naci H, Smalley KR, Kesselheim AS. Characteristics of pre- view. Ophthalmology. 2018;125(5):642-­648. approval and postapproval studies for drugs granted acceler- 12. Del Paggio JC, Tannock IF. The fragility of phase 3 trials support- ated approval by the US Food and Drug Administration. JAMA. ing FDA-­approved anticancer medicines: a retrospective analysis. 2017;318(7):626-­636. Lancet Oncol. 2019;20(8):1065-­1069. 30. Wallach JD, Egilman AC, Dhruva SS, et al. Postmarket studies 13. Desnoyers A, Nadler MB, Wilson BE, Amir E. A critique of the required by the US Food and Drug Administration for new drugs fragility index. Lancet Oncol. 2019;20(10):e552. and biologics approved between 2009 and 2012: cross sectional 14. Bomze D, Meirson T. A critique of the fragility index. Lancet analysis. BMJ. 2018; 361. Oncol. 2019;20(10):e551. 31. Gyawali B, Hey SP, Kesselheim AS. Assessment of the clinical 15. Machado T, Duarte GS, Gonçalves N, Ferreira JJ, Costa J. A cri- benefit of cancer drugs receiving accelerated approval. JAMA tique of the fragility index. Lancet Oncol. 2019;20(10):e553. Intern Med. 2019;179(7):906-­913. 16. FDA. Drugs@FDA Database. https://www.acces​ sdata.fda.gov/ 32. Tap WD, Jones RL, Van Tine BA, et al. Olaratumab and doxorubi- scrip​ts/cder/daf/. (accessed April 2020). cin versus doxorubicin alone for treatment of soft-­tissue sarcoma: 17. Parmar MK, Torri V, Stewart L. Extracting summary statistics to an open-­label phase 1b and randomised phase 2 trial. The Lancet. perform meta-­analyses of the published literature for survival end- 2016;388(10043):488-­497. points. Stat Med. 1998;17(24):2815-­2834. 33. Tap WD, Wagner AJ, Schöffski P, et al. Effect of doxorubicin plus 18. Michiels S, Piedbois P, Burdett S, Syz N, Stewart L, Pignon J-­P. olaratumab vs doxorubicin plus placebo on survival in patients Meta-­analysis when only the median survival times are known: with advanced soft tissue sarcomas: The ANNOUNCE random- a comparison with individual patient data results. Int J Technol ized clinical trial. JAMA. 2020;323(13):1266-­1276. Assess Health Care. 2005;21(1):119-­125. 19. Vokinger KN, Hwang TJ, Grischott T, et al. Prices and clinical benefit of cancer drugs in the USA and Europe: a cost–­benefit SUPPORTING INFORMATION analysis. Lancet Oncol. 2020;21(5):664-­670. Additional supporting information may be found online in 20. Akl EA, Briel M, You JJ, et al. Potential impact on estimated treat- the Supporting Information section. ment effects of information lost to follow-­up in randomised con- trolled trials (LOST-­IT): systematic review. BMJ. 2012;344. 21. Stone A, Bushnell W, Denne J, et al. Research outcomes and How to cite this article: Wilson BE, Desnoyers A, recommendations for the assessment of progression in can- Nadler MB, Tibau A, Amir E. Fragility of randomized cer clinical trials from a PhRMA working group. Eur J Cancer. trials supporting cancer drug approvals stratified by 2011;47(12):1763-­1771. approval pathway and review designations. Cancer 22. Templeton AJ, Amir E, Tannock IF. Informative censoring—­a neglected cause of bias in oncology trials. Nat Rev Clin Oncol. Med. 2021;10:5405–­5414. https://doi.org/10.1002/ 2020;17(6):327-­328. cam4.4029 23. Sanoff HK, Chang Y, Lund JL, O’Neil BH, Dusetzina SB. Sorafenib effectiveness in advanced hepatocellular carcinoma. Oncologist. 2016;21(9):1113. 24. Sheikh N, Chambers CR. Efficacy vs effectiveness: erlotinib in previously treated non-­ small-­cell lung cancer. J Oncol Pharm Pract. 2013;19(3):228-­236.