www.nature.com/bmt ARTICLE OPEN Benchmarking of survival outcomes following Haematopoietic Stem Cell Transplantation (HSCT): an update of the ongoing project of the European Society for Blood and Marrow Transplantation (EBMT) and Joint Accreditation Committee of ISCT and EBMT (JACIE) ✉ Riccardo Saccardi1 , Hein Putter 2, Dirk-Jan Eikema3, María Paula Busto4, Eoin McGrath 5, Bas Middelkoop4, Gillian Adams6, Marina Atlija , Francis Ayuketang Ayuk7, Helen Baldomero 8, Yves Beguin9, Rafael de la Cámara10, Ángel Cedillo11, 4 Anna María Sureda Balari12, Christian Chabannon 13, Selim Corbacioglu14, Harry Dolstra15, Rafael F. Duarte16, Rémy Dulery 17, Raffaella Greco18,19, Andreu Gusi6, Nada Hamad 20,21,22, Michelle Kenyon23, Nicolaus Kröger 7, Myriam Labopin24,25, Julia Lee26, Per Ljungman27, Lynn Manson28, Florence Mensil29, Noel Milpied 30, Mohamad Mohty 31, Elena Oldani32, Kim Orchard33, Jakob Passweg34, Rachel Pearce26, Régis Peffault de Latour35, Hélène A. Poirel 36, Tuula Rintala37, J. Douglas Rizzo 38, Annalisa Ruggeri39, Carla Sanchez-Martinez40, Fermin Sanchez-Guijo 41, Isabel Sánchez-Ortega6, Marie Trnková42, David Valcárcel Ferreiras11,43, Leonie Wilcox44, Liesbeth C. de Wreede 2 and John A. Snowden 45 1234567890();,: © The Author(s) 2023 From 2016 EBMT and JACIE developed an international risk-adapted benchmarking program of haematopoietic stem cell transplant (HSCT) outcome to provide individual EBMT Centers with a means of quality-assuring the HSCT process and meeting FACT-JACIE accreditation requirements relating to 1-year survival outcomes. Informed by previous experience from Europe, North America and Australasia, the Clinical Outcomes Group (COG) established criteria for patient and Center selection, and a set of key clinical variables within a dedicated statistical model adapted to the capabilities of the EBMT Registry. The first phase of the project was launched in 2019 to test the acceptability of the benchmarking model through assessment of Centers’ performance for 1-year data completeness and survival outcomes of autologous and allogeneic HSCT covering 2013–2016. A second phase was delivered in July 2021 covering 2015–2019 and including survival outcomes. Reports of individual Center performance were shared directly with local principal investigators and their responses were assimilated. The experience thus far has supported the feasibility, acceptability and reliability of the system as well as identifying its limitations. We provide a summary of experience and learning so far in this ‘work in progress’, as well as highlighting future challenges of delivering a modern, robust, data-complete, risk-adapted benchmarking program across new EBMT Registry systems. Bone Marrow Transplantation (2023) 58:659–666; https://doi.org/10.1038/s41409-023-01924-6 INTRODUCTION organizations, including patients’ associations, private insurance HSCT is a complex medical procedure involving consideration of companies, accreditation bodies and health authorities [9, 10]. numerous biological and clinical components and outcomes are Benchmarking, as well as best practice management guidelines, is impacted by many factors, including patients, donors, disease and considered a reliable tool to improve the quality of clinical output transplant characteristics. The field is constantly developing, for and the proficiency in care delivery, while contributing to the example with the introduction of innovative molecules for financial sustainability of health care systems [11]. An international controlling and/or preventing relapse after the transplant [1–4], benchmarking project can provide new insights into different novel cellular therapies [5] and the growing use of alternative health systems and organizational models in HSCT practice across mismatched donors [6–8], which continually increase the com- Europe and further afield and demonstrate the need to improve plexity of clinical management and laboratory processes required and harmonize in line with an agreed standard. for high-quality patient and donor care. Procedures also come In 2016, the EBMT Board and the JACIE Committee launched a with an economic cost in terms of healthcare resources, and there benchmarking project directed at EBMT transplant Centers. is a broader responsibility for transparency called for by different According to the FACT-JACIE standards the accreditation covers A full list of author affiliations appears at the end of the paper. Received: 10 August 2022 Revised: 4 January 2023 Accepted: 16 January 2023 Published online: 9 March 2023 R. Saccardi et al. 660 the entire transplantation process, from the selection of the autologous transplants performed in the period January 1st 2015 up to donor/ patient to the follow-up, including collection, characteriza- December 31st 2019 were analyzed. Patients who received a second tion, processing and storage of the graft. A Clinical Outcomes transplant before 1 year were not censored for survival at time of the Group (COG) was established to review the existing national and second transplant. international experience and to define the fundamentals of an international system for benchmarking of survival outcomes Case-mix following HSCT. A statistical model was developed by the The variables selected as covariates for case-mix adjustment were Department of Biomedical Data Sciences of the Leiden University previously reported [12]. All data were included in the EBMT MED-A form, Medical Center (LUMC) in collaboration with the EBMT Statistical either as single values or composite scores to be calculated from the raw data. Two composite variables were included in the case-mix. Unit, taking into account case-mix correction including factors associated with disease risk, transplant technology and clinical 1. The Disease Risk Index (DRI) [13] is used as an independent disease characteristics of patients. The initial steps of the project were risk classification system and is validated for allogeneic HSCT. published in 2019 [12] and a ‘first phase’ was launched using the Adjustments have been made based on the available data. In the data extracted from the EBMT ProMISe database over a 4-year case of Myelodysplastic Syndromes (MDS) stage, all except RAEB-1 observation time period. The objectives were to test the statistical and RAEB-2 and patients transformed to Acute Myeloid Leukaemia model, to analyse the data completeness in the Registry and to (AML) are classified as low risk MDS. develop a reporting methodology for the EBMT programs selected The DRI requires information on cytogenetic abnormalities to risk- for the project. National scientific societies and registries were also stratify MDS and AML. For MDS, the categorization follows Armand’s involved to facilitate project dissemination. In 2021 a ‘second original categorization scheme [13]. For AML, cytogenetic classifica- phase’ was started, targeted at a 5-year observation period: a tion is done according to the UK Medical Research Council (MRC) cytogenetic classification [14]. For either MDS or AML, cytogenetics refinement of the case-mix factors and the data collection forms is was considered missing if no chromosome analysis (by any method) currently ongoing. We report here an update on the project: in results were registered, or if chromosome analysis results were particular, the trend to join the project by EBMT centers and a abnormal, with no further details registered. descriptive comparison of data completeness in the two analyzed The original definition of the DRI did not account for the following periods is reported. diagnostic groups; bone marrow failure, inherited disorders, auto- immune diseases, histiocytic disorders and hemoglobinopathies. In the adjusted DRI used in benchmarking, these diagnoses were classified as ‘low risk’. METHODS 2. The Hematopoietic Cell Transplantation-specific Comorbidity Index Observation periods (HCT-CI) is an independent patient-associated risk indicator, initially During the first phase, the 2013–2016 timeframe was selected. The second developed in the allogeneic setting [15] and then also reported in phase observation period was a 5-year interval, from January 1st 2015 to autologous HSCT [16, 17]. HCT-CI was calculated based on the December 31st 2019; the 1-year follow-up of patients transplanted by the individual comorbidities as reported in ProMISe: when a Center end of 2019 was collected. Future benchmarking activities will follow the indicated that any comorbidity was present, at least one comorbid- same schedule, that is: 5-year observation interval, minimum 1-year follow- ity needed to be entered. When some comorbidities were entered, up and data extraction in May of the following year. The latter allows the remaining comorbidities were assumed to be absent. Age- Centers 4 months to update the follow-up of patients transplanted by the adjusted HCT-CI [18] was not used as adjustment for age is end of the observation period. accommodated in the benchmarking statistical model. Center selection DRI and HCT-CI were analysed through the same methodology in both Transplant Centers were identified through their Center Identification Code the phases. (CIC), as extracted from ProMISe database. The criteria for patient and The complete list of covariates used in the prediction model for the allo- Center selection were agreed by the COG in the initial phase of the project and auto benchmarks is reported in the Supplementary materials #2. and were maintained throughout [12]. In particular, the minimum number of transplants per year performed in the observed period was consistent Statistical methods and data completeness with the JACIE accreditation requirements and adequate data reporting to A description of statistical methods was previously reported [12]; funnel plot the EBMT Registry was also required. The Center selection criteria are model in benchmarking analysis was also detailed elsewhere [19]. Missing reported below: values in case-mix and other baseline patient characteristics were imputed, ● Full EBMT membership prior to the estimation of the follow-up and clinical outcomes benchmarks. ● Transplants reported to the EBMT Registry >80% of the activity Multiple Imputation by Chained Equations (MICE) [20, 21] was applied in case reported in the Activity Survey of missing case mix variable values; 5 imputation datasets were generated, ● Allogeneic transplants: a minimum of 10 allografts/year on average in each by 10 iterations. Outcome information was included as the Nelson- the observation period Aalen estimator and the event indicator (dead/lost to follow-up). This ● Autologous transplants: a minimum of 5 autografts/year on average in method provides unbiased estimation of the hazard ratios of the case-mix the observation period variables, which are then used to calculate the expected number of censored patients for FU or the number of dead patients for clinical outcomes. After fitting the case-mix model for the actual benchmarking, a method Satisfactory clinical follow-up reporting is essential for fair and accurate of single value imputation is applied. Continuous variables are imputed by benchmarking. Therefore, 1-year mortality was assessed only in Centers the median value and categorical values are imputed by mode, each with a ratio of total observed to total potential follow-up higher than 80%. among 12 month survivors only. Therefore, missing values are replaced by Centers that met the threshold for 1-year follow-up received their Clinical drawing from a relatively healthy sub-population, resulting in a lower value Outcomes assessment in the same report. of the predicted probability of death, and subsequently a lower number of expected events in a Center increasing the ratio of observed and expected Selection of cases events upward. Generally, the higher the O/E ratio (observed over Only first allogeneic and autologous transplants were included in both expected), the worse the Center’s performance within the benchmarking phases, excluding solid tumors in allogeneic transplants and non- exercise. It is anticipated that this method of imputation should incentivise haematological diagnoses and paediatric patients in autologous procedures. Centers to maximise their data completeness. Data extraction Center reports The second analysis was carried out from data extracted from ProMISe on The report of each Center performance was made available to the local the 14th of May 2021. From this dataset, all first allogeneic and first Principal Investigators (PI) in both.pdf and html format and the data Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 661 Allogeneic and autologous transplants in the period 2015-2019 concerning autologous and allogeneic transplants were kept separate. The Patients: n = 162432 Centres: n = 596 report included an overview of the project and the methodology and an extended list of tables and plots describing both data completeness and Center performance. Excluding centres not full members at any time in 2015-2019 Patients: n = 8999 Centres: n = 124 Report distribution The Center PI was notified by e-mail that their report was available Patients: n = 153433 providing them with the instructions to download the individual report Centres: n = 472 through a secure web-based drive (Google). Centers that were excluded were also notified with the reason for their exclusion. In the second phase, Excluding centres <80% the Center’s data manager also received a separate notification that the Patients: n = 9366 Centres: n = 62 report was available to increase awareness within the Center that the second exercise was underway and that their Center was included. When expressly authorized by the PI, the corresponding National Registry was Patients: n = 144067 Centres: n = 410 also allowed access to the report in order to take advantage of their proximity to the Centers in their country. Excluding patients transplanted for solid tumours Patients: n = 4706 Center feedback Centres: n = 1 In both project phases, a questionnaire was circulated to the PI who had downloaded their report, to assess the interest in the project and to Patients: n = 139361 evaluate the utility and fairness of the reports, their accessibility and clarity. Centres: n = 409 Excluding autologous transplants for specific diagnoses” Patients: n = 1414 RESULTS Patient cohorts The flow-chart of the selection process in the second phase is Patients: n = 137947 reported in Fig. 1. Overall, 136,320 patients out of 162,432 (83.9%) and 395 Centers out of 596 (66.3%) were selected for the analysis. Excluding patients dead before transplant The selected Centers were then analysed for 1-year follow-up Patients: n = 21 benchmarking: 218 (76.8%) allogeneic Centers and 203 (60.2%) autologous Centers were eligible for the outcome analysis. Figure 2 Patients: n = 137926 reports the distribution of loss to 1-year FU per country. Figures 3 and 4 show the distribution in the funnel plot of Centers selected for the outcome analysis for autologous and allogeneic trans- Excluding centers transplanting fewer than 10 eligible allogeneic or 5 eligible autologous transplants per year on average plants, respectively. Each dot represents a Center, x-axis repre- Patients: n = 1254 Centres: n = 14 sents the effective sample size (adjusted for case mix and Center follow-up) whilst the y-axis shows the observed/expected 1-year Patients: n = 136672 mortality. Centers within the inner funnel are performing within Centres: n = 395 range; Centers outside the outer funnel are performing worse (upper half) or better (lower half) than average, according to a Excluding pediatric autologous transplants multiple-testing adjusted significance level of 5%. A detailed Patients: n = 352 description of the model has already been published [12]. Patients: n = 136320 Report distribution Centres: n = 395 Table 1 shows the distribution of reports over the 2 phases. For Fig. 1 Patients selection. Selection process of patients to be the small minority of Centers who reported problems accessing a analysed in the second phase of the Benchmarking Project. web-based drive, delivery of the report was facilitated either Allogeneic HSCT Autologous HSCT United Kingdom 28 United Kingdom 40 Netherlands, The 8 Netherlands, The 11 Belgium 10 Switzerland 8 50 Italy 66 Italy 6 Belgium 15 Sweden Sweden 7 France 31 Spain 26 Saudi Arabia 5 Czech Republic 7 Spain 28 Saudi Arabia 6 Germany 42 Germany 54 Czech Republic 6 Turkey 11 Austria 5 Austria 8 Poland 8 Poland 7 Turkey 16 France 35 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 2 Patient Follow up per country. The bars report the percentage of reported 1-year follow-up for Centers in countries with at least 5 Centers, divided into allogeneic (left) and autologous (right) transplants in the second phase of the project (observation interval 2015–2019). On the right side of the plots, the number of Centers per country is reported. Centers with very good follow-up (>90% of transplanted patients) were classified as “Green”; Centers with a borderline completeness (80–90%) were classified as “Amber”; Centers with a Follow-up <80% were classified as “Red” and were not selected for the outcome analysis. Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 662 One-year mortality / Autologous transplants Table 1. Report distribution at the first and second phases. 3 First phase Second phase (2013–2016) (2015–2019) N (%) Observed / Expected 2 Performance Total reports 268 (100) 395 (100) Clearly worse than average Worse than average No PI email 0 (0) 5 (1) Within range Bounced email 6 (2) 9 (2) Better than average 1 Clearly better than average Successful notifications 262 (98) 381 (97) Total Reports picked-up 127 (48) 260 (68) Of the distributed – – reports 0 CICs accessed drive 123 (97) 200 (77) 0 500 1000 1500 CICs manual distribution 4 (3) 24 (9) Sample size* Distributed by National 0 (0) 36 (14) *Adjusted for case mix and centre follow-up Registry Fig. 3 Autologous Transplants One-year mortality Funnel plot. Reports not picked-up 135 (52) 121 (32) The Funnel plot shows 1-year mortality after an autologous Numbers and percentages of report distribution in the two phases of the transplant in Centers selected for a reliable follow-up in the project. 2015–2019 interval (second phase) comparing observed over expected mortality, adjusted for case mix and Center follow-up. in 2019. The majority of improvement was observed between 2015 and 2016, mainly due to the introduction of new and more One-year mortality / Allogeneic transplants comprehensive MED-A forms in 2015. In allogeneic HSCT, DRI 3 could not be calculated due to missing data in 26.1% of cases in 2015, which was reduced over time to 9.2% in 2019 (Fig. 5d). In 2015, 23.0% of undefined DRI was due to missing cytogenetics for MDS and AML, whereas in 2019, undefined DRI due to missing Observed / Expected 2 Performance cytogenetics in MDS and AML was reduced to 6.7%. ‘Missingness’ Clearly worse than average due to disease stage remained stable over time, observed in 3% of Worse than average patients in 2015 and 2.5% in 2019. In the benchmarking model, Within range missing DRI was imputed as ‘Intermediate’. Better than average 1 Clearly better than average Management of missing variables was described in the methods. A trend of improving data completeness was observed, such as in the case of genetic markers in AML and MDS (Fig. 6), which increasingly are being entered in the database, therefore 0 allowing a better assessment of the prognosis [22]. A comparison of baseline data completeness in Centers included in both the 0 500 1000 1500 analysis is also reported in Supplementary document #3, showing Sample size* that the vast majority of Centers improved by >10% points. *Adjusted for case mix and centre follow-up Center feedback Fig. 4 Allogeneic Transplants One-year mortality Funnel plot. The Funnel plot shows 1-year mortality after an allogeneic transplant in A total of 395 Centers were eligible to participate in the June/July Centers selected for a reliable follow-up in the 2015–2019 interval 2021 Benchmarking exercise. (second phase) comparing observed over expected mortality, Approximately 95% of the Centers were successfully notified adjusted for case mix and Center follow-up. regarding the availability of their benchmarking report. 14 Centers could not be notified due to a lack of or problems with PI email directly by email to the PI with a password-protected report or via addresses. the National Registries to the Center (meaning that the National From the notified Centers, 68% (260 Centers) were able to pick Registry also accessed the report). In the first phase about half of up the report: 200 directly through Google Center PIs did not pick-up their report in spite of multiple Drive access, 36 through their national registries and 24 were reminders. Report pick-up improved in the second phase, sent directly via the Helpdesk e-mail to the Centers that were not probably due to improved communication and the support of able to access the drive themselves. the National Registries, all leading to greater awareness. Twenty-nine percent of the Centers (76) that accessed the report provided feedback. The majority (95%) of Centers that evaluated Data completeness the report found the content comprehensible. Ninety-eight percent Entering case-mix data in the Registry is essential for a fair and evaluated the quality as good and considered the current report an accurate benchmarking. Figure 5 reports the percentage of improvement on the previous release/version. Kaplan–Meier curves missing variables in the analysed cohort. For composite indices, were considered as one of the most relevant results of the report such as HCT-CI, the bars report the frequency of missing variables, (77% of the Centers found this most informative). Fifty-eight either total or partial, and the frequency of complete reports. percent of the Centers found the funnels plots to be a relevant Overall, the proportion of missing comorbidities saw a notable result for their Center. Most Centers (93%) found that the score of reduction between 2015 and 2019. The availability of HCT-CI in their program was appropriate. Ninety-eight percent of the Centers allogeneic transplants increased from 51.3% in 2015 to 90.9% found the EBMT Benchmarking project to be relevant and of in 2019. A similar pattern was observed in autologous transplants, importance. Ninety-five percent of the Centers reported a positive in which HCT-CI availability increased from 49.9% in 2015 to 89.6% impact on their data quality as a result of the benchmarking project. Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 663 a Overall missingness b Overall missingness HCT-CI 13.6% DRI* 12.8% HCT-CI 14.9% Donor age (years) 8.5% Karnofsky score 9.7% Karnofsky score 7.7% Patient CMV serostatus 2.6% DRI* 3.2% Disease stage 2.6% Sex match Disease stage 2.9% 2.4% Donor type 0.5% Stem cell source 0.4% Stem cell source 0.3% Prior autologous HSCT 0.2% Patient sex 0.2% Patient sex 0.2% Age (years) 0.1% Disease stage (AL) 0.0% Diagnosis to HSCT (months) 0.0% HSCT year 0.0% HSCT year 0.0% Diagnosis to HSCT (months) 0.0% Age (years) 0.0% Disease 0.0% Disease 0.0% 0 20 40 60 80 100 0 20 40 60 80 100 % available % available c HCT-CI completeness by year d DRI completeness by year 100 100 Complete data Any missing data All items reported At least one item missing 80 80 All items missing 60 60 40 40 20 20 0 0 2015 2016 2017 2018 2019 2015 2016 2017 2018 2019 Fig. 5 Data completeness. In the second phase of the projects (2015–2019 transplants), the lack of case-mix data is reported in allogeneic transplants (a) and autologous (b), respectively. HCT-CI was calculated based on the raw data entered in ProMISe data field; blank fields were assumed as negative. The trend to data completeness in the two composite indexes HCT-CI (c) and DRI (d) in allogeneic HSCT in the period 2015–2019 is shown in the lower panels. DISCUSSION number of issues in refinement of the EBMT benchmarking system The JACIE accreditation program was established by EBMT more and how it can be best delivered across all participating EBMT than 20 years ago with the aim of quality improvement and members (who through criteria for ‘full’ membership are required ultimately better survival outcomes in HSCT patients through a to commit to routine data submission for all transplanted patients system of external inspection and assessment of compliance to the EBMT registry) and applied across the range of relevant against minimum quality standards. The positive impact of the countries and their healthcare systems. JACIE accreditation program has been supported by a retrospective The advantages and disadvantages of EBMT-wide versus national registry analysis of outcomes in allogeneic transplantation in Europe benchmarking need to be fully considered. There is a broad range [23] with similar results also reported in North America [24]. The of ‘lost-to-follow-up’ between countries for both autologous and current and recent versions of the FACT-JACIE Standards require allogeneic HSCT (Fig. 2). If only Centers with high data completeness that Clinical Programs should achieve 1-year survival and non- are compared with countries where a minority of Centers are relapse mortality at 100 days outcomes within or above the included, Centers in some countries are disproportionately com- expected range when compared with national and international pared with others. It is not inconceivable that the ability to report outcome data and the Center should also benchmark for non- high quality data may be related to aspects of the Center, such as relapse mortality at 100 days. Thus, the EBMT-JACIE benchmarking size or quality of care which may impact on outcomes. Thus, an project has enabled Centers reporting to the EBMT registry to assess individual Center (irrespective of country) is potentially bench- their performance in a validated system of comparison with the marked against ‘high performers’ rather than ‘all Centers’ across the EBMT Registry, thereby providing an evidence-based evaluation for registry and potentially with only a small proportion of Centers in accreditation, whilst meeting the increasing clinical quality report- their own country. Some work needs to be done to address this. The ing demands of health authorities, regulators and payers. simplest way is by exploring how our model works on a country-by- There has been initial good progress in terms of both number of country basis, as well as across the whole of the EBMT. These participating Centers and data completeness, but there are a international differences in data completeness are likely to become Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 664 influence clinical decision making, especially in relation to high-risk 2nd phase treatments such as HSCT. The implementation of more comprehen- sive risk indices in the case-mix, aimed at improving the stratification 1st phase of patients in the benchmark model is expected in the next phases of 100 the project. This issue is strictly dependent on the availability of data in ProMISe, with special reference to the cytogenetic and molecular characterization of Acute Leukaemias (AL) and Myelodysplastic 80 Syndromes (MDS). In our project, the Disease Risk Index (DRI) [13] was adopted, based on disease type and status at transplant, 60 regardless of age, conditioning intensity, graft source, or donor type. % DRI was validated for risk assessment in the setting of allogeneic 40 transplantation; considering the lack of well-established prognostic indexes validated for all the diagnosis in patients undergoing an 20 autologous transplant, it was decided to incorporate the DRI for autologous transplants too. A refinement of the DRI index, including 0 2013 2014 2015 2016 2017 2018 2019 some cytogenetic markers, was introduced in the project [12, 14] in Year Acute Myeloid Leukaemia (AML) and MDS; when cytogenetic data were missing, patients were classified as “intermediate” in comput- Fig. 6 Reporting of molecular markers. The bars represent the ing the DRI score. The increased reporting of both molecular (see percentage of MED-A forms reporting 1 (blue bars), more than 1 Fig. 4) and cytogenetic markers in the registry will allow a better (green bars) or no molecular markers (red bars), per year of stratification of patients in the years to come, possibly allowing the transplant. The horizontal bars above indicate the time intervals implementation of more powerful risk indexes in both autologous analysed in the two phases of the Benchmarking project. [25] and allogeneic [26] transplantation to be implemented. The concept of “best treatment” versus “best transplant” [27] has more pronounced with the planned increase in threshold to include been proposed as a goal for JACIE in the future. Increasingly HSCT is only Centers completing more than 90% of data returns (i.e., green combined with post-transplant treatments, which reduce the cohort). It is hoped that ongoing open publication of individual incidence of relapse post-autologous HSCT (such as lenalidomide national data performance may provide impetus on a national level maintenance in myeloma) and allogeneic HSCT (with a range of to enhance data completeness across individual Centers. Such evolving treatments in AL). Whilst these treatments are clearly open, public discussion of the importance of completeness of data additional to the transplant, and may often be administered by at Center, national and international levels will encourage clinicians and teams outside of the BMT program, they may affect the appreciation of its impact on the benchmarking process (including benchmarking of outcomes in the first year and beyond. Therefore, in the need to ‘impute’ data and the potentially negative impacts on the foreseeable future, the model will also consider both pre-emptive accuracy of reports if data are incomplete), hopefully supporting the and prophylactic treatments administered after the transplant in concept of data managing as a primary requirement in process order to provide a fair clinical assessment of the process. control. This will hopefully drive improvement in reporting across all Moreover, there is the ongoing development and widespread the key variables in the benchmarking model. Clearly there are use of CAR-T and other cellular therapies across a range of some variables that are more significant than others (e.g., 1 year conditions. Such haematopoietic cellular therapies are delivered survival versus other individual variables used more for risk- alongside HSCT by transplant and haematology programs and are adaptation) and there needs to be further fine-tuning of the model subject to FACT-JACIE standards and JACIE accreditation pro- in that respect. Nevertheless, the overall message should be for cesses. At present, the available outcomes data are insufficient to Centers to aim for 100% data completeness if the goal is for their deliver a reliable benchmarking system, but with time bench- outcomes assessments to be accurate and robust to not only reflect marking of CAR-T and other cellular therapies will be an essential previous performance, but also for future quality improvement in feature of quality improvement integrated into Registry systems. patient care. It would also be a mutually useful exercise to validate Of course, the experience and learning from HSCT will be key, the EBMT model against the various well-established national including the need for accurate data completeness. benchmarking systems which informed its development, and these Some other future developments will be the inclusion of 100-days are planned in the next phase of the project. survival, as requested by the current 8th edition of the FACT-JACIE Moving forward for analysis of 2020–21 data (and potentially Standards, and a separate benchmarking process for adult and beyond), one very significant aspect to consider is how the variable paediatric patient populations. EBMT is committed to equality, impact of the COVID-19 pandemic is ‘fairly’ accommodated in the diversity and inclusion (EDI) and future consideration of benchmarking model. Clearly, benchmarking exercise can proceed as a ‘work in outcomes in relation to ethnicity, sex/gender and other characteristics progress’, but in the exceptional times of 2020–21, during which may help to identify and correct health inequalities [25]. waves of the pandemic impacted variably across geographical Overall, the EBMT community has been widely engaged in these regions with differing socio-economic factors, there were differ- early phases of the benchmarking project. The experience thus far ences in the broader public health measures and the ultimate has supported the feasibility, acceptability and reliability of the uptake of vaccination. The full appraisal of the pandemic on HSCT system as well as identifying some limitations. As the benchmarking patients is yet to be evaluated, but from an early stage there was model increasingly demonstrates its robustness, acceptability and recognition that outcomes following HSCT (and CAR-T cell therapy) relevance in modern HSCT practice (and ultimately CAR-T), there is were poorer in patients in whom SARS-2-CoV infection was an expectation that interest from patients and external stakeholders detected. In addition, there was a reduction in HSCT rates, with will grow. The benchmarking project is now a major opportunity for prioritization and likely delays in treatment that may have impacted EBMT and JACIE with direct impact on clinical care, Center upon outcomes. Moving forward, how we accommodate COVID-19 functioning and quality of survival outcomes. in the model is yet to be determined, or whether a broader compensation (or even ‘amnesty’) could be based on under- standing of the variable challenges between Centers and regions. REFERENCES Haematology, oncology and HSCT are rapidly advancing fields 1. Craddock C, Hoelzer D, Komanduri KV. Current status and future clinical direc- which has led to ongoing development of prognostic scores which tions in the prevention and treatment of relapse following hematopoietic Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 665 transplantation for acute myeloid and lymphoblastic leukemia. Bone Marrow 23. Gratwohl A, Brand R, McGrath E, van Biezen A, Sureda A, Ljungman P, et al. Use of Transpl. 2019;54:6–16. the quality management system “JACIE” and outcome after hematopoietic stem 2. Saini N, Marin D, Ledesma C, Delgado R, Rondon G, Popat UR, et al. Impact of TKIs cell transplantation. Haematologica. 2014;99:908–15. post-allogeneic hematopoietic cell transplantation in Philadelphia chromosome- 24. Marmor S, Begun JW, Abraham J, Virnig BA. The impact of center accreditation on positive ALL. Blood. 2020;136:1786–9. hematopoietic cell transplantation (HCT). Bone Marrow Transpl. 2015;50:87–94. 3. Schlenk RF, Weber D, Fiedler W, Salih HR, Wulf G, Salwender H, et al. Midostaurin 25. Shouval R, Labopin M, Bomze D, Baerlocher GM, Capria S, Blaise D, et al. Risk added to chemotherapy and continued single-agent maintenance therapy in stratification using FLT3 and NPM1 in acute myeloid leukemia patients auto- acute myeloid leukemia with FLT3-ITD. Blood. 2019;133:840–51. grafted in first complete remission. Bone Marrow Transpl. 2020;55:2244–53. 4. Burchert A. Maintenance therapy for FLT3-ITD-mutated acute myeloid leukemia. 26. Shouval R, Fein JA, Labopin M, Cho C, Bazarbachi A, Baron F, et al. Development Haematologica. 2021;106:664–70. and validation of a disease risk stratification system for patients with haemato- 5. Chabannon C, Kuball J, Bondanza A, Dazzi F, Pedrazzoli P, Toubert A, et al. logical malignancies: a retrospective cohort study of the European Society for Hematopoietic stem cell transplantation in its 60s: A platform for cellular thera- Blood and Marrow Transplantation registry. Lancet Haematol. 2021;8:e205–e15. pies. Sci Transl Med. 2018;10:eaap9630. 27. Snowden JA, McGrath E, Orchard K, Kroger N, Sureda A, Gratwohl A. Visions for a 6. Passweg JR, Baldomero H, Bader P, Bonini C, Duarte RF, Dufour C, et al. Use of JACIE quality management system 4.0. Bone Marrow Transplant. 2021;56:2876–81. haploidentical stem cell transplantation continues to increase: the 2015 European Society for Blood and Marrow Transplant activity survey report. Bone Marrow Transpl. 2017;52:811–7. AUTHOR CONTRIBUTIONS 7. Nagler A, Labopin M, Houhou M, Aljurf M, Mousavi A, Hamladji RM, et al. Out- RS and JS contributed to the study design and development, drafted and edited the come of haploidentical versus matched sibling donors in hematopoietic stem cell manuscript. D-JE designed and run the statistical analysis, drafted the report and reviewed transplantation for adult patients with acute lymphoblastic leukemia: a study the manuscript. HP designed and run the statistical analysis, reviewed the manuscript. from the Acute Leukemia Working Party of the European Society for Blood and MPB contributed to the study design, data management and analysis, reviewed the Marrow Transplantation. J Hematol Oncol. 2021;14:53. manuscript. MA contributed to data management and analisys, reviewed the manuscript. 8. Sanz J, Galimard JE, Labopin M, Afanasyev B, Sergeevich MI, Angelucci E, et al. BM contributed to the study design and data management, reviewed the manuscript. EM, Post-transplant cyclophosphamide containing regimens after matched sibling, IS-O, MM and HD contributed to the study design and development, reviewed the matched unrelated and haploidentical donor transplants in patients with acute manuscript. AG, AS, JDR, KO, ML, NK, YB, FS-G, PL and RFD contributed to the study design, lymphoblastic leukemia in first complete remission, a comparative study of the reviewed the manuscript. ÁC, AR, CC, DVF, FM, FAA, GA, HA-P, JP, JL, LCdW, MT, MK, NH, ALWP of the EBMT. J Hematol Oncol. 2021;14:84. NM, RPdL, RD, SC, HB, LW, RP, TR, LM, CS-M, EO and RdlC reviewed the manuscript. 9. Dongelmans DA, Pilcher D, Beane A, Soares M, Del Pilar Arias Lopez M, Fernandez A, et al. Linking of global intensive care (LOGIC): An international benchmarking in critical care initiative. J Crit Care. 2020;60:305–10. 10. Reponen E, Rundall TG, Shortell SM, Blodgett JC, Juarez A, Jokela R, et al. COMPETING INTERESTS Benchmarking outcomes on multiple contextual levels in lean healthcare: a AMSB declares honoraria from Takeda, BMS/Celgene, MSD, Janssen, Amgen, Novartis, systematic review, development of a conceptual framework, and a research Gilead Kite, Sanofi, Roche, Alexion and Consultancy: Takeda, BMS/Celgene, Novartis, agenda. BMC Health Serv Res. 2021;21:161. Janssen, Gilead, Sanofi, Speaker’s Bureau: Takeda and Research Support from Takeda, BMS/ 11. von Eiff W. International benchmarking and best practice management: in Celgene. GA declares paid consultancy project for Pfizer in 2021. MK declares honoraria for search of health care and hospital excellence. Adv Health Care Manag. educational events from Jazz and Mallinkrodt, for advisory board membership from Jazz. 2015;17:223–52. RPdL declares Expert consultant/speaker: Alexion, Amgen, Gilead, Jazz, Keocyte, MSD, 12. Snowden JA, Saccardi R, Orchard K, Ljungman P, Duarte RF, Labopin M, et al. Novartis, Pfizer, Roche, Samsung & Therakos. Research grant: Alexion, Amgen, Jazz Benchmarking of survival outcomes following haematopoietic stem cell trans- pharmaceutical, Novartis, Pfizer. Rémy Dulery declares personal fees from Takeda, Novartis, plantation: A review of existing processes and the introduction of an interna- Incyte and Biotest and non-financial support from Gilead, (not related the submitted work). tional system from the European Society for Blood and Marrow Transplantation S-G, Fermindeclares honoraria and/or research support from Novartis, Kite/Gilead, Celgene/ (EBMT) and the Joint Accreditation Committee of ISCT and EBMT (JACIE). Bone BMS, Pfizer, Incyte, Amgen, Takeda and Roche. RS declares honoraria and/or research Marrow Transpl. 2020;55:681–94. support from Novartis, Kite/Gilead, Jazz Pharmaceuticals, Mallinkrodt, Sanofi and Cytiva. JS 13. Armand P, Kim HT, Logan BR, Wang Z, Alyea EP, Kalaycio ME, et al. Validation and declares honoraria for educational events from Gilead and Janssen, for advisory board refinement of the Disease Risk Index for allogeneic stem cell transplantation. membership from Medac, and for trial IDMC membership from Kiadis Pharma. Blood. 2014;123:3664–71. 14. Grimwade D, Hills RK, Moorman AV, Walker H, Chatters S, Goldstone AH, et al. Refinement of cytogenetic classification in acute myeloid leukemia: determina- ADDITIONAL INFORMATION tion of prognostic significance of rare recurring chromosomal abnormalities Supplementary information The online version contains supplementary material among 5876 younger adult patients treated in the United Kingdom Medical available at https://doi.org/10.1038/s41409-023-01924-6. Research Council trials. Blood. 2010;116:354–65. 15. Sorror ML, Storer B, Storb RF. Validation of the hematopoietic cell transplantation- Correspondence and requests for materials should be addressed to Riccardo specific comorbidity index (HCT-CI) in single and multiple institutions: limitations Saccardi. and inferences. Biol Blood Marrow Transpl. 2009;15:757–8. 16. Berro M, Arbelbide JA, Rivas MM, Basquiera AL, Ferini G, Vitriu A, et al. Hema- Reprints and permission information is available at http://www.nature.com/ topoietic cell transplantation-specific comorbidity index predicts morbidity and reprints mortality in autologous stem cell transplantation. Biol Blood Marrow Transpl. 2017;23:1646–50. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims 17. Saad A, Mahindra A, Zhang MJ, Zhong X, Costa LJ, Dispenzieri A, et al. Hema- in published maps and institutional affiliations. topoietic cell transplant comorbidity index is predictive of survival after auto- logous hematopoietic cell transplantation in multiple myeloma. Biol Blood Marrow Transpl. 2014;20:402–8.e1. Open Access This article is licensed under a Creative Commons 18. Sorror ML, Storb RF, Sandmaier BM, Maziarz RT, Pulsipher MA, Maris MB, et al. Attribution 4.0 International License, which permits use, sharing, Comorbidity-age index: a clinical measure of biologic age before allogeneic adaptation, distribution and reproduction in any medium or format, as long as you give hematopoietic cell transplantation. J Clin Oncol. 2014;32:3249–56. appropriate credit to the original author(s) and the source, provide a link to the Creative 19. Putter H, Eikema DJ, de Wreede LC, McGrath E, Sanchez-Ortega I, Saccardi R, et al. Commons license, and indicate if changes were made. The images or other third party Benchmarking survival outcomes: A funnel plot for survival data. Stat Methods material in this article are included in the article’s Creative Commons license, unless Med Res. 2022;31:1171–1183. indicated otherwise in a credit line to the material. If material is not included in the 20. van Buuren S, Boshuizen HC, Knook DL. Multiple imputation of missing blood article’s Creative Commons license and your intended use is not permitted by statutory pressure covariates in survival analysis. Stat Med. 1999;18:681–94. regulation or exceeds the permitted use, you will need to obtain permission directly 21. White IR, Royston P. Imputing missing covariate values for the Cox model. Stat from the copyright holder. To view a copy of this license, visit http:// Med. 2009;28:1982–98. creativecommons.org/licenses/by/4.0/. 22. Dohner H, Estey E, Grimwade D, Amadori S, Appelbaum FR, Buchner T, et al. Diagnosis and management of AML in adults: 2017 ELN recommendations from an international expert panel. Blood. 2017;129:424–47. © The Author(s) 2023 Bone Marrow Transplantation (2023) 58:659 – 666 R. Saccardi et al. 666 1 Cellular Therapy and Transfusion Medicine Unit, Careggi University Hospital, Florence, Italy. 2Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands. 3EBMT Statistical Unit Leiden, Leiden, the Netherlands. 4EBMT Registry, Leiden, the Netherlands. 5ICCBBA, Barcelona, Spain. 6EBMT Executive Office, Barcelona, Spain. 7Department of Stem Cell Transplantation, University Medical Center Hamburg, Hamburg, Germany. 8Centre for Hematology and Oncology, University Hospital Basel, Basel, Switzerland. 9Department of Hematology, CHU of Liège and University of Liège, Liège, Belgium. 10Infectious Diseases Working Party (IDWP), Barcelona, Spain. 11Grupo Español de Trasplantes Hematopoyéticos y Terapia Celular (GETH-TC), Madrid, Spain. 12Hematology Department, Institut Català d’Oncologia-Hospitalet, Barcelona, IDIBELL, Universitat de Barcelona, Barcelona, Spain. 13Centre de Thérapie Cellulaire, Institut Paoli-Calmettes Comprehensive Cancer Centre, Aix-Marseille Université School of Medicine & Inserm CBT-1409, Centre d’Investigations Cliniques en Biothérapies, Marseille, France. 14Department of Paediatric Oncology, Haematology and Stem Cell Transplantation, University Children’s Hospital Regensburg, Regensburg, Germany. 15Laboratory of Hematology, Department of Laboratory Medicine, Radboud University Medical Center, Nijmegen, the Netherlands. 16Hospital Universitario Puerta de Hierro Majadahonda, Universidad Autónoma de Madrid, Madrid, Spain. 17 Sorbonne Université, INSERM UMRs938, Service d’Hématologie Clinique et de Thérapie Cellulaire, Hôpital Saint Antoine, AP-HP, Paris, France. 18Blood and Marrow Transplant (BMT) Unit, IRCCS San Raffaele Hospital, Milan, Italy. 19Autoimmune Diseases Working Party (ADWP), EBMT, Barcelona, Spain. 20Department of Haematology, St Vincent’s Hospital Sydney, Sydney, NSW, Australia. 21School of Clinical Medicine, UNSW Medicine & Health, Kensington, NSW, Australia. 22School of Medicine, Sydney, University of Notre Dame Australia, Sydney, WA, Australia. 23Kenyon, M, Department of Haematology, King’s College Hospital NHS Foundation Trust, London, United Kingdom. 24Sorbonne University, Department of Hematology, Hôpital Saint Antoine, and INSERM UMRs 938, Paris, France. 25EBMT Paris Study Office, Paris, France. 26British Society of Blood and Marrow Transplantation and Cellular Therapy (BSBMTCT) Data Registry, London, United Kingdom. 27Department of Cellular Therapy and Allogeneic Stem Cell Transplantation, Karolinska University Hospital, Division of Hematology, Department of Medicine Huddinge, Karolinska Institutet, Stockholm, Sweden. 28Edinburgh & SE Scotland Clinical Transfusion Centre Patient Services and the Edinburgh & SE Scotland Clinical Apheresis Unit, Edinburgh, United Kingdom. 29Agence de la Biomédecine, Saint-Denis, France. 30Centre Hospitalier Universitaire (CHU) Bordeaux, Service d’Hematologie et Therapie Cellulaire, Bordeaux, France. 31Sorbonne University, Hôpital Saint Antoine (AP-HP), and INSERM UMRs 938, Paris, France. 32Italian National BMT Registry (GITMO), Bergamo, Italy. 33University Hospital Southampton NHS, London, United Kingdom. 34Hematology division, Basel University Hospital, Basel, Switzerland. 35BMT unit Paris Cité University, French reference center for aplastic anaemia and PNH, French rare network for rare immunological and haematological disorders (MaRIH), Severe aplastic anaemia working party of the European blood and marrow transplantation group (SAAWP EBMT) Saint-Louis Hospital, Paris, France. 36Belgian Transplant Registry, Belgian Cancer Registry, Brussels, Belgium. 37King’s College Hospital NHS Foundation Trust, London, United Kingdom. 38CIBMTR, Medical College of Wisconsin, Milwaukee, WI, USA. 39Hematology and Bone Marrow Transplantation Unit, IRCCS San Raffaele Scientific Institute, Milano, Italy. 40JACIE, EBMT Executive Office, Barcelona, Spain. 41CIC-University of Salamanca, IBSAL-University Hospital of Salamanca, Salamanca, Spain. 42Czech registry, Prague, Czech Republic. 43Hematology Department, Vall d’hebron Institut Oncologic (VHIO), Hospital Vall d’hebron, Universitat Autònoma de Barcelona, Barcelona, Spain. 44Australasian Bone Marrow Transplant Recipient Registry (ABMTRR), Darlinghurst, NSW, Australia. 45Department of Haematology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, United Kingdom. ✉email: riccardo.saccardi@aouc.unifi.it Bone Marrow Transplantation (2023) 58:659 – 666