European Journal of Heart Failure (2021) 23, 2035–2044 RESEARCH ARTICLE doi:10.1002/ejhf.2352 Head-to-head comparison of contemporary heart failure risk scores Pau Codina1,2, Josep Lupón1,2,3, Andrea Borrellas1, Giosafat Spitaleri1, Germán Cediel1,2, Mar Domingo1, Joanne Simpson4, Wayne C. Levy5, Evelyn Santiago-Vacas1,3, Elisabet Zamora1,2,3, David Buchaca6, Isaac Subirana7, Javier Santesmases1,2, Crisanto Diez-Quevedo1, Maria I. Troya1, Maria Boldo1, Salvador Altmir1, Nuria Alonso1, Beatriz González1, Carmen Rivas1, Julio Nuñez3,8,9, John McMurray4, and Antoni Bayes-Genis1,2,3* 1 HeartFailure Clinic and Cardiology Service, University Hospital Germans Trias i Pujol, Badalona, Spain; 2 Department of Medicine, Universitat Autonoma de Barcelona, Barcelona, Spain; 3 CIBERCV, Instituto de Salud Carlos III, Madrid, Spain; 4 British Heart Foundation Cardiovascular Research Centre, University of Glasgow, Glasgow, UK; 5 UW Medicine Heart Institute, University of Washington, Seattle, WA, USA; 6 Barcelona Supercomputing Center, Barcelona, Spain; 7 Hospital del Mar Medical Research Institute (IMIM), Barcelona, Spain; 8 Cardiology Department. Hospital Clínico Universitario, INCLIVA Valencia, Valencia, Spain; and 9 Departament of Medicine, Universidad de Valencia, Valencia, Spain Received 2 March 2021; revised 14 September 2021; accepted 20 September 2021 ; online publish-ahead-of-print 1 October 2021 Aims Several heart failure (HF) web-based risk scores are currently used in clinical practice. Currently, we lack head-to-head comparison of the accuracy of risk scores. This study aimed to assess correlation and mortality prediction performance of Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC-HF) risk score, which includes clinical variables + medications; Seattle Heart Failure Model (SHFM), which includes clinical variables + treatments + analytes; PARADIGM Risk of Events and Death in the Contemporary Treatment of Heart Failure (PREDICT-HF) and Barcelona Bio-Heart Failure (BCN-Bio-HF) risk calculator, which also include biomarkers, like N-terminal pro B-type natriuretic peptide (NT-proBNP). ..................................................................................................................................................................... Methods A total of 1166 consecutive patients with HF from different aetiologies that had NT-proBNP measurement at first and results visit were included. Discrimination for all-cause mortality was compared by Harrell’s C-statistic from 1 to 5 years, when possible. Calibration was assessed by calibration plots and Hosmer–Lemeshow test and global performance by Nagelkerke’s R2 . Correlation between scores was assessed by Spearman rank test. Correlation between the scores was relatively poor (rho value from 0.66 to 0.79). Discrimination analyses showed better results for 1-year mortality than for longer follow-up (SHFM 0.817, MAGGIC-HF 0.801, PREDICT-HF 0.799, BCN-Bio-HF 0.830). MAGGIC-HF showed the best calibration, BCN-Bio-HF overestimated risk while SHFM and PREDICT-HF underestimated it. BCN-Bio-HF provided the best discrimination and overall performance at every time-point. ..................................................................................................................................................................... Conclusions None of the contemporary risk scores examined showed a clear superiority over the rest. BCN-Bio-HF calculator provided the best discrimination and overall performance with overestimation of risk. MAGGIC-HF showed the best calibration, and SHFM and PREDICT-HF tended to underestimate risk. Regular updating and recalibration of online web calculators seems necessary to improve their accuracy as HF management evolves at unprecedented pace. .......................................................................................................... *Corresponding author. Heart Institute, Hospital Universitari Germans Trias i Pujol, Department of Medicine, UAB, Carretera del Canyet s/n, 08916 Badalona, Spain. Tel: +34 934 978915, Fax: +34 934 978939, Email: abayesgenis@gmail.com © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. 2036 P. Codina et al. Graphical Abstract Head-to-head comparison of contemporary heart failure risk scores. ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; ARNI, angiotensin receptor–neprilysin inhibitor; BB, beta-blocker; CRT, cardiac resynchronization therapy; ICD, implantable cardioverter-defibrillator; HFmrEF, heart failure with mildly reduced ejection fraction; HFpEF, heart failure with preserved ejection fraction; HFrEF, heart failure with reduced ejection fraction; MRA, mineralocorticoid receptor antagonist. .......................................................................................................... Keywords Heart failure • Mortality • Risk models • Risk prediction Introduction and Canada with LVEF < 30% and New York Heart Association ................................................... (NYHA) functional class IIIB to IV. The SHFM incorporates clin- Scales for predicting death in patients with heart failure (HF) ical variables, medical treatment (excluding sacubitril/valsartan), are continuously evolving, and risk prediction is a cornerstone of and conventional laboratory analytes (e.g. haemoglobin, lympho- HF management. Over the last three decades, several web-based cyte count, uric acid, total cholesterol, and sodium). Remarkably, scores have been devised to aid clinicians in assessing patient the relative effects of HF medications could not be obtained from prognosis, and ultimately, improving the appropriateness and timing the derivation cohort; instead, the benefits were estimated from of disease-modifying treatments. published trials or meta-analyses. Model discriminant ability was Among contemporary risk scores, the Seattle Heart Failure determined by the 1-year receiver operating characteristic area Model (SHFM), published in 2006, was the pioneer, and it has under the curve (AUC). The 1-year AUC for PRAISE-1, the deriva- been extensively validated.1 In the last decade of the 20th cen- tion cohort, was 0.725 [95% confidence interval (CI) 0.69–0.76]. tury, the SHFM was derived using the data previously collected The SHFM online calculator was updated in 2013 with a modifica- in the Prospective Randomized Amlodipine Survival Evaluation tion of the baseline survival function to provide a more accurate (PRAISE-1) cohort (n = 1125), and thereafter validated using five estimate of 2–5 year survival. other cohorts (n = 9942) with predominantly reduced left ven- The Meta-Analysis Global Group in Chronic Heart Failure tricular ejection fraction (LVEF). PRAISE-1 was a randomized trial (MAGGIC-HF) score was published in 2012. The MAGGIC-HF of amlodipine vs. placebo among patients in the United States score was derived around the turn of the 20th century, based © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. Contemporary heart failure risk scores 2037 on a large, heterogeneous cohort using individual data on 39 372 Despite consistent evidence that has linked NT-proBNP to an ........................................................................................................................................................................ patients with HF, both reduced and preserved LVEF, from 30 increased risk of all-cause mortality in patients with HF,8,9 this cohort studies, six of which were clinical trials. Thirteen inde- biomarker was not included in MAGGIC-HF or SHFM. In the pendent variables were identified to predict all-cause mortal- present study, we aimed to perform head-to-head comparisons of ity at 1 and 3 years,2 including age, LVEF, NYHA class, serum the MAGGIC-HF, SHFM, PREDICT-HF and BCN-Bio-HF, by deter- creatinine, diabetes, systolic blood pressure, body mass index, mining the correlation between them and comparing discrimination HF duration, current smoking, chronic obstructive pulmonary capacity for mortality risk prediction. disease (COPD), male sex, and treatment with beta-blockers, angiotensin-converting enzyme inhibitors (ACEI), and angiotensin II receptor blockers (ARBs). However, the MAGGIC-HF score does Methods not incorporate mineralocorticoid receptor antagonists (MRAs), angiotensin receptor–neprilysin inhibitors (ARNIs), or biomark- Study population and follow-up ers. Discrimination of the model in the derivation cohort was not All consecutive ambulatory patients with HF from different aetiologies available. However, the score was validated in 51 043 patients from that were admitted to a structured multidisciplinary HF clinic at the national Swedish Heart Failure Registry (SwedeHF), obtaining a university hospital between July 2010 and December 2018 were a C-index of 0.741 at 3 years.3 eligible. The patients that participated previously in the derivation cohort (May 2006–July 2010) were not included in the present study. The PARADIGM Risk of Events and Death in the Contempo- To be included the patients had to have undergone a measurement rary Treatment of Heart Failure (PREDICT-HF) was developed of NT-proBNP at the first visit. Patients were referred to the HF using data (including natriuretic peptides) from the Prospective clinic mostly by the cardiology or internal medicine departments and Comparison of ARNI With ACEI to Determine Impact on Global to a lesser extent by the emergency or other hospital departments. Mortality and Morbidity in Heart Failure (PARADIGM-HF) trial The criteria for referral to the HF clinic were HF according to patient cohort (n = 8399). Patients were eligible at screening if the ESC definition, with at least one hospitalization and/or reduced they had NYHA class II–IV, LVEF ≤35% and had an elevated natri- systolic function, as described previously.10,11 For follow-up, all patients uretic peptide level. The model was validated using the Aliskiren regularly visited the HF clinic according to their clinical needs, and they Trial to Minimize Outcomes in Patients with Heart Failure Trial were treated according to a unified protocol. Follow-up visits comprised a minimum of quarterly visits with a (ATMOSPHERE) study and the SwedeHF.4 The cohorts were nurse, one visit with a physician (cardiologist, internist, or family treated with contemporary evidence-based treatment and were physician) every 6 months, and optional visits with specialists in geographically representative, even though it did not incorporate geriatrics, psychiatry, rehabilitation, endocrinology, or nephrology. MRAs, cardiac resynchronization therapy (CRT) or implantable During the baseline visit, patients provided written consent for the cardioverter-defibrillators (ICD). The Harrell’s C-statistic was used use of their clinical data for research purposes. Demographic, clinical, to assess the discriminative ability of the model. For all-cause death, echocardiographic, and analytical data were recorded in a specific the C-statistic for 1 and 2 years was 0.71 (95% CI 0.69–0.74) and database (REGI-UNIC). Data that were not routinely recorded in that 0.70 (95% CI 0.67–0.72), respectively. database were obtained by reviewing the electronic patient health The Barcelona Bio-Heart Failure (BCN-Bio-HF) risk calculator records. We excluded patients that lacked data on more than three of the variables (five for PREDICT-HF) in any of the risk estimation was derived from consecutive outpatients (n = 864). The principal tools. referral criterion was HF according to the European Society of Car- In the present study we used the updated version of the diology (ESC) guidelines irrespective of aetiology with at least one BCN-Bio-HF calculator, which incorporated three new clinical vari- HF hospitalization or LVEF <40%. The first version of the calculator ables (duration of HF in months, number of HF-related hospitalizations was published in 2014. It included clinical variables, medications, in the preceding year, and diabetes mellitus) and four new treatments conventional laboratory analytes (sodium, estimated glomerular (MRA, ARNI, CRT, and ICD) to the original variables. In the updated filtration rate), and the following biomarkers: N-terminal pro version CRT, ICD, diabetes and MRA became significant predictors. B-type natriuretic peptide (NT-proBNP), high-sensitivity troponin Beta values for ARNI were derived from the PARADIGM-HF trial. T (hs-TnT), and interleukin-1 receptor-like-1 (known as ST2). The BCN-Bio-HF calculator can operate with none, one, two, or all three biomarkers.5 The BCN-Bio-HF was externally vali- Outcomes dated with the PROTECT cohort (Boston)6 and updated in 2018 Fatal events were identified by reviewing the patient health records by incorporating the use of ARNIs, CRT, and ICD, being this ver- from hospital wards, emergency room, and general practitioners, or sion validated with the PARADIGM-HF cohort.7 The C-statistic by contacting their relatives. Data were verified with the databases of taking into account time to-event up to 5 years with the model the Catalan and Spanish Health Systems, and also, with the Spanish National Death Index. In only a very small number of patients we containing NT-proBNP was 0.774, while AUC were 0.826 for needed to contact with their relatives and in those cases it was to 1-year, 0.809 for 2-year, 0.823 for 3-year and 0.838 for 5-year precise the exact date and cause of death (only when in the medical mortality, when considering death as a binary event. Of the records the exact date and cause of death were not reported). Annual three biomarkers, only NT-proBNP is routinely available in most all-cause death was the main endpoint for comparing the predictive HF clinics, and it is the only one we included in the current abilities of the different risk calculators. Follow-up was closed on the analyses. 30th September 2020. Observed overall mortality at 1, 3, and 5 years © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. 2038 P. Codina et al. for all models was calculated using the Kaplan–Meier estimate of Statistical analyses were performed with STATA V.15.1 software ........................................................................................................................................................................ survival. (StataCorp., College Station, TX, USA) and R software (A Language The study was performed in compliance with the laws that protect and Environment for Statistical Computing), distributed by the R Core personal data, in accordance with the international guidelines on clinical Team (2017; R 9 Foundation for Statistical Computing, Vienna, Austria). investigations from the World Medical Association’s Declaration of A two-sided P < 0.05 was considered significant. Helsinki. The local ethics committee approved the study. NT-proBNP levels were determined in plasma samples with an immuno-electrochemiluminescence assay on the Modular Analytics Results E 170 instrument (Roche Diagnostics). This assay had <0.001% cross-reactivity with bioactive B-type natriuretic peptide. In the studies Out of 1267 consecutive patients admitted to the HF clinic dur- included in this report, the assay had inter-run coefficients of variation ing the inclusion period, 22 lacked an NT-proBNP measurement, ranging from 0.9% to 5.5%. 30 were excluded because they had more than three missing val- ues in MAGGIC, SHFM or BCN-Bio-HF and 49 because they had more than five missing values in PREDICT-HF. Our final Statistical analysis cohort included 1166 patients (online supplementary Figure S1). Table 1 provides the baseline demographic, clinical, biochemical, Categorical variables are expressed as absolute numbers and percent- and echocardiographic characteristics and treatments of the stud- ages. Continuous variables are expressed as the mean ± standard devi- ied cohort. Online supplementary Table S1 compares included vs. ation or the median and interquartile range (Q1 to Q3), according to normal or non-normal data distributions. Normal distributions were excluded patients. assessed with normal Q–Q plots. Comparisons between groups were The included patients were predominantly men, aged performed with the chi-square and Fisher’s test for categorical vari- 65.9 ± 13.4 years, with reduced LVEF (37.7 ± 14.7%), and they ables, and the Student’s t-test or Mann–Whitney U test for continuous were mostly classified as NYHA class II (71.8%). Ischaemic heart variables, as appropriate. disease was the most prevalent aetiology (40.7%), followed by Missing values were treated by imputing either the default values of dilated cardiomyopathy (19.5%). Contemporary HF treatments the calculators or the median values. were optimized according to international guidelines.12 A compar- The discrimination abilities of the different prediction tools were ison was performed between surviving and non-surviving groups compared with Harrell’s C-index for 1-, 2-, 3-, 4- and 5-year all-cause at 1 year (Table 1). mortality. The Harrell’s C-index is a measure of the goodness of fit Online supplementary Table S2 shows the number of missing for models that produce risk scores in survival analyses, where data values for each calculator and default or median values used for may be censored; thus, Harrell’s C-index takes into account the time management of such missing values. to event. It measures the ability of the model to discriminate between patients that will and patients that will not experience the event. AUCs Follow-up was extended up to 5 years; the mean follow-up for considering death as a binary event at every time-point were also surviving patients was 4.2 ± 1.1 years, and the median was 5 years. assessed in a sensitivity analysis. Calibration was assessed comparing We recorded 358 deaths. Mortality rates at 1, 2, 3, 4 and 5 years the predicted and observed survival with the Kaplan–Meier estimate were 9.5%, 16.4%, 21.9%, 30.6% and 35.8%, respectively. of survival. The D’Agostino–Nam version of the Hosmer–Lemeshow Online supplementary Figure S2 shows scatter plots between the test using deciles of risk at all time-points of the study was used, and risks of death at 1 year estimated by the different calculators.. calibration plots comparing observed vs. expected events based on Correlation between the scores was relatively poor (rho value estimated risk by each calculator also by deciles, with the incorporation from 0.66 to 0.79). of LOWESS curves, which allow for the assessment of calibration The BCN-Bio-HF improved the discrimination of all-cause mor- at individual level, were plotted. Finally, overall performance of the tality risk, based on Harrell’s C-index. Moreover, the BCN-Bio-HF tools was evaluated with the Royston modification of Nagelkerke’s R2 showed the highest overall performance at every time-point that statistic for the same time-points at which mortality was recorded. Nagelkerke’s R2 statistic summarizes the proportion of variance in the risk was estimated (Table 2). Table 3 provides observed overall dependent variable associated with the independent variables. Larger survival vs. the predicted by each model, while Figure 1 shows R2 values indicated that more of the variation could be explained by calibration plots with LOWESS lines of expected and observed the model, and the maximum R2 value was 1. mortality at 1 year by risk deciles for every risk prediction tool Several sensitivity analysis were performed: (i) the performance of and the results of the Hosmer–Lemeshow test; these figures at the four calculators was assessed in patients with reduced LVEF; (ii) 2, 3 and 5 years are shown in online supplementary Figures S3–S5. the performance of the different risk prediction tools was assessed The BCN-Bio-HF systematically overestimated risk whereas SHFM applying the median of our cohort to all missing variables; (iii) a and PREDICT-HF underestimated the risk of death. MAGGIC-HF sub-analysis in 421 patients without missing values (except allopurinol showed the best calibration, although also moderately overesti- for SHFM) was performed for SHFM, MAGGIC-HF and BCN-Bio-HF; mated risk of death. Finally, Figure 2 shows survival curves based (iv) the performance of the BCN-Bio-HF model with NT-proBNP was on quintiles of risk estimation by every tool. compared with the model without NT-proBNP; (v) a sub-analysis of the BCN-Bio-HF in the last 413 patients in which ST2 and hs-TnT In the sensitivity analyses, (i) the BCN-Bio-HF model remained were available was performed, comparing its results with those of the with best performance in patients with reduced LVEF (online sup- model containing only NT-proBNP (up to 4 years); and (vi) an analysis plementary Table S3); (ii) the performance of the different risk of the BCN Bio-HF with the three biomarkers after recalculation of prediction tools applying our cohorts’ median to all missing vari- beta-coefficients in this more contemporary cohort. ables tended to show similar general results, with improvement of © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. Contemporary heart failure risk scores 2039 Table 1 Baseline population characteristics and comparison between alive and death groups at the end of the follow-up period Characteristic Total cohort Alive at end Dead at end P-value* (n = 1166) of follow-up of follow-up (n = 808) (n = 358) ........................................................................................................................................... Age, years 65.9 ± 13.4 62.5 ± 13.2 73.7 ± 10.4 <0.001 Male sex 824 (70.7) 574 (71.0) 250 (69.8) 0.705 BMI, kg/m2 26.9 [24.3–30.1] 27.1 [24.6–30.4] 26.5 [23.6–29.2] 0.011 Ischaemic aetiology 475 (40.7) 289 (35.8) 186 (51.2) <0.001 Heart failure duration, months 6 [1–36] 5 [1–28] 12 [2–48] 0.006 Hypertension 771 (66.1) 487 (60.3) 284 (79.3) <0.001 Diabetes 516 (44.3) 318 (39.4) 198 (55.3) <0.001 Current smoker 208 (17.8) 175 (21.7) 33 (9.2) <0.001 COPD 176 (15.1) 93 (11.5) 83 (23.2) <0.001 Region Central Europe 12 (1.0) 8 (1.0) 4 (1.1) 0.871 Latin America 2 (0.2) 1 (0.1) 1 (0.3) 0.559 Race/ethnicity (Asian) 2 (0.2) 1 (0.1) 1 (0.3) 0.495 Systolic BP 128.6 ± 21.6 128.0 ± 20.6 130.2 ± 23.7 0.238 NYHA functional class I 103 (8.8) 99 (12.3) 4 (1.1) <0.001 II 837 (71.8) 620 (76.7) 217 (60.6) <0.001 III 223 (19.1) 89 (11.0) 134 (37.4) <0.001 IV 3 (0.3) 0 (0) 3 (0.8) 0.006 Atrial fibrillation/flutter 265 (22.7) 162 (20.0) 103 (28.8) 0.001 LVEF, % 37.7 ± 14.7 37.0 ± 14.2 39.2 ± 15.6 <0.001 Blood tests Haemoglobin, g/dL 12.9 ± 1.9 13.3 ± 1.8 11.9 ± 1.7 <0.001 Lymphocytes, % 21.1 [16.0–27.1] 22.4 [17.2–28.3] 18.7 [13.0–24.0] <0.001 Monocytes, % 8.4 [7.1–10.1] 8.4 [7.1–10.0] 8.5 [7.0–10.2] 0.329 Absolute neutrophils, /mm3 4800 [3800–6200] 4800 [3700–6000] 4900 [3800–6600] 0.021 Sodium, mmol/L 137.6 ± 3.5 137.7 ± 3.2 137.5 ± 3.9 0.052 Potassium, mmol/L 4.3 ± 0.5 4.3 ± 0.5 4.3 ± 0.6 0.288 Chloride, mmol/L N/A N/A N/A Uric acid, μmol/L 438 ± 139.7 427.4 ± 135.3 464.0 ± 147.1 <0.001 eGFR, mL/min/1.73 m2 61.8 ± 28.5 68.0 ± 27.5 47.9 ± 25.5 <0.001 BUN, mmol/L 4.2 [3.1–6.2] 3.9 [2.9–5.4] 5.6 [3.9–8.6] <0.001 Total cholesterol, mmol/L 4.11 ± 1.16 4.22 ± 1.19 3.86 ± 1.05 <0.001 LDL, mmol/L 2.47 ± 1.02 2.57 ± 1.05 2.19 ± 0.90 <0.001 Triglycerides, mmol/L 1.43 ± 0.76 1.45 ± 0.74 1.37 ± 0.81 0.137 AST, IU/L 23 [17–35] 24 [18–36] 21 [15–31] 0.031 Bilirubin, μmol/L N/A N/A N/A Albumin, g/L 37.7 ± 5.5 38.0 ± 5.1 37.0 ± 6.3 <0.001 NT-proBNP, pg/mL 1700 [707–4238] 1310 [532–2875] 3580 [1510–7580] <0.001 Treatments (at baseline) Beta-blocker 943 (80.9) 676 (83.7) 267 (74.6) <0.001 ACEI/ARB/ARNI 799 (68.5) 596 (73.8) 203 (56.7) <0.001 Loop diuretics Furosemide >40 mg/day 622 (53.3) 409 (50.6) 213 (59.5) 0.001 Furosemide ≤40 mg/day 544 (46.7) 399 (49.4) 145 (40.5) 0.001 Weight-adjusted diuretic dose, mg/kg 0.47 [0.18–0.59] 0.45 [0.16–0.57] 0.54 [0.28–0.63] <0.001 MRA 555 (47.6) 401 (49.6) 154 (43.0) 0.08 CRT 58 (5.0) 44 (5.4) 14 (3.9) 0.165 ICD 105 (9.0) 86 (10.6) 19 (5.3) 0.002 Allopurinol N/A N/A N/A Values are mean ± standard deviation, n (%), or median [interquartile range]. ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; ARNI, angiotensin receptor–neprilysin inhibitor; AST, aspartate aminotransferase; BMI, body mass index; BP, blood pressure; BUN, blood urea nitrogen; COPD, chronic obstructive pulmonary disease; CRT, cardiac resynchronization therapy; eGFR, estimated glomerular filtration rate; ICD, implantable cardioverter-defibrillator; LDL, low-density lipoprotein; LVEF, left ventricular ejection fraction; MRA, mineralocorticoid receptor antagonist; N/A, not available; NT-proBNP, N-terminal pro B-type natriuretic peptide; NYHA, New York Heart Association. * Based on Cox regression analysis. © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. 2040 P. Codina et al. SHFM and PREDICT-HF (online supplementary Tables S4 and S5); ...................................... ........................................................................................................................................................................ Statistical comparison (Harrell’s C-index): At 1 year: P-value BCN-Bio-HF vs. SHFM 0.28; vs. MAGGIC = 0.054; vs. PREDICT-HF = 0.056/P-value SHFM vs. MAGGIC = 0.40; vs. PREDICT-HF = 0.30/P-value MAGGIC-HF vs. PREDICT-HF = 0.91. At 2 years: P-value BCN-Bio-HF vs. SHFM 0.045; vs. PREDICT-HF <0.001/P-value SHFM vs. PREDICT-HF = 0.082. At 3 years: P-value BCN-Bio-HF vs. SHFM 0.006; vs. MAGGIC-HF = 0.53/P-value SHFM vs. MAGGIC-HF = 0.69. At 4 years: P-value BCN-Bio-HF vs. SHFM 0.002. At 5 years: BCN-Bio-HF, Barcelona Bio-Heart Failure risk calculator; C-index, Harrell’s C-index for evaluating discrimination ability; MAGGIC-HF, Meta-Analysis Global Group in Chronic Heart Failure risk score; PREDICT-HF, PARADIGM Risk of Events and Death in the Contemporary ................................................................................................................................................................................................................... (iii) the sub-analysis in 421 patients without missing values showed 0.41 0.44 R2 somewhat better performance of the studied calculators (online – – supplementary Tables S6 and S7); (iv) the BCN-Bio-HF model used in the present study containing NT-proBNP, compared with the 0.742 (0.714–0.770) 0.771 (0.743–0.798) 5-year mortality model of the same tool without NT-proBNP, numerically increased C-statistic at every time-point and reached statistical significance at 1 and 4 years, while patients were significantly reclassified with C-index the biomarker addition (online supplementary Table S8); (v) the – – BCN-Bio-HF model including also ST2 and hs-TnT performed in 413 patients provided minor discrimination improvement and less risk overestimation (online supplementary Table S9); (vi) the 0.41 0.46 ................................ R2 – – BCN-Bio-HF analysis after recalibration of beta-coefficients in this more contemporary cohort, showed better results, mainly on cal- ibration (online supplementary Tables S10 and S11 and online sup- 0.740 (0.711–0.768) 0.772 (0.745–0.798) 4-year mortality plementary Figures S6 and S7). C-index Discussion – – This study compared the performances of the MAGGIC-HF, SHFM, PREDICT-HF and BCN-Bio-HF scores (the two latter included 0.44 0.47 0.53 ................................ NT-proBNP) for predicting mortality in 1166 HF outpatients R2 – managed in a multidisciplinary HF clinic. None of the contemporary risk scores examined showed a clear superiority over the rest. 0.782 (0.753–0.811) 0.753 (0.722–0.785) 0.758 (0.727–0.789) The BCN-Bio-HF calculator, which included NT-proBNP, provided 3-year mortality good discrimination and overall performance, but overestimated Table 2 Performance of the different risk prediction tools for all-cause mortality risk at all time-points, particularly for patients at high risk of death. C-index In contrast, SHFM and PREDICT-HF tended to underestimate the risk of death. MAGGIC-HF showed good calibration with modest – overestimation of risk. Our results support the routine use of natriuretic peptides in risk stratification tools for HF and eventually Treatment of Heart Failure; R2 , Royston modification of Nagelkerke’s R2 statistic; SHFM, Seattle Heart Failure Model. 0.42 0.22 0.56 ................................ R2 the inclusion of other biomarkers such as ST2 and hs-TnT to better – refine risk of death. Importantly, no patient involved in the development of the 0.788 (0.757–0.818) 0.765 (0.732–0.798) 0.739 (0.704–0.773) BCN-Bio-HF (derivation cohort) was included in the current study. 2-year mortality Risk prediction models are frequently used in HF to aid clinicians in assessing patient prognosis. Ultimately, they improve the appro- C-index priateness and timing of disease-modifying treatments. These tools are also valuable for comparing risk between populations from dif- – ferent studies. Over the last three decades, several multivariate risk models have been devised, though only some are available on the 0.45 0.55 0.25 0.66 ................................ R2 web and are currently used in clinical practice. The widely used SHFM includes important prognostic variables; however, some of these variables are not readily available in routine 0.801 (0.761–0.841) 0.817 (0.782–0.851) 0.799 (0.761–0.837) 0.830 (0.797–0.864) clinical practice. Indeed, in some of the validation cohorts for the 1-year mortality SHFM, up to 65% of uric acid levels and up to 100% of lymphocyte values were missing. Moreover, in the SHFM derivation cohort, P-value BCN-Bio-HF vs. SHFM 0.009. C-index only 3% of patients were taking an MRA, and none were prescribed a beta-blocker. Consequently, the relative effects of HF medications could not be analysed in the derivation cohort. Instead, the drug benefits were estimated from published trials or meta-analyses, PREDICT-HF MAGGIC-HF BCN-Bio-HF which usually included selected patients. Therefore, the effects of these medications may actually differ in the routine management of SHFM patients with HF. The update of this calculator performed in 2013 undoubtably has improved its performance in more contemporary © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. Contemporary heart failure risk scores 2041 Table 3 Observed vs. predicted overall survival Observeda SHFMb MAGGIC-HFb PREDICT-HFb BCN-Bio-HFb ........................................................................................................................................... 1-year survival 90.5% 94.0% 85% 94.1% 80.6% 2-year survival 83.6% 85.5% – 89.1% 66.4% 3-year survival 78.1% 83.2% 67.2% – 55.5% 4-year survival 69.4% 78.1% – – 47.1% 5-year survival 64.2% 73.2% – – 39.3% BCN-Bio-HF, Barcelona Bio-Heart Failure risk calculator; MAGGIC-HF, Meta-Analysis Global Group in Chronic Heart Failure risk score; PREDICT-HF, PARADIGM Risk of Events and Death in the Contemporary Treatment of Heart Failure; SHFM, Seattle Heart Failure Model. a Observed average survival according to Kaplan–Meier survival curve. b Average predicted survival. Figure 1 Calibration plots comparing observed vs. predicted mortality at 1 year: (A) Seattle Heart Failure Model (SHFM); (B) Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC-HF) risk score; (C) PARADIGM Risk of Events and Death in the Contemporary Treatment of Heart Failure (PREDICT-HF); (D) Barcelona Bio-Heart Failure (BCN-Bio-HF) risk calculator. Y axis, observed mortality; X axis, expected mortality; dashed line represents best fitting curve; LOWESS smoother curve (blue line) allows assessing calibration at individual patient level; circles represent groups automatically created by the test. cohorts (AUC at 1 year 0.725 in the derivation PRAISE-1 cohort, final model included 13 highly significant independent predictors of .............. Harrell’s C-statistic 0.817 in the present cohort). mortality. The MAGGIC-HF risk score is easier to use than the SHFM. The development of the MAGGIC-HF score was based on The MAGGIC-HF risk score was based on multivariable piece-wise 39 372 patients from 30 studies. Notably, only 34% of those patients Poisson regression methods with stepwise variable selection. The were treated with beta-blockers, and 21% were treated with © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. 2042 P. Codina et al. Figure 2 Survival curves based on quintiles of risk of death estimated by each calculator: (A) Seattle Heart Failure Model (SHFM); (B) Meta-Analysis Global Group in Chronic Heart Failure (MAGGIC-HF) risk score; (C) PARADIGM Risk of Events and Death in the Contemporary Treatment of Heart Failure (PREDICT-HF); (D) Barcelona Bio-Heart Failure (BCN-Bio-HF) risk calculator. Q1, lowest quintile of risk; Q5, highest quintile of risk. MRAs. As mentioned, all the subjects included in the derivation The PREDICT-HF included natriuretic peptides and the cohorts .............. cohorts for the MAGGIC-HF and SHFM scores were participating used had a more extensive and complete collection of clinical in clinical trials or registries; therefore, it is unknown how well they and laboratory data. It was derived from the PARADIGM-HF represented patients in routine clinical practice. Moreover, neither cohort and patients were receiving contemporary levels of the MAGGIC-HF nor the SHFM score incorporated biomarkers. guideline-recommended therapies. It includes some variables © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. Contemporary heart failure risk scores 2043 infrequent in routine clinical practice, such as chloride, bilirubin the cardiology department. Thus, our cohort comprised mostly ........................................................................................................................................................................ and albumin but, in contrast, it did not incorporate MRAs, CRT, relatively young men with HF of ischaemic aetiology. Consequently, or ICD. Despite a high proportion of missing values in these our results might not be generalizable to a global HF popula- variables, in our study the model showed similar discrimination tion that may include patients with HF with preserved ejection to the MAGGIC-HF and SHFM. However, the consistent absence fraction. Although patients with more than three missing values of chloride and bilirubin could have carried that PREDICT-HF lost (five for PREDICT-HF) were excluded, we could not rule out some discriminative ability. The online calculator provides the risk the possibility that some bias occurred due to the missing vari- of cardiovascular death, but does not provide this for all-cause ables. Our sample is limited, single centre, and included during mortality. a long time period. A more robust comparison of risk scores In contrast, the BCN-Bio-HF was derived from a clinical HF should ideally be undertaken in a larger multicentre contempo- cohort of patients managed in a multidisciplinary HF unit. It rary patient population. All patients were derived from the same incorporates contemporary treatments, such as ARNI, and it can clinic as the original BCN-Bio-HF calculator, so we cannot dis- operate with or without biomarkers that are known to refine card a potential bias in the analysis. Finally, the BCN-Bio-HF model pathophysiological pathways in HF. Previous studies have revealed that incorporated hs-TnT and ST2 could not be used in the that the addition of NT-proBNP to the classic scores was beneficial, whole cohort, because the measurements of these two biomark- in terms of predicting mortality in patients with HF13,14 but neither ers at the first visit were not available for a substantial proportion MAGGIC-HF nor SHFM added this biomarker into their online of patients. Although the addition of only NT-proBNP did not calculator (Graphical Abstract). improve significantly the performance of this tool in the deriva- As decisions regarding patient management may be influenced tion cohort, in the present study C-statistic numerically improved by the magnitude of the patient’s predicted risk, a model is most at any time-point, reaching statistical significance in several com- useful when it can significantly discriminate but also when is parisons and reclassification measurements were statistically sig- appropriately calibrated. In the present study, using the model nificant at any time-point. Indeed, external validation of this tool of the BCN-Bio-HF calculator containing NT-proBNP only, this has only been performed with the model containing the three score showed the best discrimination capacity, but systematically biomarkers. Yet, we could perform a sensitivity analysis with the overestimated the risk of the patients at all time-points; by contrast, three biomarkers that were available in 413 of the last included the MAGGIC-HF score showed the best calibration, although also patients, which provided an improvement of the performance of moderately overestimated the risk. On the other hand, the SHFM the calculator. and the PREDICT-HF score underestimated risk at any time-point. Finally, sensitivity analyses – especially those with subgroup The observation that the SHFM underestimated the risk in patients analysis – should be considered with caution both due the smaller at high risk of HF-related mortality at 1 year was consistent with number of patients and to potential selection bias. findings in previous studies.15 To our knowledge, this study was the first to compare these four web-based contemporary HF risk scores directly and comprehen- Conclusions sively in a real-life prospective cohort of patients managed in a mul- tidisciplinary HF clinic. The best discrimination at every time-point This study illustrates the complexity of risk prediction in that risk was estimated in BCN-Bio-HF was likely due to the incor- HF. Four comprehensive and quite contemporary online risk poration of NT-proBNP. Consequently, these data suggest that scores were compared head-to-head, and no one was found NT-proBNP should be included routinely in all risk-stratification uncontrovertibly better than the rest. BCN-Bio-HF showed tools for HF. The sensitivity analysis with such calculator including best discrimination and overall performance, and MAGGIC-HF also ST2 and hs-TnT performed in 413 patients provided a slight showed best calibration. The four studied scores either overesti- improvement of discrimination and tended to lower overestima- mated (BCN-Bio-HF, MAGGIC-HF) or tended to underestimate tion of risk. (SHFM, PREDICT-HF) the risk of death. Biomarkers, in particular Finally, the significant improvement found in the sensitivity anal- NT-proBNP, seem to provide value in risk stratification. Regular ysis of BCN-Bio-HF after its recalibration, together with the signif- updating and calibration of online web calculators are recom- icant reduction in HF mortality trends observed in recent years16 mended to keep them with value in the ever-changing landscape of and the introduction of new mainstay drugs in the treatment of HF HF management. patients, suggest that recalibration and updating of all these online calculators should be regularly performed for improving accuracy of these tools in estimating risk of death. Supplementary Information Additional supporting information may be found online in the Study limitations Supporting Information section at the end of the article. Our study has some limitations. First, our sample comprised patients with general HF. Most patients had depressed ejection Acknowledgements fractions and were treated at a multidisciplinary HF unit in a ter- We wish to thank the nurses in the heart failure unit for data tiary hospital. Additionally, most of the patients were referred from collection and for their invaluable work in the unit. © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology. 2044 P. Codina et al. 8. Hartmann F, Packer M, Coats AJ, Fowler MB, Krum H, Mohacsi P, Rouleau Funding ................................................................................................... JL, Tendera M, Castaigne A, Anker SD, Amann-Zalan I, Hoersch S, Katus The NT-proBNP assays were partially provided by Roche Diagnos- HA. Prognostic impact of plasma N-terminal pro-brain natriuretic peptide in severe chronic congestive heart failure: a substudy of the Carvedilol tics. Roche Diagnostics had no role in the design of the study or Prospective Randomized Cumulative Survival (COPERNICUS) trial. Circulation the collection, management, analysis, or interpretation of the data. 2004;110:1780–1786. Conflict of interest: A.B.G. has received lecture honoraria 9. Bettencourt P, Azevedo A, Pimenta J, Friões F, Ferreira S, Ferreira A. N-terminal-pro-brain natriuretic peptide predicts outcome after hospital from Abbott, AstraZeneca, Boehringer-Ingelheim, Novartis, Vifor, discharge in heart failure patients. Circulation 2004;110:2168–2174. Roche Diagnostics and Critical Diagnostics. A.B.G. and J.L. report a 10. Zamora E, Lupón J, Vila J, Urrutia A, de Antonio M, Sanz H, Grau M, Ara J, relationship with Critical Diagnostics. All other authors have noth- Bayés-Genís A. Estimated glomerular filtration rate and prognosis in heart failure: value of the Modification of Diet in Renal Disease Study-4, Chronic Kidney ing to disclose. Disease Epidemiology Collaboration, and Cockroft-Gault formulas. J Am Coll Cardiol 2012;59:1709–1715. 11. Lupón J, Gavidia-Bovadilla G, Ferrer E, de Antonio M, Perera-Lluna A, References López-Ayerbe J, Domingo M, Núñez J, Zamora E, Moliner P, Díaz-Ruata P, 1. Levy WC, Mozaffarian D, Linker DT, Sutradhar SC, Anker SD, Cropp AB, Santesmases J, Bayés-Genís A. Dynamic trajectories of left ventricular ejection Anand I, Maggioni A, Burton P, Sullivan MD, Pitt B, Poole-Wilson PA, Mann DL, fraction in heart failure. J Am Coll Cardiol 2018;72:591–601. Packer M. The Seattle Heart Failure Model: prediction of survival in heart failure. 12. Ponikowski P, Voors AA, Anker SD, Bueno H, Cleland JGF, Coats AJS, Falk V, Circulation 2006;113:1424–1433. González-Juanatey JR, Harjola VP, Jankowska EA, Jessup M, Linde C, Nihoy- 2. Pocock SJ, Ariti CA, McMurray JJ, Maggioni A, Køber L, Squire IB, Swedberg K, annopoulos P, Parissis JT, Pieske B, Riley JP, Rosano GMC, Ruilope LM, Rus- Dobson J, Poppe KK, Whalley GA, Doughty RN; Meta-Analysis Global Group in chitzka F, Rutten FH, van der Meer P; ESC Scientific Document Group. 2016 Chronic Heart Failure. Predicting survival in heart failure: a risk score based on ESC Guidelines for the diagnosis and treatment of acute and chronic heart fail- 39 372 patients from 30 studies. Eur Heart J 2013;34:1404–1413. ure: the Task Force for the diagnosis and treatment of acute and chronic heart 3. Sartipy U, Dahlström U, Edner M, Lund LH. Predicting survival in heart failure: failure of the European Society of Cardiology (ESC). Developed with the special validation of the MAGGIC heart failure risk score in 51,043 patients from the contribution of the Heart Failure Association (HFA) of the ESC. Eur J Heart Fail Swedish Heart Failure Registry. Eur J Heart Fail 2014;16:173–179. 2016;18:891–975. 4. Simpson J, Jhund PS, Lund LH, Padmanabhan S, Claggett BL, Shen L, Petrie 13. Khanam SS, Choi E, Son JW, Lee JW, Youn YJ, Yoon J, Lee SH, Kim JY, Ahn SG, MC, Abraham WT, Desai AS, Dickstein K, Køber L, Packer M, Rouleau JL, Ahn MS, Kang SM, Baek SH, Jeon ES, Kim JJ, Cho MC, Chae SC, Oh BH, Choi Mueller-Velten G, Solomon SD, Swedberg K, Zile MR, McMurray JJV. Prognostic DJ, Yoo BS. Validation of the MAGGIC (Meta-Analysis Global Group in Chronic models derived in PARADIGM-HF and validated in ATMOSPHERE and the Heart Failure) heart failure risk score and the effect of adding natriuretic peptide Swedish Heart Failure Registry to predict mortality and morbidity in chronic for predicting mortality after discharge in hospitalized patients with heart failure. heart failure. JAMA Cardiol 2020;5:432–441. PLoS One 2018;13:e0206380. 5. Lupón J, de Antonio M, Vila J, Peñafiel J, Galán A, Zamora E, Urru- 14. Arzilli C, Aimo A, Vergaro G, Ripoli A, Senni M, Emdin M, Passino C. N-terminal tia A, Bayes-Genis A. Development of a novel heart failure risk tool: the fraction of pro-B-type natriuretic peptide versus clinical risk scores for prognostic Barcelona Bio-Heart Failure risk calculator (BCN Bio-HF calculator). PLoS One stratification in chronic systolic heart failure. Eur J Prev Cardiol 2018;25:889–895. 2014;9:e85466. 15. Sartipy U, Goda A, Yuzefpolskaya M, Mancini DM, Lund LH. Utility of the 6. Lupón J, Januzzi JL, de Antonio M, Vila J, Peñafilel J, Bayes-Genis A. Validation of Seattle Heart Failure Model in patients with cardiac resynchronization therapy the Barcelona Bio-Heart Failure risk calculator in a cohort from Boston. Rev Esp and implantable cardioverter defibrillator referred for heart transplantation. Am Cardiol (Engl Ed) 2015;68:80–81. Heart J 2014;168:325–331. 7. Lupón J, Simpson J, McMurray JJV, de Antonio M, Vila J, Subirana I, Barallat J, 16. Spitaleri G, Lupón J, Domingo M, Santiago-Vacas E, Codina P, Zamora E, Moliner P, Domingo M, Zamora E, Bayes-Genis A. Barcelona Bio-HF Calculator Cediel G, Santesmases J, Diez-Quevedo C, Troya MI, Boldo M, Altmir S, Version 2.0: incorporation of angiotensin II receptor blocker neprilysin inhibitor Alonso N, González B, Núñez J, Bayes-Genis A. Mortality trends in an ambulatory (ARNI) and risk for heart failure hospitalization. Eur J Heart Fail 2018;20:938–940. multidisciplinary heart failure unit from 2001 to 2018. Sci Rep 2021;11:732. © 2021 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology.