Informatics in Medicine Unlocked 19 (2020) 100351 Contents lists available at ScienceDirect Informatics in Medicine Unlocked journal homepage: http://www.elsevier.com/locate/imu Mortality prediction enhancement in end-stage renal disease: A machine learning approach Edwar Macias a, *, Antoni Morell a, Javier Serrano a, Jose Lopez Vicario a, Jose Ibeas b a Wireless Information Networking (WIN) Group, Universitat Aut� onoma de Barcelona (UAB), Bellaterra, 08193, Spain b Nephrology Department, Parc Taulí Hospital Universitari, Institut de Investigaci� o i Innovaci� o Parc Taulí I3PT. Universitat Aut� onoma de Barcelona, Sabadell, 08208, Spain A R T I C L E I N F O A B S T R A C T Keywords: In this work, we propose to combine massive variables collected during the evolution of patients in end-stage Mortality prediction renal disease (ESRD), along with machine learning techniques to improve mortality prediction in ESRD. This End-stage renal disease work was carried out with a retrospective cohort of 261 patients, their evolution from diagnoses, laboratory tests, Machine learning and variables recorded during haemodialysis sessions was combined. Random forest (RF) was used to explore the LSTM Random forest inference of the variables and define a base performance for long short-term memory (LSTM) recurrent neural networks. Then, LSTMs were trained with several groups of variables chosen by expert staff, the ones found by RF and all the available ones. The best performance was obtained using all the variables, but the ones found by RF had better predictive capacity than those chosen with expert knowledge. Integrating the three sources of in­ formation supposes an improvement in more than 4% in the area under the receiver operating characteristic curve. The approach is sufficiently robust to predict mortality at different time ranges. The massive integration of variables from patients in ESRD, together with the use of LSMTs, supposes an exceptional improvement in the predictive models of mortality. In conclusion, the machine learning approach can lead to a change in the paradigm in the analysis of predictive factors in mortality in ESRD. 1. Introduction an open issue. Thus, in the era of machine learning (ML), it is possible to generate Chronic kidney disease (CKD) represents an epidemiological prob­ complex models supported by large amounts of data [8–10]. Moreover, lem, USA 11% and Spain 9.2% in the adult population [1]. According to large-scale studies have begun to be described with ML to establish the World Health Organization (WHO), it has an indirect impact on the prognosis of mortality in the general population using routine clinical morbidity and mortality of the global population, increasing the mor­ data [11–14]. However, those that exist in ESRD use approaches based tality risk of the deadliest diseases [2,3]. CKD is closely related to car­ on classical statistics [15–18] and some of them present a doubtful diovascular (CV) risk, which is responsible for the highest mortality, benefit [19]. especially on the end-stage renal disease (ESRD), where death from CV is There are few studies where ML techniques are applied to CKD. one of the leading causes [1]. Salekin [20] and Abdullah [21] detect CKD using different classifiers The most widely used way to detect the risk of suffering these kinds (support vector machine, k-Nearest Neighbors, Random Forest (RF) and of pathologies is based on evidence-based medicine, which is translated artificial neural networks (ANN)), Doi [22] trains logistic regression to into best practice guidelines, such as the American Heart Association/ predict mortality in patients starting with haemodialysis, and Tita­ American College of Cardiology (ACC/AHA) [4], QRISK2 [5], Fra­ piccolo [23] stratifies cardiovascular risk with RF. Predictive models of mingham [6] or Reynolds [7]. They are based on assuming linear re­ mortality using ML are even scarcer in the ESRD population, Akbilgic lationships between risk factors and events. Nevertheless, the [24] used RF to predict mortality from one month to one year with an application of more sophisticated algorithms that use non-linear re­ Area Under the Receiver Operating Characteristics (AUROC) of 0.736. lationships, and can offer better performance in predictive models is still The aim of this manuscript is twofold, to present an exploratory * Corresponding author. E-mail addresses: edwar.macias@uab.cat (E. Macias), antoni.morell@uab.cat (A. Morell), javier.serrano@uab.cat (J. Serrano), jose.vicario@uab.cat (J.L. Vicario), jibeas@telefonica.net (J. Ibeas). https://doi.org/10.1016/j.imu.2020.100351 Received 15 February 2020; Received in revised form 10 May 2020; Accepted 10 May 2020 Available online 30 May 2020 2352-9148/© 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 analysis of the potential by using massive variables and exploit temporal variables are filtered based on their percentage of missing values (MV), dependencies through long short-term memory (LSTM) recurrent neural variables with more than 43.2% of MVs are discarded. In Table 1 can be networks (RNN) for improving predictive models of mortality in ESRD. appreciated the selected features. Next are described the most relevant It is evaluated the predictive capacity of several groups of variables. This sources of information for this study. study also takes advantage of the number of samples generated by the continuous monitoring of patients to propose predictive models of short- 2.1.1. Diagnoses term mortality, which in our knowledge have not yet been achieved an Refers to the historical hospital admissions that a patient has had. AUROC higher than 0.736. This study points to the potential benefits of Each entry is associated with some particular diagnoses determined by ML approaches to assess the medical staff with ESRD patients. It en­ examinations and evaluations of medical staff, which is encoded using courages the development of more powerful models using specialized the international classification of diseases (ICD9). ANNs as a predictive mechanisms. 2.1.2. Laboratory 2. Materials and methods All the associated variables with samples from haematology, biochemistry, or some ESRD related hormones are stored as laboratory This retrospective study was carried out on a homogenous cohort of events. Some of them are taken with more or less periodicity. For 1178 HD patients from a single centre with a reference population of instance, the most regular is the haemoglobin, which is measured every almost half a million inhabitants. Of 1178, it was possible to extract month, while proteins and PTH are measured every four months. Other information from 537 deceased patients, and of these, 261 provided the measurements like immunology or tumour markers are taken more necessary massive data. These data were taken from the Information System of the Parc Tauli University Hospital, from the Haemodialysis (HD) Unit at the Nephrology Department from 2007 to 2018. This Table 1 project passed through the ethics committee (Code 2018/508) and Selected variables from the data sources. The outcome is coded according to the death date of the patient. SBP and DBP refer to systolic and diastolic blood subsequently anonymised following the usual protocol. Inclusion pressure, HR to heart rate and Temp to temperature. criteria was being of legal age (> 18years). The available data include diagnoses, laboratory tests, and variables from haemodialysis sessions. Laboratory tests Haemodialysis Diagnoses The exposure period is from the moment the information of a patient is Calcium Acc weight Arteriopathy registered in ESRD in the information system of the hospital until the Creatinine Average flow Cardiopathy death of the patient. Ferritin Blood vol dia Diabetes Glucose DBP post HD Enteropathy To exploit the predictive capacity of variables and their temporal Haemoglobin DBP pre HD Fracture dependencies in the follow-up of patients in ESRD, data were first Haemoglobin Dry weight Haemorrhage selected, then pre-processed and finally, the predictive models were HDL cholesterol HD time Hepatopathy generated in two stages. The first one using RF, due to its easy to tune Hematocrit HR post HD Hypertension Iron HR pre HD Infection and computational cost, to find the most important variables and for KTV Hypotension Neoplasia setting a baseline performance for more sophisticated algorithms. The LDL cholesterol SBP post HD Pneumopathy second stage has the twofold purpose of exploiting temporal de­ leucocytes SBP pre HD pendencies through LSTMs and analysing the impact of sets of variables Lymphocytes Temp post HD including the ones found in the previous stage, groups of variables Monocyts Temp pre HD Demographics Neutrophil Vascular access Age chosen by the expert staff and using all the available ones. All the Phosphorus Sex necessary steps to carry out the prediction of mortality for patients in Platelets ESRD can be seen in Fig. 1 and are described below. Potassium PTH Outcome Reticulocytes Months to decease 2.1. Data selection Sodium Total cholesterol Total proteins Variables from the history of diagnoses, laboratory tests, HD sessions Triglycerides and demographics are taken as input for developing the predictive Urea models. The outcome to predict is the mortality of patients. The Fig. 1. Framework for developing predictive models in ESRD, G11 to G33 refer to set of variables ranked by their importance based on the expert staff experience. In pre-processing stage some features were generated based on 1-hot encoded for categorical features. 2 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 exceptionally. features for the final predictor. Then, due to the structure of the data, the different temporal dependencies are exploited using LSTMs. 2.1.3. HD variables During the HD sessions, 3–4 per week, some variables are recorded at 2.3.1. Feature selection-Random forest the beginning and the end of the HD session. Registered information RF combines predictions based on decision trees [25]. They are includes the type of vascular access, duration of the session, episodes of trained with random subsets of data Dn . Branches of the decision trees hypotension and other variables taken from the haemodialysis machine, are generated based on the calculation of the impurity of their features such as dry weight, temperature, systolic and diastolic blood pressure, through the Gini index, heart rate, average flow, among others. X m GðDn Þ ¼ 1 p2i (1) 2.2. Data pre-processing i¼1 where m is the number of classes (2 in our case, dead or alive), and pi is In general, raw data from electronic health records (EHR) are not the relative frequency of class i in a given branch of the tree. Initially, appropriately structured to generate or test learning models. Thus, to GðDn Þ is calculated for all the possible combinations of features and prepare the information for predictive models, it is necessary to explore splitting thresholds. The combination which achieves the lowest value of the variables, clean them and coherently correct wrong values. Below GðDn Þ is chosen as far as it represents the best possible classification of are the problems found in the samples. Dn at this point of the tree. In subsequent branches, the same procedure is repeated up to the specified depth. In an RF approach, several trees are � Data structure computed and fed with subsets of the data. Finally, the outcome pro­ � Incorrect values in variables duced by most of the trees is taken as the final decision (see Fig. 3). � Missing values On the other hand, the Gini index allows quantifying the importance of the features. This characteristic is used in this work to find the most Initially, the information has to be structured. The diagnoses were relevant variables, more robustly, for predictors by combining a recur­ grouped into 11 general ones based on expert knowledge. Then, they sive feature elimination (RFE) [26] approach with RF. The traditional were structured in such a way that they became new variables for the way to find the importance of features is to relate them individually with final data set. In Fig. 2, this transformation can be appreciated. Later, the the outcome, without taking into account the interactions between three information sources are combined based on their date they were variables. RFE solves this issue generating several predictors iteratively. measured. Finally, the follow-up of patients was summarized into one- Thus, in each iteration a predictor offer a performance measurement and month records, i.e., using the mean of variables in case of having the ranking of features. In the next iteration, the less important feature is more than one sample per month. To correct the outliers and imputed the MVs, the ranges of the var­ iables in laboratory tests and HD sessions were decided by the expert staff. The outliers of variables are identified and replaced with MVs to avoid losing the time stamp of variables. Then, the MVs are treated in two stages. The first one based on the individual imputation of the variables of each patient using second-order interpolations, to preserve trends in the evolution of the patient. In the second stage, MVs are imputed for patients without samples in some variables. Thus, from patients in training set, without MVs, the average value from each variable is extracted and used to impute MVs of the remaining patients. 2.3. Learning models Initially, RF was used with the twofold purpose of establishing a Fig. 3. Random forest data flow, at the end the class decision is made by voting baseline performance in terms of prediction, due to its easy set of pa­ of each tree. rameters and computational cost, and quantifying the importance of the Fig. 2. Initially, each entry is associated with a series of diagnoses. In the new scheme, the most important diagnoses are selected and coded using one-hot encoding. 3 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 eliminated and the new predictor will yield another performance and a �n o� �n o� λ XL X Nl � Nl 1 X �2 new ranking, and so on. C’ wli;j ¼ C wli;j þ wli;j (4) 2N l¼1 i¼1 j 2.3.2. Predictive model-LSTM With L2 regularization, controlled by λ, we limit the adaptation ca­ ANNs have been successfully applied in the past to classification pacity of the network by penalizing large weights. problems. The goal is, given a set of N input examples xn , where n ¼ 1; Training is carried out with a gradient descent approach and the so- …; N and its corresponding classification targets tn ¼ ½t1;n ; …; tK;n �, to called back-propagation algorithm [28]. Thus, weights of the network learn the best non-linear model that maps the input to its respective are updated iteratively towards the opposite of the gradient, being the target. Note that tn has all values equal to 0 except the class where xn step controlled by the learning rate (LR). Currently, some algorithms belongs to, which has a value of 1. In the case of the prediction of accelerate the learning process by dynamically changing the LR. In this mortality of patients in ESRD, the objective is to classify data collected work we consider adaptive moment estimation (ADAM) [29]. during a period of ndata months, to be able to determine if the patient will On the other hand, RNNs are a variation of the networks shown in be alive or not after npred months. Thus, driven by a considerable number Fig. 4. Unlike feed-forward ANNs, RNNs use feedback connections to of training samples, an ANN can learn an optimised non-linear function retain information about past events. In recent years one of the RNN in an iterative process so that the error between the input and the output implementations that have been successful is LSTM. To carry out mor­ is minimised. Fig. 4 shows the structure of a basic ANN. tality prediction in ESRD, LSTMs are used to exploit temporal de­ An ANN is composed of L layers (l ¼ 0;…;L), where l ¼ 0 represents pendencies in the follow-up of the patients. Fig. 5 shows the component the input layer, Nl neurons per layer and their corresponding in­ of an LSTM cell. The memory mechanism is controlled by the gates, terconnections. Samples are first presented through the input layer, made up of ANNs with a specific activation function at the output layer. whose neurons connect with one or more hidden layers, and they link to Each cell is responsible for filtering relevant information. The core idea the output layer where the result of the model is obtained. The output of is to combine the information from the gates in the cell gate, ct . The the i-th neuron at the l-th layer (the so-called activation) is the linear forget gate, ft , indicates which information from the combination of the combination of the outputs at the previous layer, taking into account the previous state, ht , and the input, xt , is discarded. Then new information weights learned, and modified by a specific non-linear function fð �Þ, is added to ct through the combination of two gates, the input gate, it , usually the sigmoid, the hyperbolic tangent or the Rectified Linear Unit which decides the information to updated and the candidate values, c’t . (ReLU). In other words, ! Finally, ct is updated and the output is a filtered version of the cell gate XNl 1 modulated. alj ¼ f wli;j 1 ali 1 (2) In the case of mortality in ESRD, LSTMs are fed with concatenated i¼0 vectors that contain the evolution of n months and the prediction is where wli;j 1 is the weight that connects the i-th activation at layer l 1 to carried out to p months. For instance, Fig. 6 illustrates the follow-up of a patient during m months, from the first encounter with the hospital’s al0 the input of the j-th neuron at layer l. Note that ¼ 1 in all layer except system to the death event. The follow-up is structured into samples, the output layer to take into account the bias term. taking information of n months of evolution. Then, using the timestamp At the output layer, activations are usually normalised (e.g., softmax) of the samples and date of death, d in the figure, the moths to the death so that the resulting values take a value between 0 and 1. They can be event of the structured samples are computed. Thus, the binary target of interpreted as a probability estimation; for example, yk represents the the generated data depends on the prediction range using the rule, probability that the input example belongs to the k-th class. � The network is trained in order to achieve a minimum of a given cost f ðtd Þ ¼ 0; if ​ td > p (5) function that measures the error between predictions and the corre­ 1; otherwise sponding true values. A common function is cross-entropy, where p is the prediction range, td is the time to the death event. ’00 and �n o� � � n o�� X ’10 indicates the class sample, alive and deceased respectively. N X K C wli;j ¼ tk;n log yk xn ; wli;j (3) n¼1 k¼1 3. Results A known problem using ANN is overfitting, which happens when the network does not learn a model from the underlying data but memorizes The samples for this analysis were extracted from 261 patients who the individual examples. A common way to reduce this effect is applying had the three types of variables, as described in Section 2.1. Table 2 L2 weight regularization [27], a quadratic penalty function is added to shows the description of the population. Because the duration of HD the weight, i.e. (3) is modified to treatment varies across the cohort, each patient generates a different number of monthly samples. In total, 8394 monthly samples were extracted. In this work mortality is predicted to 1, 2, 3, 6 and 12 months. Thus, five datasets with the same data but different targets, after applying the transformation in Eq. (5), are generated. Fig. 7 shows the mortality trajectories for patients in the training and test sets. For models development, patients were split into training and test sets (80-20%). The training set was divided into 5-folds for cross- validation (CV), see Fig. 8. With this approach, it is possible to find the hyperparameters for RFE-RF and LSTMs. Such parameters are the ones that can be calibrated manually. For RFE-RF the number of trees, depth of the decision trees and splitting criteria. For LSTMs the number of cells, neurons per cell, LR, among others. Then, with the hyper­ parameters fixed, the parameters of the network (weights of the LSTMs) are computed and five different models, M1; M2; …; M5 in Fig. 8, from the 5-folds are obtained as a result. The evaluation is done in the initial Fig. 4. Structure of a feed-forward artificial neural network with two hid­ test set. den layers. 4 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 Fig. 5. LSTM cell, σ is the sigmoid activation function. Fig. 6. Sample structuring from follow-up of a patient with m months in HD treatment to the death event, d. To estimate the performance of the classifiers in the test set, AUROC The features not considered by RF-RFE were: cardiopathy, enterop­ was used. It measures the area under the graphic representation of the athy, haemorrhage, hepatopathy, hypertension, neoplasia, pneumo­ general accuracy, showing the variation of the sensitivity and specificity pathy, fracture, infection and the type of vascular access. of a binary classifier when the decision threshold varies. The metric takes values between 0 and 1, with 1 corresponding to the perfect classifier. 3.2. Predictive model-LSTM In the second experiment, we consider a more powerful model based 3.1. Feature selection-RF on LSTMs. After parameter optimization, we found that the best configuration was using an LSTM with two cells and with 750 and 500 Our first experiment studies the importance of the available features units, respectively. We used ADAM optimizer with LR ¼ 0.001 and L2 automatically, using an RF approach, together with an RFE approach. regularization with λ ¼ 0:001. Then, the LSTM approach is evaluated in The optimal hyperparameters for RF were 103 trees, maximum depth of several groups of variables chosen by the experience of the expert staff, 3, using the Gini index for splitting the nodes and calculate the impor­ the group of variables found by RFE-RF and all the available ones. tance of features. For RFE approach, the 5-folds were used to find the Table 3 shows the importance level of both laboratory and HD variables best features more robustly. With the approach, it was found that 42 as determined by the experience of the hospital expert staff. features offered the best performance for all the predictors. AUROCs of Fig. 10 shows ROC curves comparing the groups of variables using a 0.737, 0.714, 0.712, 0.668 and 0.615 were the baseline performance 4 months to feed the LSTM and predicting mortality to 1 month. As predicting mortality to 1, 2, 3, 6 and 12 months, respectively. Fig. 9 illustration, Group_12 considers laboratory variables with an impor­ illustrate the AUROC as a function of the number of considered features tance label of 1 and HD session variables with an importance level of 2 for the prediction of mortality to one month. and Group_RFE refers to the ones found by RFE-RF. Note that diagnosis 5 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 Table 2 Cohort description. Variable Samples/Patient contains the information about the number of samples that the patients generate. For diagnoses, Number of patients column represents the total of patients with a specific diagnose. VA refers to vascular access, SBP and DBP to systolic and diastolic blood pressure. Feature Units Number of patients MV (%) Mean Std Min Max Age Years – 0.0 71.41 10.69 24.00 91.00 Sex (Women) – 104 0.0 – – – – Sex (Men) – 157 0.0 – – – – Samples/Patient – – – 26 22 1 116 Calcium mg/dL 261 10.8 9.10 0.69 6.30 13.00 Creatinine mg/dL 261 25.0 6.80 2.30 0.30 15.50 Ferritin ng/mL 261 28.1 472.1 368.32 8.10 6590.00 Glucose mg/dL 261 25.7 123.30 67.85 13.00 1370.00 Haemoglobin g/L 261 29.2 6.21 1.26 4.10 13.60 HDL cholesterol mg/dL 261 18.3 43.73 14.60 4.40 115.60 Hematocrit L/L 261 1.0 349.990 0.04 0.17 0.49 Hemoglobin g/L 261 1.1 111.69 14.21 46.00 161.00 Iron μg/dL 261 38.1 59.44 26.70 10.00 340.00 KTV mL/min 261 17.3 1.43 0.28 0.42 02.09 LDL cholesterol mg/dL 261 18.9 83.40 33.20 8.00 240.00 Leucocytes x109 =L 261 1.0 7.63 4.97 1.25 11.3 Lymphocytes x109 =L 261 5.8 1.50 0.76 0.22 12.74 Monocyts x109 =L 261 5.8 0.56 0.22 0.03 2.69 Neutrophil x109 =L 261 5.8 5.24 2.29 0.22 7.25 Phosphorus mg/dL 261 26.1 4.33 1.39 0.20 11.80 Platelets x109 =L 261 1.1 223.37 83.17 14.40 1067.00 Potassium mEq/L 261 35.0 4.95 0.80 0.30 8.90 PTH pg/mL 261 28.3 228.05 189.17 6.00 3264.00 Reticulocytes x109 =L 261 28.4 5.37 2.69 0.23 35.23 Sodium mEq/L 261 31.5 138.66 3.59 121.00 198.00 Total cholesterol mg/dL 261 38.1 149.98 39.41 45.00 432.00 Total proteins g/L 261 27.6 66.02 6.84 28.5 96.00 Triglycerides mg/dL 261 18.1 140.49 107.92 20.00 2673.00 Urea mg/dL 261 43.2 102.40 51.12 20.20 317.20 Accumulative weight Kg 261 21.7 1.95 0.77 3.05 4.44 Average flow mL/min 261 16.2 290.28 34.48 200.00 414.55 Blood vol dia mL/min 261 12.0 65.08 10.52 40.00 98.43 DBP post HD mmHg 261 10.4 65.34 10.22 40.00 105.61 DBP pre HD mmHg 261 10.5 64.44 10.62 40.00 106.08 Dry weight Kg 261 0.9 66.78 15.24 31.29 149.63 HD session time Hours 261 0.0 3.73 0.35 3.50 7.30 HR post HD BPM 261 10.6 75.59 11.49 41.00 122.00 HR pre HD BPM 261 6.6 73.19 10.57 42.00 121.17 Hypotension Cases/month 261 0.0 2 4 0 24 SBP post HD mmHg 261 13.3 138.11 22.7 57.00 205.00 SBP pre HD mmHg 261 6.3 137.31 22.19 56.07 218.60 Temp post HD ∘C 261 16.9 35.58 0.33 33.00 38.20 Temp pre HD ∘C 261 11.6 35.52 0.34 33.85 38.00 Arteriopathy – 177 0.0 – – – – Cardiopathy – 241 0.0 – – – – Diabetes – 204 0.0 – – – – Enteropathy – 94 0.0 – – – – Fracture – 9 0.0 – – – – Hemorrhague – 6 0.0 – – – – Hepatopathy – 18 0.0 – – – – Hypertension – 223 0.0 – – – – Infection – 102 0.0 – – – – Neoplasia – 79 0.0 – – – – Pneumopathy – 115 0.0 – – – – VA (AVF) – 168 0.0 – – – VA (Catheter) – 164 0.0 – – – – VA (Graft) – 6 0.0 – – – – Mortality Months – 0.0 25.52 21.89 1.00 116.00 variables (11 in total) are included in all cases. w.r.t. the other works in the literature; ii) including knowledge expert Finally, in Fig. 11 we test the performance of our algorithm by not always leads to better models and iii) solutions can guide the considering: i) all variables; ii) HD data only and iii) diagnosis and research in a specific field by revealing possible causal relations not laboratory data only. explored before, possibly far from human intuition. Table 4, includes a performance comparison in terms of AUROC with the existing solutions 4. Discussion in the literature. Although one-year mortality does not exceed that stated in the literature, the improvement in short-term mortality grows This work explored how deep learning can help in the study of ESRD. to 4% if we reduce the prediction time to 3 months. When we compared After the experiments conducted, in this case, focused on the evaluation our approach to these other works, we realised that we combined three of mortality, the lessons learned are: i) we can improve model accuracy sources of data, i.e., diagnosis, laboratory and HD data, being not the 6 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 Fig. 7. Kaplan Meier mortality model for training and test set. p ¼ 0.17. Fig. 8. Cross validation with 5-folds. Test Data is only used when the hyperparameters are found. Fig. 9. Recursive feature selection, with 5-folds cross-validation, using RF as learning model. case in the available works. Most of them use either laboratory and according to their importance level, being 1 the highest level and 3 the diagnosis data or HD session data. Fig. 11 shows that the inclusion of all lowest (see Table 3). Accordingly, in Fig. 10 we tested our model with variables improves the AUROC at least by 11% in AUROC. several combinations of the subsets of variables. We could expect to In order to study how considering knowledge expert influences the achieve the best possible performance by using level 1 laboratory data performance of algorithms, expert staff labelled HD and laboratory data together with level 1 HD variables, i.e., Group_11 (recall that diagnostic 7 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 Table 3 5. Conclusion Ranking of features chosen by the experience of the expert. Their importance are marked from 1 to 3, being 3 the less important features. VA refers to vascular In this work, we demonstrate the potential of the massive use of access. variables together with machine learning techniques for the improve­ Laboratory Importance HD variables Importance ment of mortality predictive models in ESRD. We designed a baseline Calcium 1 HD time 3 predictor and feature selector using an RFE-RF approach. Then we Creatinine 3 HR post HD 1 improve it using LSTM strategy that exploits temporal dependencies in Ferritin 2 HR pre HD 1 the data. We conclude that thanks to considering diagnostic variables Glucose 3 Hypotension 1 along with laboratory and HD session data, we could improve perfor­ Haemoglobin 1 SBP post HD 1 HDL cholesterol 2 SBP pre HD 1 mance in the prediction of mortality in the ESRD patient by at least 4% Iron 3 Temp post HD 3 w.r.t. existing works for short-term mortality. Furthermore, results show KTV 1 Temp pre HD 3 that expert knowledge has to contribute to the analysis, but we shall not LDL cholesterol 2 VA (AVF) 1 limit our algorithms to it. In our experiment, the best performance Leucocytes 2 VA (Catheter) 1 achieved by the groups chosen does not exceed the RF-RFE. Therefore, Lymphocytes 2 VA (Graft) 1 Monocytes 2 machine learning methods like the ones explored here can provide Neutrophil 2 feedback to the experts, improve our knowledge and can lead to a Phosphorus 1 change in the paradigm in the analysis of predictive factors in mortality Platelets 3 in ESRD. Potassium 2 PTH 1 Reticulocytes 2 Ethical statement Sodium 1 Total cholesterol 2 The authors of the manuscript certify that the manuscript entitle Total proteins 3 Triglycerides 2 “Mortality prediction enhancement in end-stage renal disease: A machine Urea 1 learning approach” has not been and will not be submitted to or published in any other publication before its appearance in the "Informatics in Medicine Unlocked" journal. data is included in all cases). However, the performance achieved is The data used for this study were taken from the information system similar to Group_33, and the inclusion of all variables boosts the AUROC of the Parc Taulí University Hospital, from the Haemodialysis unit at the value in 9%. In other words, expert knowledge is undoubtedly relevant, Nephrology Department. This project passed through the ethics com­ but it is also important to explore beyond it. mittee (Code 2018/508). Finally, machine learning approaches can also help the research of Next the contributions of the authors are listed. physicians by revealing causal relations possibly not explored before. In Fig. 9, we tested how an automatic feature selection tool such as RFE Author contribution may help. In this case, 42 features gave us the best classification per­ formance using an RF approach, and when we consider this selection as Conception and design of the study: E Macias, A Morell, J Serrano, JL input to our LSTM solution, the performance is close to best one, which Vicario and J Ibeas. is obtained with all the features. Therefore, physicians can explore the Adquisition of data: J Ibeas. subset of variables selected, reduce or increase it as far as performance is Analysis and/or interpretation of data: E Macias, A Morell, J Serrano, sustained (see Fig. 9) and investigate the importance and effects of the JL Vicario and J Ibeas. chosen features. However, it should be noted that RF-RFE did not take Drafting the manuscript: E Macias, A Morell, J Serrano, JL Vicario into account most of the diagnoses or the type of vascular access. This and J Ibeas. evidence could suggest that those variables that were not considered Revising the manuscript critically for important intellectual content: important could lead to new medical research. A Morell, J Serrano, JL Vicario and J Ibeas. Fig. 10. Performance comparison between best features found by RFE-RF, the combinations of features chosen by expert staff and using all the available information. The groups are generated based on the experience of the medical staff, from Group_11 which can be inferred from the combination of the most important analytics with the most important HD variables to Group_33, the least significant ones. 8 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 Fig. 11. Comparison of performance using laboratory tests and diagnoses, all the available variables and just variables taken during haemodialysis sessions. and also by the Catalan Government under Project 2017 SGR 1670. Table 4 Comparison methods in literature with proposed one. References Reference Mortality Study Prediction AUROC (CI prediction population algorithm 95%) [1] Martínez-Castelao Alberto, G� orriz Jos� e L, Bover Jordi, Segura-de la Morena Juli�an, Morales-Olivas Francisco, Cebollada Jesús, Su� arez Carmen, Escalada Javier, Mauri et at 1 year 5738 Logistic 0.670 Mu~noz-Torres Manuel, Esmatjes Enric, De Pablos-Velasco Pedro, F� acila Lorenzo, [15] regression (0.668–0.675) Maz�on Pilar, Gamarra Javier, Hern� and-Moreno Julio, Gr� acia Silvia, Akbilgic 1 month 27615 Random forest 0.736 Monta~ n�es Rosario, Llisterri-Caro Jos�e L, De Santiago Ana, S�anchez-Celaya Marta, et al. [24] (0.715–0.757) Tranche Salvador. Consensus document for the detection and management of 3 months 0.764 chronic kidney disease. Endocrinol Nutr 2014. [2] Luyckx Valerie A, Tonelli Marcello, Stanifer John W. The global burden of kidney (0.754–0.774) disease and the sustainable development goals. Bull World Health Organ 2018. 6 months 0.760 [3] Rhee Connie M, Kovesdy Csaba P. Spotlight on CKD deaths—increasing mortality (0.747–0.775) worldwide. Nat Rev Nephrol 2015. 1 year 0.757 [4] Goff David C, Lloyd-Jones Donald M, Bennett Glen, Coady Sean, D’Agostino Ralph (0.746–0.769) B, Gibbons Raymond, Greenland Philip, Lackland Daniel T, Levy Daniel, Wagner 3 years 5447 Cox 0.730 O’Donnell Christopher J, Robinson Jennifer G, Sanford Schwartz J, Shero Susan T, Smith Sidney C, Paul Sorlie, Stone Neil J, Wilson Peter WF. ACC/AHA guideline on et al. [30] proportional (0.700–0.760) the assessment of cardiovascular risk: a report of the American college of 7–12 hazards 0.698 cardiology/American heart association task force on practice guidelines. J Am Coll months (0.679–0.717) Cardiol 2013;2014. 13–18 0.717 [5] Hippisley-Cox Julia, Coupland Carol, Vinogradova Yana, Robson John, months (0.696–0.737) Rubin Minhas, Aziz Sheikh, Brindle Peter. Predicting cardiovascular risk in 19–24 0.670 England and Wales: prospective derivation and validation of QRISK2. BMJ 2008. months (0.646–0.694) [6] D’Agostino Ralph B, Vasan Ramachandran S, Pencina Michael J, Wolf Philip A, Cobain Mark, Massaro Joseph M, Kannel William B. General cardiovascular risk Isshiki et al. 3–36 62 Hazard ratio 0.696 profile for use in primary care: the Framingham heart study. Circulation 2008. [31] months [7] Ridker Paul M, Buring Julie E, Rifai Nader, Cook Nancy R. Development and Proposed 1 month 261 LSTM 0.873 validation of improved algorithms for the assessment of global cardiovascular risk approach (0.871–0.876) in women: the Reynolds Risk Score. J Am Med Assoc 2007. [8] Dreiseitl Stephan, Ohno-Machado Lucila. Logistic regression and artificial neural 2 months 0.813 network classification models: a methodology review. J Biomed Inf 2002. (0.811–0.815) [9] Macias E, Morell A, Serrano J, Vicario JL. Knowledge extraction based on wavelets 3 months 0.798 and dnn for classification of physiological signals: arousals case. In: Computing in (0.796–0.800) Cardiology conference (CinC). vol. 45. IEEE; 2018. p. 1–4. 2018. 6 months 0.752 [10] Alanazi Hamdan O, Abdullah Abdul Hanan, Qureshi Kashif Naseer. A critical (0.751–0.753) review for developing accurate and dynamic predictive models using machine 1 year 0.720 learning methods in medicine and health care. J Med Syst 2017. (0.703–0.737) [11] Miller Rebecca, Tumin Dmitry, Cooper Jennifer, Hayes Don, Joseph D, Tobias. Prediction of mortality following pediatric heart transplant using machine learning algorithms. Pediatr Transplant 2019. Approval of the manuscript to be published: A Morell, J Serrano, JL [12] Ross Elsie Gyang, Shah Nigam H, Dalman Ronald L, Nead Kevin T, Cooke John P, Leeper Nicholas J. The use of machine learning for the identification of peripheral Vicario and J Ibeas. artery disease and future mortality risk. J Vasc Surg 2016;64(5):1515–22. [13] Çelik Güner, Baykan Omer€ K, Kara Yakup, Tireli Hülya. Predicting 10-day mortality in patients with strokes using neural networks and multivariate statistical Declaration of competing interest methods. J Stroke Cerebrovasc Dis 2014;23(6):1506–12. [14] Wang Guanjin, Lam Kin Man, Deng Zhaohong, Choi Kup Sze. Prediction of The authors confirm that there are no known conflicts of interest mortality after radical cystectomy for bladder cancer by machine learning techniques. Comput Biol Med 2015. associated with this publication. [15] Mauri Joan M, Cl�eries Montse, Vela Emili, Catalan Renal Registry. Design and validation of a model to predict early mortality in haemodialysis patients. Nephrol Acknowledgements Dial Transplant 2008. [16] Srinivasan Beddhu, Bruns Frank J, Saul Melissa, Patricia Seddon, Mark L, Zeidel. A simple comorbidity scale predicts clinical outcomes and costs in dialysis patients. This work is supported by the Spanish Government under Project Am J Med 2000. TEC2017-84321-C4-4-R co-funded with European Union ERDF funds 9 E. Macias et al. Informatics in Medicine Unlocked 19 (2020) 100351 [17] Liu Jiannong, Huang Zhi, Gilbertson David T, Foley Robert N, Collins Allan J. An models to stratify cardiovascular risk in incident hemodialysis patients. Expert Syst improved comorbidity index for outcome analyses among dialysis patients. Kidney Appl 2013. Int 2010. [24] Oguz Akbilgic, Obi Yoshitsugu, Potukuchi Praveen K, Karabayir Ibrahim, [18] Couchoud C�ecile, Labeeuw Michel, Olivier Moranne, Vincent Allot, Nguyen Danh V, Soohoo Melissa, Streja Elani, Molnar Miklos Z, Rhee Connie M, Esnault Vincent, Frimat Luc, B� en�edicte Stengel. A clinical score to predict 6-month Kalantar-Zadeh Kamyar, Kovesdy Csaba P. Machine learning to identify dialysis prognosis in elderly patients starting dialysis for end-stage renal disease. Nephrol patients at high death risk. Kidney Int. Rep. 2019;4(9):1219–29. Dial Transplant 2009. [25] Breiman Leo, Leo. Random forests. Mach Learn 2001;45(1):5–32. [19] Otero-L� opez M, Martínez-Oca~ na Juan C, Betancourt-Castellanos Loreley, [26] Darst Burcu F, Malecki Kristen C, Engelman Corinne D. Using recursive feature Rodríguez-Salazar Eleonora, García-García Manuel. Two prognostic scores for early elimination in random forest to account for correlated variables in high mortality and their clinical applicability in elderly patients on haemodialysis: poor dimensional data. BMC Genet 2018;19(1):65. predictive success in individual patients. Nefrologia 2012;32(2):213–20. [27] Anders Krogh, Hertz John A. A simple weight decay can improve generalization. [20] Salekin Asif, Stankovic John. Detection of chronic kidney disease and selecting In: Advances in neural information processing systems; 1992. p. 950–7. important predictive attributes. In: IEEE international conference on healthcare [28] Rumelhart David E, Hinton Geoffrey E, Williams Ronald J. Learning informatics (ICHI), vol. 2016. IEEE; 2016. p. 262–70. representations by back-propagating errors. Nature 1986;323(6088):533–6. [21] Almansour Njoud Abdullah, Syed Hajra Fahim, Khayat Nuha Radwan, [29] Kingma Diederik P, Jimmy Ba. Adam: a method for stochastic optimization. 2014. Altheeb Rawan Kanaan, Juri Renad Emad, Alhiyafi Jamal, Alrashed Saleh, arXiv preprint arXiv, 1412.6980. Sunday O, Olatunji. Neural network and support vector machine for the prediction [30] Wagner Martin, Ansell David, Kent David M, Griffith John L, Naimark David, of chronic kidney disease: a comparative study. Comput Biol Med 2019;109: Wanner Christoph, Tangri Navdeep. Predicting mortality in incident dialysis 101–11. patients: an analysis of the United Kingdom renal registry. Am J Kidney Dis 2011. [22] Doi Toshiki, Yamamoto Suguru, Morinaga Takatoshi, Ken-ei Sada, Kurita Noriaki, [31] Isshiki Keiji, Nishio Toshiki, Isono Motohide, Makiishi Tetsuya, Shikano Tsutomu, Onishi Yoshihiro. Risk score to predict 1-year mortality after haemodialysis Tomita Koubin, Nishio Toshiji, Kanasaki Masami, Maegawa Hiroshi, Uzu Takashi. initiation in patients with stage 5 chronic kidney disease under predialysis Glycated albumin predicts the risk of mortality in type 2 diabetic patients on nephrology care. PloS One 2015;10(6). hemodialysis: evaluation of a target level for improving survival. Ther Apher Dial [23] Ion Titapiccolo Jasmine, Ferrario Manuela, Cerutti Sergio, Barbieri Carlo, 2014;18(5):434–42. Mari Flavio, Gatti Emanuele, Gabriella Signorini Maria. Artificial intelligence 10