Since January 2020 Elsevier has created a COVID-19 resource centre with free information in English and Mandarin on the novel coronavirus COVID- 19. The COVID-19 resource centre is hosted on Elsevier Connect, the company's public news and information website. Elsevier hereby grants permission to make all its COVID-19-related research that is available on the COVID-19 resource centre - including this research content - immediately available in PubMed Central and other publicly funded repositories, such as the WHO COVID database with rights for unrestricted research re-use and analyses in any form or by any means with acknowledgement of the original source. These permissions are granted for free by Elsevier for as long as the COVID-19 resource centre remains active. Journal of Clinical Epidemiology - (2023) - REVIEW ARTICLE Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation Gabriela Cardenas-Fuentesa,b,c,d,*, Magda Bosch de Baseaa,b,c, Ines Coboa,b,c, Isaac Subiranae,f, Mario Ceresag, Ernest Famadab, Elena Gimeno-Santosa,b,c,h, Laura Delgado-Ortiza,b,c,  Rosa Faneri,j,k, Marıa Molina-Molinak,l,m, Alvar Agustıh,i,j,k, Xavier Mu~ nozk,n,o, Oriol Sibilah,i,k, b,k,p,q a,b,c Joaquim Gea , Judith Garcia-Aymerich a ISGlobal, Barcelona, Spain b Universitat Pompeu Fabra (UPF), Barcelona, Spain c CIBER Epidemiologıa y Salud Publica (CIBERESP), Barcelona, Spain d School of Health Sciences, Blanquerna-Universitat Ramon Llull, Barcelona, Spain e Instituto Hospital del Mar de Investigaciones Medicas (IMIM), Barcelona, Spain f CIBER Enfermedades Cardiovasculares (CIBERCV), ISCIII, Spain g BCNMedTech, Universitat Pompeu Fabra (UPF), Barcelona, Spain h Respiratory Institute, Hospital Clinic, Barcelona, Spain i Instituto de Investigaciones Biomedicas August Pi i Sunyer (IDIBAPS), Barcelona, Spain j Universitat de Barcelona, Barcelona, Spain k CIBER Enfermedades Respiratorias (CIBERES), ISCIII, Spain l Servicio de Neumologıa, Hospital Universitario de Bellvitge, L’Hospitalet de Llobregat, Spain m Instituto de Investigacion Biomedica de Bellvitge (IDIBELL), L’Hospitalet de Llobregat, Spain n Servicio de Neumologıa, Hospital Universitario Vall d’Hebron, Barcelona, Spain o Departamento de Biologıa celular, fisiologıa e inmunologıa, Universitat Autonoma de Barcelona, Barcelona, Spain p Servicio de Neumologıa, Hospital del Mar-IMIM, Barcelona, Spain q Fundacio Barcelona Respiratory Network (BRN), Barcelona, Spain Accepted 25 April 2023; Published online xxxx Abstract Objectives: To identify prognostic models which estimate the risk of critical COVID-19 in hospitalized patients and to assess their validation properties. Competing interests: X.M. has received fees as a speaker, scientific Availability of data and materials: The data that support the findings of advisor, and as a participant of clinical studies of (in alphabetical order) As- this study are available from HM and Institut Catala de la Salut, but restric- traZeneca, Boehringer Ingelheim, Chiesi, Faes, GlaxoSmithKline, Menarini, tions apply to the availability of these data and so are not publicly available. Mundipharma, Novartis, Sanofi, and Teva. R.F. has received grants from Data are however available from the authors upon reasonable request and GlaxoSmithKline LLC, Menarini, AstraZeneca, ISC-III, and the Spanish with permission of HM and Institut Catala de la Salut. Health Service; consulting fees from GlaxoSmithKline LLC; and other pay- Funding: Supported by AstraZeneca Farmaceutica (ESR-20-20,762). IS- ments (lectures, presentations, and speaker) from Chiesi. J.G. has received Global acknowledges support from the Spanish Ministry of Science and grants from SEPAR and from FIS, ISCIIIeEU Program Horizon; fees from Innovation through the ’Centro de Excelencia Severo Ochoa 2019e2023’ GSK and Menarini; and other payments (lectures, presentations, and Program (CEX2018-000806-S) and support from the Generalitat de Catalu- speaker) from Menarini. M.M.M. has received grants supported by Roche nya through the CERCA Program. No involvement of funding sources in and Boehringer Ing.; other payments (lectures, presentations, and speaker) study design; in the collection, analysis, and interpretation of data; in the from Esteve-Teijin, Chiesi; and support for attending meetings/travel from writing of the report; or in the decision to submit the article for publication. Boehringer Ing. The rest of the authors declared that they have no conflict Author contributions: G.C.F., M.B.B., I.C., and J.G.A. performed the of interest. systematic review of research articles; G.C.F., I.S., J.G.A., and I.C. analyzed Ethics approval and consent to participate: The study protocol was and interpreted the data; G.C.F., M.B.B., M.C., E.F., E.G.S., L.D.O., R.F., approved by the Ethics Committee of Parc de Salut Mar (2020/9206/I). M.M.M., A.A., X.M., O.S., J.G., and J.G.A. were major contributors in The study used only retrospective and routinely collected data, and there- writing the manuscript. All authors read and approved the final manuscript. fore, informed consent was waived. * Corresponding author. Barcelona Institute for Global Health, Carrer Consent for publication: Not applicable. Doctor Aiguader, 88, 08003 Barcelona, Spain. Tel.: þ34-93-214-7300. E-mail address: gabriela.cardenas@isglobal.org (G. Cardenas-Fuentes). https://doi.org/10.1016/j.jclinepi.2023.04.011 0895-4356/Ó 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/ licenses/by-nc-nd/4.0/). 2 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Study Design and Setting: We conducted a systematic review in Medline (up to January 2021) of studies developing or updating a model that estimated the risk of critical COVID-19, defined as death, admission to intensive care unit, and/or use of mechanical ventilation during admission. Models were validated in two datasets with different backgrounds (HM [private Spanish hospital network], n 5 1,753, and ICS [public Catalan health system], n 5 1,104), by assessing discrimination (area under the curve [AUC]) and calibration (plots). Results: We validated 18 prognostic models. Discrimination was good in nine of them (AUCs  80%) and higher in those predicting mortality (AUCs 65%e87%) than those predicting intensive care unit admission or a composite outcome (AUCs 53%e78%). Calibration was poor in all models providing outcome’s probabilities and good in four models providing a point-based score. These four models used mortality as outcome and included age, oxygen saturation, and C-reactive protein among their predictors. Conclusion: The validity of models predicting critical COVID-19 by using only routinely collected predictors is variable. Four models showed good discrimination and calibration when externally validated and are recommended for their use. Ó 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Keywords: COVID-19; Critical disease; Intensive care unit; Prognostic models; External validation; Epidemiology 1. Introduction 2. Methods Since the beginning of the pandemic, the substantial 2.1. Systematic identification of prognostic models number of critical COVID-19 cases has overwhelmed the healthcare systems around the world [1]. Despite that only A systematic review was registered in PROSPERO re- pository (CRD42021235106) and performed as per a previ- about 5% of COVID-19 patients require critical care [2], ously written protocol (Supplementary Material, Item 1) the deaths from COVID-19 have already surpassed six based on the Preferred Reporting Items for Systematic Re- million [3]. Therefore, a prompt and standardized identifi- views and Meta-Analyses (PRISMA) guidelines [6,7] and cation of patients at risk of developing critical COVID-19 the guide to perform systematic reviews of prognostic fac- is crucial. tor studies described by Riley et al. [8]. Many studies have developed models aiming to predict critical COVID-19 in patients hospitalized with the disease [4]. These models have claimed a potential to decrease the 2.1.1. Search strategy chance of fatal outcomes, assist clinicians in performing A systematic search was performed using the Medline risk stratification, and optimize the use of health resources database (Supplementary Material, Item 2) covering the [4]. In fact, prior to the pandemic, prognostic models have period from December 2019 (first detection of COVID-19 been extensively used in the medical field. These rely on disease) to January 2021, including models developed prior the use of patient-level information (e.g., demographic, to COVID-19 vaccination. The search was supplemented clinical, and laboratory results) to estimate the probability through hand search of the reference lists of relevant of developing a future clinical event [5]. To apply a prog- studies. Retrieved articles were saved in the Mendeley plat- nostic model in clinical settings, it is imperative to previ- form and screened for inclusion criteria using the CADIMA ously validate its performance in a group of patients software [9]. Two reviewers (G.C.F. and M.B.B.) indepen- different from the one used for model development. This dently screened titles and abstracts of retrieved references process, known as external validation, helps determining (consistency with Kappa of 0.8) and screened the full text whether the model properties estimated in the development of those considered eligible. Where consensus was not dataset are real or are due to overfitting. A proper validation achieved, a third reviewer (J.G.A.) was consulted. implies assessing the capacity of a model of, first, ordering individuals as per their risk of presenting the event (i.e., 2.1.2. Selection criteria discrimination), and second, providing a risk estimate with Eligible articles had to (1) be peer-reviewed and pub- similar magnitude to the real risk of presenting the event lished in English language; (2) have a longitudinal (pro- (i.e., calibration). Unfortunately, very few of the more than spective or retrospective) design, from hospital admission a hundred published prognostic models aiming to predict to event or discharge, with any length of follow-up; (3) critical COVID-19 have been adequately externally vali- aim at developing or updating a multivariable prognostic dated [4], and therefore, there is no evidence on how they model; (4) include general population (not disease sub- would perform on samples coming from different hospitals groups) hospitalized due to COVID-19, diagnosed through and different locations. real-time reverse transcription polymerase chain reaction, Hence, we aimed to (1) identify prognostic models de- genetic sequencing, or radiological imaging compatible signed to estimate the risk of critical COVID-19 in patients with COVID-19; (5) include only information routinely hospitalized because of the disease and (2) validate their collected at hospital admission as model’s predictors (i.e., performance by assessing their discrimination capacity demographics, clinical signs/symptoms, laboratory results, and calibration in two external datasets. medical history, and clinical scores); (6) use critical G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 3 in two datasets, representing two longitudinal retrospective What is new? cohorts with different backgrounds: the private and the pub- lic health system. These datasets included anonymous in- Key findings formation extracted from electronic health records of  We identified nine models that discriminate the patients hospitalized due to COVID-19, followed from hos- risk of critical COVID-19 well in two external pital admission to the development of critical COVID-19 datasets. (as defined in each validated model) or discharge. The Hos- pital Universitario de Madrid (HM) dataset included pa-  Four of them are additionally well calibrated, pro- tients hospitalized in the dependencies of the HM vide a point-based score, and include predictors network, formed by 16 private hospitals in Spain, three of easy to obtain. which are in Catalonia (further details in Supplementary Material, Item 3). We included patients with COVID-19 What this adds to what was known? admitted between February 5 and April 20, 2020, covering  The several critical COVID-19 prognostic models the first COVID-19 wave in Spain, when COVID-19 vacci- that lacked external validation have been identified nation was still not available [11]. The Institut Catala de la and validated in the context of this study. Salut (ICS) dataset included patients hospitalized in the de- pendencies of the Catalan Public Health System, which What is the implication and what should change serves almost six million people living in Catalonia, a re- now? gion in the Northeast of Spain. We included patients with  The use of any of these four models will assist cli- COVID-19 admitted between March 1 and August 14, nicians in performing risk stratification, thus 2020, covering the first and second COVID-19 wave in decreasing the chances of fatal outcomes and opti- Spain who were, therefore, also unvaccinated [11]. The mizing the use of health resources. study protocol was approved by the Ethics Committee of Parc de Salut Mar (2020/9206/I). Diagnosis of COVID-19 was defined as a registration of ‘confirmed COVID-19’ in the HM dataset and as a positive COVID-19 as the model outcome, defined by the use of real-time reverse transcription polymerase chain reaction invasive or noninvasive mechanical ventilation, a registered test in the ICS dataset. We excluded individuals younger admission to the intensive care unit (ICU), in-hospital or than 18 years (n 5 5); those who died, were admitted to out-hospital death, or by a combination of them; (7) pro- the ICU, or were discharged on the day of admission (n vide at least one indicator of model performance; and (8) 5 205); those who remained hospitalized at the time of data perform an internal or external validation (only in models dumping (n 5 224); anddbecause most prognostic models labeled as having a high potential of over/under fitting) included signs and symptomsdthose with no available in- (Supplementary Material, Item 1). Studies including only formation on these variables at hospital admission (n 5 a population subgroup (e.g., patients with cancer); 21,951). The final study sample included 2,857 participants excluding population subgroups (e.g., pregnant partici- (1,753 in HM dataset and 1,104 in ICS dataset) pants); not providing regression coefficients, point-based (Supplementary Material, Item 4). scores, or equivalent predictors’ weights; or not providing predictors’ units or summary statistics were discarded. 2.2.2. Variables and procedures We obtained data collected as part of routinely clinical 2.1.3. Data extraction assistance, including information on: (1) demographics, un- Key features of selected articles (study sample, model’s derlying comorbidities, clinical signs, symptoms, and labo- predictors, outcome variables, and modeling techniques) ratory results at hospital admission; (2) procedures were extracted by two independent reviewers (G.C.F. and performed during the hospitalization (e.g., administration M.B.B.). Data were organized in standardized forms de- of invasive mechanical ventilation); and (3) administrative signed as per the Prediction Study Risk of Bias Assessment registries (e.g., dates of admission and discharge of the hos- Tool Guidelines (PROBAST) [8]. Discrepancies during the pital and ICU). In variables with repeated measurements data extraction were solved by consulting a fourth reviewer during hospitalization (e.g., laboratory results, signs, or (I.C.). symptoms), we only included the first one. 2.2. Validation of prognostic models 2.2.3. Definition of critical COVID-19 2.2.1. Study design and population Critical COVID-19 has been defined by the World Results were reported as per the validation section of the Health Organization as presenting sepsis, septic shock, or Transparent Reporting of a Multivariable Prediction Model other conditions that would normally require the provision for Individual Prognosis or Diagnosis statement [10]. of life-sustaining therapies such as mechanical ventilation Models identified in the systematic review were validated or vasopressor therapy [12]. In the present validation study, 4 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - we defined critical COVID-19 as in each original study, this observations (Supplementary Material, Items 6-7) and were is: models using mortality as the outcomedregardless of considered completely at random (e.g., a hospital did not being inside or outside the hospitaldwere validated against collect a variable at all) or at random (e.g., patients in-hospital mortality, respecting the original time frames (if admitted to the ICU had fewer missing data), we performed stated); models using a registered ICU admission were vali- multiple imputations using chained equations, and per- dated against a registered ICU admission; and models using formed the main analysis using the imputed datasets. a composite outcome were validated against a standard For each participant and prognostic model, we calcu- composite outcome, created as having an ICU admission lated (1) the predicted probability of developing critical and/or in-hospital death. COVID-19 (in models providing regression coefficients or similar) or (2) a final score (in models providing a point- based scoring system). Discrimination was assessed by 2.3. Statistical analysis calculating the area under the curve (AUC) in models A detailed description of the statistical analysis is pro- providing outcome’s probabilities based on a logistic vided in the Supplementary Material, Item 5. In brief, we regression or in those providing a point-based scoring sys- validated the prognostic models separately in both datasets. tem, or by calculating the Harrell’s C index in models Since missing data in each dataset accounted for ! 20% of providing probabilities based on a Cox regression. Records identified through Additional records identified database searching through hand search (n= 1,240) (n= 17) Records after duplicates removal (n= 1,257) Records screened at Records excluded title/abstract level (n= 929) Full-text articles Reasons for exclusion assessed for eligibility (n= 314, sequential exclusion): (n= 328) - Duplicate records (n= 2) - Not general population (n= 108) - Outcome not critical COVID-19 (n= 58) - Predictors not at baseline/not routinely collected (n= 38) - No prognostic model (n= 42) - No model performance/ validation (n= 58) - Coefficients/scores not provided (n= 3) - Predictor units not provided (n= 1) - Not in English (n= 2) - Full text not accessible (n= 2) Included in the review (14 studies, 19 models) Included in the validation (13 studies, 18 models) Fig. 1. PRISMA flow diagram. G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 5 Calibration was assessed by inspecting calibration plots in only internally validated, whereas 10 had been externally models providing outcome’s probabilities or by plotting a (and, in some cases, also internally) validated. final score against the observed outcome’s probability in models providing a point-based scoring system. 3.2. Characteristics of the external validation samples As secondary analysis, we assessed discrimination and calibration in all models against 28-day in-hospital mortal- We included a total of 2,857 COVID-19 patients from ity. As sensitivity analyses, we repeated the analysis (1) in the HM and ICS datasets (Table 2). On average, patients the complete-case datasets, to account for the possibility of were aged more than 60 years (mean [standard deviation] inducing bias during imputation and (2) after excluding 66.9 [15.8] years in HM; 63.6 [16.2] years in ICS), there participants transferred to other hospitals at the moment were slightly more men (HM: 61%; ICS: 58%), and they of discharge (only in HM dataset, as this situation did not were mostly nonsmokers (HM: 97%; ICS: 86%). A occur in the ICS dataset), to account for potential outcome considerable proportion of patients had a recorded diag- misclassification. All analyses were conducted using R nosis of high blood pressure (HM: 35%; ICS: 49%), dia- 4.1.2 (R Foundation for Statistical Computing, 2021). betes (HM: 16%; ICS: 21%), and/or obesity (HM 5%; ICS: 42%). At hospital admission, more than half of the participants reported dyspnea as the main (in HM) or as one of the symptoms (in ICS); and 7% and 70% reported 3. Results cough as the main (in HM) or as one of the symptoms (in ICS), respectively. By the end of follow-up, in the HM and 3.1. Identified prognostic models of critical COVID-19 ICS datasets, respectively, roughly 7% and 28% of the pa- The search retrieved 1,257 articles (fig 1) (1,240 from tients received some form of mechanical ventilation; 4% Medline and 17 from the hand search). Among them, 328 and 18% were admitted to the ICU; and a 16% and 13% were eligible for full-text screening: 327 due to fulfilling died. In the crude analysis (Supplementary Material, Item the title and abstract screening criteria and oneddespite 9), an increased risk of 28-day mortality was observed including participants with confirmed and nonconfirmed among participants who were older, male, had an underly- COVID-19 diagnosisdbecause of its large sample size, in- ing comorbidity, high respiratory rate, low oxygen satura- ternational coverage, and statistical rigor [13]. During full- tion, and/or altered laboratory results at hospital text screening, and after removing duplicate records admission. (n 5 2), we excluded those not meeting participant’s, out- come’s, or predictor’s selection criteria (n 5 204); those not 3.3. External validation of the prognostic models deriving a prognostic model (n 5 42); those not assessing model’s performance or validity (n 5 58, 67% of the arti- Due to the unavailability of some predictors in the vali- cles deriving a model); and those not providing model co- dation datasets (Supplementary Material, Item 10), we vali- efficients or predictor units (n 5 3, 3% of the articles dated 18 (of 19) models: four using both datasets, 13 using deriving a model). Fourteen studies were selected after one of the datasets (seven in HM and six in ICS), and one full-text screening, from which we extracted 19 prognostic using only the complete case ICS dataset. models [13e26]. Models differed in their study popula- Nine of 17 models showed a good discrimination capac- tions, selection of predictors and outcomes, and modeling ity (AUC  80%) (Table 3, Supplementary Material, Item techniques (Table 1 and Supplementary Material, Item 8). 11). All these models included age among their predictors, They were developed in seven countries (China, England, six included oxygen saturation, and seven included at least Italy, Scotland, Spain, the United States, and Wales), had one laboratory variable obtained at hospital admission. In samples sizes ranging between 176 and 35,463 individuals both datasets, discrimination was higher in models predict- (median 893 individuals), and were mostly developed in ing mortality (AUCs ranging from 65% to 87%) than in retrospective cohorts. In the identified models, the most those predicting ICU admission or a composite outcome prevalent critical COVID-19 definition was death (74%), (AUCs ranging from 53% to 78%). followed by ICU admission (21%) and composite outcome All models estimating outcome’s probabilities exhibited (n 5 5%). Forty four different predictors were included in an overall poor calibration (Supplementary Material, Item the models, the most frequent being age (89%), oxygen 12), with some models showing a general underestimation saturation (63%), C-reactive protein (37%), lactate dehy- of the risk of critical COVID-19 and others showing an drogenize (32%), and lymphocyte count (32%). Of the 19 overestimation at small probabilities and underestimation models, 17 provided the components of the regression for- at large probabilities. Of the eight models using a point- mula allowing the estimation of the outcome probability, based scoring system, four showed a good calibration from which eight also provided a scoring system. The re- (Figs. 2 and 3). These models (Torres Macho [20], Li maining two studies provided only a scoring system, which [26], 4C Mortality [13], and Altschul [18]) showed good also permitted validation. Of the 19 models, nine had been discrimination, used mortality as the outcome variable, 6 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Table 1. Characteristics of the 19 identified prognostic models Model Country Events (sample size)a Age, mean, or median (years)a Sex, % of malea ZhaoeICU USA 195 (454) 60 60% Zhaoemortality 82 (454) Weng China 21 (176) 47 42% Torres-Macho Spain 325 (1,968) 67 56% Meienonlaboratory China 103 (1,088) 58 50% Meielaboratory Wangenonlaboratory China 19 (296) 47 47% Wangelaboratory Berzuinienonlaboratory England 110 (392) 71 65% Berzuinielaboratory Gude Spain 51 (229) 68 61% Zhou China 68 (763) 51 49% LieICU USA 271 (1,108) ICU: 59, non-ICU admitted: 62 57% Liemortality 142 (1,022) Death: 76, alive: 59 57% Altschul USA 621 (2,355) 65 47% Chen China 50 (1,590) Death: 69, alive: 48 Death: 60%, alive: 57% Baronio Italy 157 (191) 65 78% 4C Mortality England, Scotland, Wales 11,426 (35,463) 73 58% Magro Italy 495 (1,810) 67 71% AST, aspartate aminotransferase; BUN, blood urea nitrogen; CKD, chronic kidney disease; CRP, C-reactive protein; COPD, chronic obstructive pulmonary disease; COVID-19, coronavirus 2019; GFR, glomerular filtration rate; HR, hazard ratio; ICU, intensive care unit; IMV, invasive mechanical ventilation; LASSO, least absolute shrinkage and selection operator; LDH, lactate dehydrogenize; MAP, mean arterial pressure; NLR, neutrophil/lymphocyte ratio; OR, odds ratio; PCT, procalcitonin; SpO2, oxygen saturation; USA, United States of America; WBC, white blood cell count. a Described for the total sample. In articles performing an external validation, these parameters are described for the derivation cohort. b Day of diagnosis considered day 1. c Severe COVID-19 defined as ICU admission, IMV, and/or all-cause mortality. d Calculated as (prothrombin time/normal range of prothrombin time) international sensitivity index. e We validated the model using 28-day survival. and included age, oxygen saturation, and C-reactive protein [18]dexhibited very similar validation properties in the among their predictors (Table 4). sensitivity analyses. 4. Discussion 3.4. Secondary and sensitivity analysis This systematic review identified 19 models aimed at The validation of all models against 28-day mortality as predicting critical COVID-19 in hospitalized individuals a unique outcome showed a 1% to 2% increase in the AUCs with the disease and validated 18 of them separately in da- of most models predicting mortality and a more than 10% tasets obtained from public and private hospitals. Main re- increase in the AUCs of those predicting ICU admission sults showed that (1) half of the validated models presented (Supplementary Material, Item 13). The validation of the good discrimination, all using mortality as the outcome; (2) models in the complete case datasets (including one addi- of those, only four models had a good calibration, and these tional model not validated in the multiple imputed datasets were all based on a scoring system (not providing out- [21]), resulted in slightly higher AUCs in the HM dataset come’s probabilities); and (3) the best performing models (0% to 10% more) and in slightly lower or higher AUCs included parameters easy to obtain in clinical settings. in the ICS dataset (3% to 4%) (Supplementary Material, Item 14). After excluding participants who were transferred 4.1. Performance of prognostic models during the to another hospital, there were no substantial changes in the external validation discrimination ability of the models (Supplementary Material, Item 15). The four models highlighted aboved In the present study, half of the models estimating Torres Macho [20], Li [26], 4C Mortality [13], and Altschul COVID-19 mortality showed good discrimination. These G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 7 Predictors Outcome Statistical method/validated parameters Smoking status, SpO2, LDH, lymphocyte count, PCT ICU admission Logistic/risk score points Age, COPD, heart failure, heart rate, SpO2, LDH, PCT In-hospital mortality Age, CRP, D-dimer, NLR All-cause mortality Logistic/intercept, beta coefficients Age, smoking status, SpO2, creatinine, CRP, hemoglobin, In-hospital mortality Logistic/risk score points lymphocyte count, platelet count, sodium Age, respiratory failure, renal failure, coronary heart disease, 60-day all-cause mortalityb Logistic/intercept, beta coefficients heart failure, interaction age-renal failure Age, respiratory failure, D-dimer, LDH, lymphocyte count, platelet count, WBC, interaction WBC platelets, interaction D-dimer-LDH Age, coronary heart disease, hypertension In-hospital mortality Logistic/OR Age, SpO2, AST, CRP, D-dimer, GFR, lymphocyte count, neutrophil count Age, smoking status, respiratory rate, SpO2 21-day mortalityb Logistic/intercept, beta coefficients Age, smoking status, respiratory rate, SpO2, NLR Age, diabetes, SpO2, lymphocyte count, pH Severe COVID-19c Logistic/intercept, beta coefficients Age, smoking status, CKD, respiratory rate, systolic blood ICU admission Logistic/risk score points pressure, fever SpO2, CRP, ferritin, LDH, PCT ICU admission Deep learning neural network/risk score points Age, SpO2, CRP, LDH, PCT, troponin In-hospital mortality Age, MAP, SpO2, BUN, CRP, international normalized ratiod In-hospital mortality Logistic/risk score points e Age, cerebrovascular disease, coronary heart disease, AST, 14-day, 21-day, and 28-day survival Cox/HR PCT, dyspnoea Age, BMI, cardiovascular disease, SpO2, lymphocyte count ICU admission Logistic/OR Age, sex, number of comorbidities, respiratory rate, SpO2, In-hospital mortality LASSO/risk score points Glasgow Coma Scale, BUN, CRP Age, sex, chronic liver disease, coronary heart disease, 40-day in-hospital mortality Competing risks/none diabetes, LDH, duration of symptoms before hospital admission results are in contrast with most findings in the field of pre- through a calibration in the large) agrees with the idea that dictive modeling, where the discrimination of a model sub- prognostic models should recalibrate the intercept based on stantially decreases when being validated in an external the frequency of the outcome in the setting where they are sample [27]. Indeed, our results confirm the importance to be applied [33]. Overall, our study went a step further by of conducting a systematic review and of including strict providing a list of models that are applicable to the clinical selection criteria to identify the most adequate models, settings and by identifying four models exhibiting the best prior to their validation. Previous studies have validated discrimination and calibration properties when validated several critical COVID-19 prognostic models [28e30], externally in two diverse and large datasets. but compared to our study, they have found worse discrim- ination results. This could be explained by the fact that they 4.2. Components of the well-performing prognostic have selected the models using more permissive criteria and models have used smaller sample sizes for the validation (from 169 to 1,612). The lower discrimination capacity of models us- We identified age, oxygen saturation, and laboratory in- ing ICU admission or composite outcomes to define critical dicators (C-reactive protein, lactate dehydrogenize, and COVID-19, compared to those using mortality, is in line lymphocyte count) as variables consistently included in with other reports [29,31] and might be explained by the the models with best prediction properties for critical context of the pandemic itself, where the criteria to be COVID-19, in agreement with previous literature identi- admitted to an ICU may have been modified due to an fying them as individual predictors of critical COVID-19 increased demand or to the unavailability of hospital re- [32,34e37]. It is important to highlight that these indicators sources [32]. Finally, our finding of poor calibration in all are easy and rapid to obtain, something that enhances the models providing an outcome’s probability (not going clinical applicability of the models. Some of the studies 8 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Table 2. Characteristics of the study population according to study dataset HM dataset (N [ 1,753) ICS dataset (N [ 1,104) Characteristics of the Study Population N N Age, years e mean (SD) 1,753 66.9 (15.8) 1,104 63.6 (16.2) Male gender, n (%) 1,753 1,064 (60.7%) 1,104 639 (57.9%) Current smoker, n (%) 1,502 47 (3.13%) 902 124 (13.7%) 2 BMI, kg/m emean (SD) n.a. n.a. 373 29.5 (5.52) Presence of comorbidity, n (%) Diabetes Mellitus 1,502 239 (15.9%) 1,104 228 (20.7%) COPD 1,502 72 (4.79%) 1,104 89 (8.06%) Asthma 1,502 69 (4.59%) 1,104 80 (7.25%) Obesity 1,502 77 (5.13%) 1,104 459 (41.6%) Hypertension 1,502 525 (35.0%) 1,104 535 (48.5%) Cardiovascular disease (any) n.a. n.a. 1,104 117 (10.6%) Cerebrovascular disease 1,502 33 (2.20%) 1,104 45 (4.08%) Heart failure 1,502 24 (1.60%) 1,104 55 (4.98%) Coronary heart disease 1,502 82 (5.46%) 1,104 87 (7.88%) Vital signs at admission e mean (SD) or median [IQR] Systolic blood pressure, mmHg 1,462 125 (19.4) 1,066 130 (20.2) Diastolic blood pressure, mmHg 1,462 71.5 (12.4) 1,062 74.5 (13.9) Temperature,  C 1,746 36.7 (0.86) 918 37.2 (1.00) Heart rate, bpm 1,721 81.3 (14.4) 1,066 91.8 (17.8) Respiratory rate, breath/min n.a. n.a. 981 20.0 [18.0e25.0] SpO2, % 1,489 94.0 [91.0-96.0] 1,030 96.0 [94.0e98.0] Clinical scores at admission e mean (SD) or median [IQR] Glasgow coma scale, units n.a. n.a. 899 15.0 [15.0e15.0] Laboratory parameters at admission e mean (SD) or median [IQR] D-dimer, mg/L 1,325 2.17 (7.10) 491 1.69 (16.9) Platelet count, 109/L 1,605 220 (94.6) 642 207 (85.6) International normalized ratioa 1,305 1.20 [1.12e1.31] 597 1.11 [1.04e1.21] Arterial O2, mmHg 599 59.0 [51.0e67.0] 175 75.0 [60.0e100] Arterial CO2, mmHg 599 34.3 (6.57) 175 66.7 (45.2) WBC, 109/L 1,605 6.47 [4.93e8.59] 575 6.72 [5.12e8.89] Lymphocyte count, 109/L 1,605 1.02 [0.73e1.41] 554 1.00 [0.70e1.40] Neutrophil count, 109/L 1,605 4.68 [3.26e6.73] 548 4.90 [3.48e7.00] NLR 1,605 6.85 (11.1) 547 7.01 (7.75) Creatinine, mg/L 1,570 0.90 [0.72e1.10] 643 0.85 [0.68e1.07] BUN, mmol/L 1,538 5.61 [4.20e7.91] 509 6.30 [4.33e9.00] GFR, mL/min 5 25.1 [19.7e27.9] 575 86.0 [61.0e90.0] AST, U/L 1,498 34.0 [24.0-51.5] 621 41.0 [30.0e60.0] CRP, mg/L 1,573 73.2 [31.1e138] 445 12.4 [5.86e24.5] PCT, ng/mL 142 0.14 [0.09e0.27] 42 0.21 [0.14e0.32] Troponin, ng/L 272 11.2 [7.08e29.5] 180 10.0 [8.43e20.0] LDH, U/L 1,527 541 [427e698] 363 352 [288e453] Sodium, mmol/L 1,552 137 (4.55) 644 136 (3.56) Ferritin, ng/mLe median [IQR] 373 936 [426e1,553] 322 582 [315e1,089] pH, unitse mean (SD) 788 7.45 (0.06) 284 7.42 (0.08) Symptoms at admission, n (%) Main symptom (Continued ) G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 9 Table 2. Continued HM dataset (N [ 1,753) ICS dataset (N [ 1,104) Characteristics of the Study Population N N Dyspnoea 1,714 940 (54.8%) n.a. n.a. Fever 1,714 231 (13.5%) n.a. n.a. Cough 1,714 123 (7.18%) n.a. n.a. Presence of symptoms Dyspnoea n.a. n.a. 876 440 (50.2%) Fever n.a. n.a. 1,023 794 (77.6%) Cough n.a. n.a. 876 642 (73.3%) Complications, n (%) NIMV 1,502 66 (4.39%) 802 103 (12.8%) IMV 1,502 33 (2.20%) 802 121 (15.1%) ICU admission 1,753 77 (4.39%) 1,104 205 (18.6%) In-hospital mortality 1,753 278 (15.9%) 1,104 145 (13.1%) ICU admission and/or death 1,753 319 (18.2%) 1,104 325 (29.4%) Time from hospital admission to e mean (SD) Discharge/death, days 1,753 8.39 (5.62) 1,104 10.7 (12.9) ICU admission, days 77 5.60 (5.99) 203 4.57 (6.78) AST, aspartate aminotransferase; BUN, blood urea nitrogen; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; COV- ID-19, coronavirus disease 2019; CRP, C-reactive protein; GFR, glomerular filtration rate; ICU, intensive care unit; IMV, invasive mechanical venti- lation; LDH, lactate dehydrogenase; NIVM, noninvasive mechanical ventilation; NLR, neutrophil/lymphocyte ratio; SpO2, oxygen saturation; WBC, white blood cell count. a Calculated as (prothrombin time/normal range of prothrombin time) international sensitivity index. Table 3. Discriminatory ability of 16 critical COVID-19 prognostic models in two external validation samples HM dataset ICS dataset Critical COVID-19 Prognostic Models AUC (95% CI) AUC (95% CI) Outcome: In-hospital mortality Zhao e mortality 0.78 (0.75 to 0.81) - Weng 0.84 (0.81 to 0.86) 0.78 (0.74 to 0.82) Torres-Macho 0.87 (0.85 to 0.89) 0.82 (0.79 to 0.86) Mei e laboratory 0.80 (0.77 to 0.84) - Wang e nonlaboratory 0.83 (0.80 to 0.85) 0.81 (0.78 to 0.84) Wang e laboratory - 0.65 (0.59 to 0.70) Berzuini e nonlaboratory - 0.85 (0.82 to 0.88) Berzuini e laboratory - 0.86 (0.83 to 0.89) Li e mortality 0.81 (0.77 to 0.85) - Altschul 0.83 (0.81 to 0.86) 0.82 (0.78 to 0.87) Chen 0.73 (1.48 to 2.93)a - 4C Mortality - 0.84 (0.81 to 0.87) Outcome: ICU admission Zhao e ICU 0.64 (0.58 to 0.71) - Zhou - 0.53 (0.49 to 0.59) Li e ICU 0.59 (0.52 to 0.66) - Baronio - 0.62 (0.55 to 0.69) Outcome: composite Gude 0.78 (0.75 to 0.81) - ICU, intensive care unit; AUC, area under the curve. a Harrell’s C index. 10 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Fig. 2. Calibration of six critical COVID-19 prognostic models providing a point-based scoring system in the HM dataset. The dashed black line indicates the LOESS fit. The five imputed datasets are differentiated in colors. developing prognostic models emphasized that laboratory predictors including age, smoking status, and results from results and vital signs (i.e., dynamic indicators of acute dis- the physical examination and laboratory analysis are avail- ease) exhibited better predictive abilities than chronic co- able (see full list of predictors in Table 4), and can be morbidities or demographic data (i.e., fixed individual computed online (www.pandemyc-score.com). The Li characteristics) [35,38]. This is of great importance at the [26] and the Altschul [18] scores include the smallest set moment of hospital admission, where diagnosis of comor- of predictors (age, one to two clinical signs, and three to bidities may not be available or accurate in electronic four laboratory indicators), making them especially useful health records, and where interviewing compromised at emergency settings; however, they do not provide an on- incoming patients may not be possible. In this context, line application and, in the case of Li score, it was validated two of our four best performing models, based only on (and proved suitable) only in our private hospitals dataset. age and parameters obtained from physical examination Finally, the application of the 4C Mortality score [13] needs and laboratory analysis, might be particularly useful a compressive set of predictors and is available as an online [18,26]. calculator (www.isaric4c.net/risk); however, in this study, its suitability was tested (and proved) only in our public hospitals dataset. 4.3. Clinical applicability of the identified models It could be argued whether the mentioned models, devel- Based on our results, half of the tested models are suit- oped and validated in prevaccination conditions, can be able for use in the general hospitalized population and for a applicable to current and future patients. Research has prompt application at hospital admission, a key moment in consistently reported that the severity of COVID-19 (and decision-making. The selection of one of the four models therefore, the proportion of cases with critical disease) with best prediction properties should be based mainly on was higher in the first than in subsequent waves, in alpha the target population and on the availability of predictors and beta than in following variants, and in unvaccinated (Table 4). In brief, the Torres-Macho score [20] is suitable than vaccinated patients [39e43]. However, the predictors for a broad population (as we found it valid in both private of critical COVID-19 have been reported to be the same and public hospitals), is especially useful when a regardless of wave, variant, and vaccination status. Indeed, comprehensivedbut still routinely collecteddset of a recent study validating one prevaccination model found G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 11 Fig. 3. Calibration of four critical COVID-19 prognostic models providing a point-based scoring system in the ICS dataset. The dashed black line indicates the LOESS fit. The five imputed datasets are differentiated in colors. good performance in a sample of patients infected with toward validating, recalibrating, and making available ex- delta and omicron variants [44], and a recently developed isting models rather than toward developing new ones. model including vaccination as a predictor did not include any interaction term between vaccine and remaining predic- 4.5. Strengths and limitations tors [45]. All in all, data suggest that models developed A strength of this study is the systematic identification prior to vaccination maintain their discrimination properties of prognostic models and the focus on those applicable to (i.e., the ability to order individuals as per their risk of crit- the general population (not to specific subgroups) and on ical COVID-19 when used for risk stratification) and would those using only routinely collected data. For the external only need to be recalibrated when used to estimate the risk of critical COVID-19, as anyways recommended when us- validation, we used two different datasets (one from a pub- lic health system and one from a private hospitals network, ing prognostic models as mentioned previously [33]. thus representing different patients’ background) with rela- tively high sample sizes, which broadens the applicability 4.4. Applicability of study results to prognostic model of our results. Additionally, the HM dataset was generated research during the first COVID-19 epidemiological period (i.e., first In our systematic review, more than two-thirds of the wave), which was dominated by the original SARS-CoV-2 studies that had developed a prognostic model did not strain [46], whereas the ICS dataset included data not only report their results in adequate detail to allow individual from the first wave but also from the second, which in prediction nor subsequent validation despite this is strongly Spain was dominated by the B.1.177. variant [47]. This recommended in Transparent Reporting of a Multivariable adds confidence regardless of the SARS-CoV-2 variant in Prediction Model for Individual Prognosis or Diagnosis circulation, in the predictive ability of models that per- guidelines [10]. Incomplete reporting has been considered formed well in the two validation datasets, such as the a form of research waste and, in the case of prognostic Torres-Macho score. model research, precludes any further validation and use The main limitation of this study is that some predictors of research results. Our findings sustain the idea that, in (e.g., signs/symptoms at hospital admission) were missing, the COVID-19 field, further efforts should be directed which prevented us from validating some of the identified 12 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Table 4. Components of the four prognostic models showing the best discrimination and calibration properties: Torres-Macho, Li, 4C Mortality, and Altschul mortality scores Torres-Macho [20] (range 70e452) Li [26] (range 0e6) Categories Pts Categories Pts Age, years Age, years !46 46 71 0 46 and 60 4 O71 1 O60 and 78 29 SpO2, % O78 50 88 0 Missing 28 !88 1 Smoking status CRP, mg/L No 25 17 0 Yes 43 O17 1 Missing 22 PCT, ng/mL SpO2, % 1.1 0 !88 59 O1.1 1 88 and 92 36 Troponin, ng/mL O92 and 96 21 0.03 0 O96 2 O0.03 1 Missing 30 LDH, U/L Platelet count, 109/L 487 0 !124 48 O487 1 124 27 BUN, mmol/L Missing 18 19.6 0 Hemoglobin, g/dL O19.6 and 39.2 1 !11.2 51 O39.2 3 11.2 and 12.2 36 O12.2 25 Missing 20 Lymphocyte count, cells/mL !500 55 500 and 799 37 O799 and 1,000 28 O1,000 22 Missing 19 Creatinine, mg/dL !0.77 14 0.77 and 1.08 21 O1.08 and 1.71 37 O1.71 48 Missing 29 CRP, mg/L !9.7 8 9.7 and 65.9 20 O65.9 and 188.5 35 O188.5 45 Missing 18 Sodium, mmol/L !144 28 (Continued ) G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 13 Table 4. Continued Torres-Macho [20] (range 70e452) Li [26] (range 0e6) Categories Pts Categories Pts 144 53 Missing 17 4C Mortality [13] (range 0e21) Altschul [18] (range 0e10) Age, years Age, years !50 0 !60 0 50 and !60 2 60 and !70 1 60 and !70 4 70 and !80 2 70 and !80 6 80 3 80 7 O80 0 Sex Mean arterial pressure, mmHg Female 0 O70 and 80 1 Male 1 O60 and 70 2 Number of comorbidities 60 3 0 0 SpO2, % 1 1 94 0 2 2 !94 1 Respiratory rate, breath/min International normalized ratioa !20 0 1.2 0 20 and !30 1 O1.2 1 30 2 BUN, mg/dl SpO2, % 30 0 92 0 O30 1 !92 2 CRP, mg/L Glasgow Coma Scale 10 0 15 0 O10 1 !15 2 BUN, mmol/L 19.6 0 O19.6 and 39.2 1 O39.2 3 CRP, mg/L !50 0 50 and !100 1 100 2 BUN, blood urea nitrogen; CRP, C-reactive protein; LDH, lactate dehydrogenase; SpO2, oxygen saturation; Pts, points. a Calculated as (prothrombin time/normal range of prothrombin time) international sensitivity index. models in both datasets. However, it is likely that these pre- predictors is variable, and only half of them showed good dictors are not available in most clinical settings, something discrimination when externally validated. Four models that hinders the clinical applicability of models including (Torres Macho [20], Li [26], 4C Mortality [13], and Alt- them and support the idea that future studies should focus on- schul [18]) exhibiting good discrimination and calibration ly on models including easily available predictors. Another included routinely collected patient’s parameters and there- limitation was the retrospective nature of our validation co- fore are proposed for their use in clinical settings. horts, which increases the chances of bias (e.g., due to a non- standardized data collection and data entry procedure). 4.6. Conclusion Acknowledgments The validity of models predicting critical COVID-19 in This study was carried out using anonymized data pro- the general population by using only routinely collected vided by the Catalan Agency for Quality and Health 14 G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - Assessment, within the framework of the PADRIS [15] Zheng Y, Zhang Y, Chi H, Chen S, Peng M, Luo L, et al. The hemo- Program. cyte counts as a potential biomarker for predicting disease progres- sion in COVID-19: a retrospective study. Clin Chem Lab Med 2020;58:1106e15. [16] Baronio M, Freni-Sterrantino A, Pinelli M, Natalini G, Tonini G, Supplementary data Marri M, et al. Italian SARS-CoV-2 patients in intensive care: to- wards an identikit for subjects at risk? Eur Rev Med Pharmacol Sci Supplementary data to this article can be found online at 2020;24:9698e704. https://doi.org/10.1016/j.jclinepi.2023.04.011. [17] Magro B, Zuccaro V, Novelli L, Zileri L, Celsa C, Raimondi F, et al. Predicting in-hospital mortality from coronavirus disease 2019: a simple validated app for clinical use. PLoS One 2021;16: References e0245281. [18] Altschul DJ, Unda SR, Benton J, de la Garza Ramos R, Cezayirli P, [1] Islam N, Shkolnikov VM, Acosta RJ, Klimkin I, Kawachi I, Mehler M, et al. A novel severity score to predict inpatient mortality Irizarry RA, et al. Excess deaths associated with covid-19 pandemic in COVID-19 patients. Sci Rep 2020;10:16726. in 2020: age and sex disaggregated time series analysis in 29 high in- [19] Weng Z, Chen Q, Li S, Li H, Zhang Q, Lu S, et al. ANDC: an early come countries. BMJ 2021;373:n1137. warning score to predict mortality risk for patients with coronavirus [2] Wu Z, McGoogan JM. Characteristics of and important lessons from disease 2019. J Transl Med 2020;18:328. the coronavirus disease 2019 (COVID-19) outbreak in China: sum- [20] Torres-Macho J, Ryan P, Valencia J, Perez-Butrague~no M, Jimenez E, mary of a report of 72314 cases from the Chinese center for disease Fontan-Vela M, et al. The PANDEMYC score. An easily applicable control and prevention. JAMA 2020;323:1239e42. and interpretable model for predicting mortality associated with [3] European centre for disease prevention and control. COVID-19 situ- COVID-19. J Clin Med 2020;9:3066. ation update worldwide, as of week 14, updated 14 April 2022. 2020. [21] Mei J, Hu W, Chen Q, Li C, Chen Z, Fan Y, et al. Development and Available at: https://www.ecdc.europa.eu/en/geographical-distribution- external validation of a COVID-19 mortality risk prediction algo- 2019-ncov-cases. rithm: a multicentre retrospective cohort study. BMJ Open 2020;10: [4] Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, e044028. et al. Prediction models for diagnosis and prognosis of covid-19: sys- [22] Wang K, Zuo P, Liu Y, Zhang M, Zhao X, Xie S, et al. Clinical and tematic review and critical appraisal. BMJ 2020;369:m1328. laboratory predictors of in-hospital mortality in patients with [5] Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, COVID-19: a cohort study in Wuhan, China. Clin Infect Dis 2020; Collins GS, et al. PROBAST: a tool to assess risk of bias and appli- 71:2079e88. cability of prediction model studies: explanation and elaboration. [23] Berzuini C, Hannan C, King A, Vail A, O’Leary C, Brough D, et al. Ann Intern Med 2019;170:W1e33. Value of dynamic clinical and biomarker data for mortality risk pre- [6] Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, diction in COVID-19: a multicentre retrospective cohort study. BMJ et al. Preferred reporting items for systematic review and meta- Open 2020;10:e041983. analysis protocols (prisma-p) 2015: elaboration and explanation. [24] Gude F, Riveiro V, Rodrıguez-Nu~nez N, Ricoy J, Lado-Baleato O,  BMJ 2015;349:1e25. Lourido T, et al. Development and validation of a clinical score to es- [7] Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, timate progression to severe or critical state in COVID-19 pneumonia Petticrew M, et al. Preferred reporting items for systematic review hospitalized patients. Sci Rep 2020;10:19794. and meta-analysis protocols (prisma-p) 2015 statement. Syst Rev [25] Zhou Y, He Y, Yang H, Yu H, Wang T, Chen Z, et al. Exploiting an 2015;4:1. early warning Nomogram for predicting the risk of ICU admission in [8] Riley RD, Moons KGM, Snell KIE, Ensor J, Hooft L, Altman DG, patients with COVID-19: a multi-center study in China. Scand J et al. A guide to systematic review and meta-analysis of prognostic Trauma Resusc Emerg Med 2020;28:106. factor studies. BMJ 2019;364:k4597. [26] Li X, Ge P, Zhu J, Li H, Graham J, Singer A, et al. Deep learning [9] Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, prediction of likelihood of ICU admission and mortality in Schiemann J, et al. Correction to: online tools supporting the conduct COVID-19 patients using clinical variables. PeerJ 2020;8:e10337. and reporting of systematic reviews and systematic maps: a case [27] Siontis GCM, Tzoulaki I, Castaldi PJ, Ioannidis JPA. External valida- study on CADIMA and review of existing tools. Environ Evid tion of new risk prediction models is infrequent and reveals worse 2018;7(8). https://doi.org/10.1186/s13750-018-0124-4. prognostic discrimination. J Clin Epidemiol 2015;68:25e34. [10] Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent re- [28] Gupta RK, Marks M, Samuels THA, Luintel A, Rampling T, porting of a multivariable prediction model for individual prognosis Chowdhury H, et al. Systematic evaluation and external validation or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:1e9. of 22 prognostic models among hospitalised adults with COVID- [11] Centro Nacional de Epidemiologıa, Instituto de Salud Carlos III, 19: an observational cohort study. Eur Respir J 2020;56:2003498. Ministerio de Ciencia e Innovacion. Informe no112. Situacion de [29] Wickstrøm KE, Vitelli V, Carr E, Holten AR, Bendayan R, COVID-19 en Espa~na a 5 de enero de 2022. Equipo COVID-19. Reiner AH, et al. Regional performance variation in external valida- 2022. Available at: https://www.isciii.es/QueHacemos/Servicios/ tion of four prediction models for severity of COVID-19 at hospital VigilanciaSaludPublicaRENAVE/EnfermedadesTransmisibles/ admission: an observational multi-centre cohort study. PLoS One Documents/INFORMES/Informes. 2021;16:e0255748. [12] World Health Organization (WHO). Living guidance for clinical [30] Yamada G, Hayakawa K, Asai Y, Matsunaga N. External validation management of COVID-19. World Heal Organ; 2021. and update of prediction models for unfavorable outcomes in hospi- [13] Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, et al. Risk talized patients with COVID-19 in Japan. J Infect Chemother 2021; stratification of patients admitted to hospital with covid-19 using the 27:1043e50. ISARIC WHO Clinical Characterisation Protocol: development and [31] Xu Y, Becker N, Ferres J, Lee A. Machine learning-based derivation and validation of the 4C Mortality Score. BMJ 2020;370:m3339. external validation of a tool to predict death and development of organ [14] Zhao Z, Chen A, Hou W, Graham JM, Li H, Richman PS, et al. Pre- failure in hospitalized patients with COVID-19. Sci Rep 2022;12:16913. diction model and risk scores of ICU admission and mortality in [32] Kim HJ, Hwang H, Hong H, Yim JJ, Lee J. A systematic review and COVID-19. PLoS One 2020;15:e0236618. meta-analysis of regional risk factors for critical outcomes of G. Cardenas-Fuentes et al. / Journal of Clinical Epidemiology - (2023) - 15 COVID-19 during early phase of the pandemic. Sci Rep 2021;11: variants on inpatient clinical outcome. Clin Infect Dis 2023;76(9): 9784. 1539e49. [33] Janssen KJM, Moons KGM, Kalkman CJ, Grobbee DE, Vergouwe Y. [41] Bienkowski C, Kowalska JD, Paciorek M, Wasilewski P, Uliczny P, Updating methods improved the performance of a clinical prediction Garbacz-qago_zna E, et al. The clinical course and outcomes of pa- model in new patients. J Clin Epidemiol 2008;61:76e86. tients hospitalized due to COVID-19 during three pandemic waves [34] dong GY, Ding M, Dong X, jin ZJ, Kursat Azkur A, Azkur D, et al. in Poland: a single center observational study. J Clin Med 2022;11: Risk factors for severe and critically ill COVID-19 patients: a review. 1e13. Allergy 2021;76:428e55. [42] Butt AA. Severity of illness in persons infected with the SARS-CoV- [35] Incerti D, Rizzo S, Li X, Lindsay L, Yau V, Keebler D, et al. Prog- 2 delta variant vs beta variant in Qatar. JAMA Intern Med 2022;182: nostic model to identify and quantify risk factors for mortality among 1e9. hospitalised patients with COVID-19 in the USA. BMJ Open 2021; [43] Vito A De, Colpani A, Saderi L, Puci M, Zauli B, Fiore V, et al. 11:1e11. Impact of early SARS-CoV-2 antiviral therapy on disease progres- [36] Meintrup D, Borgmann S, Seidl K, Stecher M, Jakob CEM, sion. Viruses 2023;15:71. Pilgram L, et al. Specific risk factors for fatal outcome in critically [44] Wei S, Ong X, Tham SM, Koh LP, Dugan C, Khoo BY, et al. External ill covid-19 patients: results from a European multicenter study. J validation of the PRIORITY model in predicting COVID-19 critical Clin Med 2021;10:1e14. illness in vaccinated and unvaccinated patients. Clin Microbiol Infect [37] Shcherbak SG, Anisenkova AY, Mosenko SV, Glotov OS, 2022;28:884.e1e3. Chernov AN, Apalko SV, et al. Basic predictive risk factors for [45] Zhang Y, Han J, Sun F, Guo Y, Guo Y, Zhu H, et al. A practical cytokine storms in COVID-19 patients. Front Immunol 2021;12: scoring model to predict the occurrence of critical illness in hospital- 1e8. ized patients with SARS-CoV-2 omicron infection. Front Microbiol [38] Adderley NJ, Taverner T, Price MJ, Sainsbury C, Greenwood D, 2022;13:1e10. Chandan JS, et al. Development and external validation of prognostic [46] Lopez M, Chiner-Oms A,  Garcıa de Viedma D, Ruiz-Rodriguez P, models for COVID-19 to support risk stratification in secondary care. Bracho M, Cancino-Mu~noz I, et al. The first wave of the COVID- BMJ Open 2022;12:e049506. 19 epidemic in Spain was associated with early introductions and fast [39] Buttenschøn HN, Lynggaard V, Sandbøl SG, Glassou ENHA. Com- spread of a dominating genetic variant. Nat Genet 2021;53:1405e14. parison of the clinical presentation across two waves of COVID-19: [47] Gonzalez-Candelas F, Shaw MA, Phan T, Kulkarni-Kale U, a retrospective cohort study. BMC Infect Dis 2022;22:493. Paraskevis D, Luciani F, et al. One year into the pandemic: short- [40] Robinson ML, Morris CP, Betz JF, Zhang Y, Wang N, Thiemann DR, term evolution of SARS-CoV-2 and emergence of new lineages. et al. Impact of severe acute respiratory syndrome coronavirus 2 Infect Genet Evol 2021;92:104869.