Received: 16 November 2018 Revised: 4 April 2019 Accepted: 23 April 2019 DOI: 10.1002/pds.4803 ORIGINAL REPORT Validity of ICD‐9 and ICD‐10 codes used to identify acute liver injury: A study in three European data sources Joan Forns1 | Miguel Cainzos‐Achirica1 | Maja Hellfritzsch2 | Rosa Morros3,4,5 | Beatriz Poblador‐Plou6 | Jesper Hallas2 | Maria Giner‐Soriano3,4,5 | Alexandra Prados‐Torres6 | Anton Pottegård2 | Jordi Cortés7 | Jordi Castellsagué1 | Emmanuelle Jacquot8 | Nicolas Deltour8 | Susana Perez‐Gutthann1 | Manel Pladevall1,9 1 Epidemiology, RTI Health Solutions, Barcelona, Spain Abstract 2 Clinical Pharmacology and Pharmacy, Purpose: Validating cases of acute liver injury (ALI) in health care data sources is Department of Public Health, University of Southern Denmark, Odense, Denmark challenging. Previous validation studies reported low positive predictive values (PPVs). 3 Fundació Institut Universitari per a la recerca Methods: Case validation was undertaken in a study conducted from 2009 to 2014 a l'Atenció Primària de Salut Jordi Gol i Gurina assessing the risk of ALI in antidepressants users in databases in Spain (EpiChron and (IDIAPJGol), Barcelona, Spain 4 Universitat Autònoma de Barcelona, SIDIAP) and the Danish National Health Registers. Three ALI definitions were evalu- Bellaterra (Cerdanyola del Vallès), Barcelona, ated: primary (specific hospital discharge codes), secondary (specific and nonspecific Spain 5 hospital discharge codes), and tertiary (specific and nonspecific hospital and outpatient Institut Català de la Salut, Barcelona, Spain 6 codes). The validation included review of patient profiles (EpiChron and SIDIAP) and of EpiChron Research Group. Aragon Health Sciences Institute (IACS), IIS Aragón, clinical data from medical records (EpiChron and Denmark). ALI cases were confirmed REDISSEC ISCIII, Zaragoza, Spain when liver enzyme values met a definition by an international working group. 7 Departament d'Estadística i Investigació Operativa, Universitat Politècnica de Results: Overall PPVs (95% CIs) for the study ALI definitions were, for the primary Catalunya, Barcelona, Spain ALI definition, 84% (60%‐97%) (EpiChron), 60% (26%‐88%) (SIDIAP), and 74% (60%‐ 8 Pharmacoepidemiology Department, Les 85%) (Denmark); for the secondary ALI definition, 65% (45%‐81%) (EpiChron), 40% Laboratoires Servier, Paris, France 9 The Center for Health Policy and Health (19%‐64%) (SIDIAP), and 70% (64%‐77%) (Denmark); and for the tertiary ALI Services Research, Henry Ford Health System, definition, 25% (18%‐34%) (EpiChron), 8% (7%‐9%) (SIDIAP), and 47% (42%‐52%) Detroit, MI, USA (Denmark). The overall PPVs were higher for specific than for nonspecific codes Correspondence and for hospital discharge than for outpatient codes. The nonspecific code “unspeci- J. Forns, RTI Health Solutions, Av. Diagonal 605, 9‐1, 08028 Barcelona, Spain. fied jaundice” had high PPVs in Denmark. Email: jforns@rti.org Conclusions: PPVs obtained apply to patients using antidepressants without Funding information preexisting liver disease or ALI risk factors. To maximize validity, studies on ALI should Les Laboratoires Servier prioritize hospital specific discharge codes and should include hospital codes for unspec- ified jaundice. Case validation is required when ALI outpatient cases are considered. K E Y W OR D S acute liver injury, antidepressants, pharmacoepidemiology, validation -------------------------------------------------------------------------------------------------------------------------------- This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2019 The Authors. Pharmacoepidemiology & Drug Safety Published by John Wiley & Sons Ltd. Pharmacoepidemiol Drug Saf. 2019;28:965–975. wileyonlinelibrary.com/journal/pds 965 966 FORNS ET AL. 1 | I N T RO D U CT I O N KEY POINTS Acute liver injury (ALI) is defined as a sudden appearance of liver test • Case validation of acute liver injury (ALI) was conducted abnormalities and includes a broad spectrum of clinical scenarios, in two Spanish databases, EpiChron and SIDIAP, and in ranging from mild abnormal biochemical liver values to acute liver 1,2 the Danish national registers. failure. Previous validation studies have shown that identification of • Validation of potential cases included patient profiles potential ALI events through diagnosis and procedural codes is chal- review and adjudication based on clinical data extracted lenging and that most validated algorithms have positive predictive from medical records. values (PPVs) below 60%, 3-5 except in one study, which reported • The overall PPVs obtained were higher for specific than 6 PPVs >75%. All previous studies highlight the need for validation by for nonspecific codes and for hospital discharge than for medical record review when conducting studies of ALI based on auto- outpatient codes. mated health care data sources. This is especially important in drug • The nonspecific code “unspecified jaundice” had high safety studies, in which reliance on algorithms alone for automated PPVs for all ALI definitions in Denmark but not in the case identification will most likely result in misclassification and over- Spanish databases. estimation of the true incidence of ALI and biased effect estimates. • To maximize validity, studies on ALI should prioritize As part of a recent post‐authorization safety study (PASS) con- hospital specific discharge codes. ducted in five European data sources investigating the potential risk of ALI associated with the use of agomelatine and nine other antide- pressant drugs,7 validation of the algorithms used to identify ALI cases Nevertheless, an external validation study was conducted in was conducted. This was done via medical record review in three of Germany,17 the results of which will be presented in a separate those data sources: two Spanish health care databases and the Danish publication. National Health Registers. 2 | METHODS 2.2 | Identification and definition of ALI The objective of this study was to determine the ability of three ALI Cases of ALI were identified in cohorts of new users of the 10 study definitions to correctly identify ALI cases in three automated health antidepressants evaluated in the agomelatine PASS study between care data sources. Specifically, we aimed to validate the following: 2009 and 20147: citalopram, agomelatine, fluoxetine, paroxetine, ser- traline, escitalopram, duloxetine, venlafaxine, mirtazapine, and amitrip- • An ALI definition including only main hospital discharge diagnosis tyline. Individuals aged 18 years or older at the date of their first‐ specific codes recorded prescription fill of any of the study antidepressants during the study period(s) entered the cohort if they (a) had not received a • An ALI definition including main hospital discharge diagnosis prescription fill for the same study antidepressant within the prior specific and nonspecific codes 12 months (new users) and (b) had at least 12 months of continuous • An ALI definition including main hospital discharge and also outpa- enrolment in the data source before the first prescription fill. Absence tient diagnosis codes (both specific and nonspecific) of pregnancy at the start date of antidepressant use was an additional inclusion criterion for women. Patients with a history of liver disease or risk factors for liver disease (eg, alcohol and drug abuse and 2.1 | Study setting dependence‐related disorders), chronic biliary or pancreatic disease, malignancy, or other life‐threatening conditions (eg, HIV infection) Five automated health care databases were used in the agomelatine were excluded from the study cohort (Supplementary eMethods). 7 PASS. Three of these were used to conduct a validation study: in Three algorithms corresponding to three ALI definitions were used Spain, the EpiChron Cohort Study from Aragon Health Sciences Insti- in the agomelatine PASS to automatically identify potential ALI cases tute (Aragón, Spain)8 and the Information System for Research in Pri- based on diagnosis codes (Table 1).7,18 These definitions include com- mary Care (SIDIAP) (Catalonia, Spain)9; and in Denmark, the Danish binations of codes that have shown higher (specific) or lower (nonspe- 10,11 National Health Registers (Denmark). The main characteristics of cific) PPVs in previous validation studies.3-6 The primary ALI definition each database are included in Supplementary eTable S1. Of the two was defined as any patient with a specific main hospital discharge databases that were not used, validation by review of medical records diagnosis code of ALI from either the International Classification of Dis- is not an option in the German Pharmacoepidemiological Research eases, Ninth Revision, Clinical Modification (ICD‐9‐CM) or the Interna- 12-14 Database (GePaRD) (Germany) and was not feasible within the tional Statistical Classification of Diseases and Related Health Problems, study timeframe in the Swedish National Registers (Sweden).15,16 Tenth Revision (ICD‐100) (Table 2). The primary ALI definition was FORNS ET AL. 967 TABLE 1 ICD‐9‐CM and ICD‐10 codes relevant to acute liver injury of the secondary study ALI definition was defined as any patient with a main hospital specific or nonspecific discharge code (ICD‐9‐CM or Code Description ICD‐10) for ALI. Finally, the algorithm for the tertiary ALI definition Specific codes was assessed using specific and nonspecific codes from either ICD‐ ICD‐9‐CM 9‐CM or ICD‐10 identified in both hospital and outpatient settings. 570.x Acute and subacute necrosis of liver In EpiChron, International Classification of Primary Care (ICPC) codes 572.2 Hepatic coma were used to identify outpatient cases of the tertiary ALI definition 573.3 Hepatitis unspecified and ICD‐9‐CM to identify hospital cases. In SIDIAP, ICD‐10‐CM was ICD‐10 used to identify primary care diagnoses and ICD‐9‐CM to identify hos- pital cases. In Denmark, primary care codes were not available, and K71.0 Toxic liver disease with cholestasis therefore only hospital ICD‐10 codes were used both for case identi- K71.1 Toxic liver disease with hepatic necrosis fication and to apply exclusion criteria. The interplay between the K71.2 Toxic liver disease with acute hepatitis three ALI definitions is displayed in Figure 1. K71.6 Toxic liver disease with hepatitis, not elsewhere classified K71.9 Toxic liver disease, unspecified 2.3 | Diagnostic criteria for ALI K72.0 Acute and subacute hepatic failure Potential cases of ALI identified with the electronic algorithms and K72.9 Hepatic failure, unspecified reviewed by adjudicators were considered confirmed (true positives)19 K75.9 Inflammatory liver disease, unspecified if any of the following three qualifying criteria for increases in serum K76.2 Central hemorrhagic necrosis of liver levels with <1 year of persistence were met (aspartate transaminase Nonspecific codes [AST] levels could be used instead of ALT levels only if ALT levels ICD‐9‐CM were unavailable and there was no known muscle pathology driving 573.8 Other specified disorders of liver the rise in AST): 573.9 Unspecified disorders of liver • ≥ 5 x upper limit of normal (ULN) alanine aminotransferase (ALT) 782.4 Jaundice, unspecified, not of newborn V42.7 Liver transplant • ≥ 2 x ULN alkaline phosphatase (ALP) 790.4 Nonspecific elevation of transaminase or lactic acid • ≥ 3 x ULN ALT and > 2 x ULN bilirubin dehydrogenase The requirement of less than 1 year of persistence of the liver func- 789.1 Hepatomegaly tion test abnormalities was introduced to ensure that cases had ALI ICD‐10 and not chronic liver injury.19 This criterion was evaluated using the most recent liver enzymes results from the period 12 to 24 months K76.8 Other specified diseases of liver before the index date to check whether they were not elevated K76.9 Liver disease, unspecified beyond 10% of the ULN (if no results were available, the criterion R17 Unspecified jaundice, excludes neonatal was considered as met). R16.0 Hepatomegaly, not elsewhere classified R16.2 Hepatomegaly with splenomegaly, not elsewhere A false‐positive case of ALI was defined as a potential case with classified enough data to be evaluated but that did not meet the criteria to be R74.0 Nonspecific elevation of transaminase and lactic acid classified as a confirmed case of ALI. A nonevaluable case of ALI was dehydrogenase defined as a potential case that lacked some of the required liver Z94.4 Liver transplant enzyme results to be evaluated. ICPC D97 Liver disease (specified or unspecified) 2.4 | Validation steps D13 Jaundice D23 Hepatomegaly The strategy for validating potential cases identified by automated A91 Abnormal results investigations algorithms across the three data sources included up to three steps: Abbreviations: ICD‐9‐CM, International Classification of Diseases, Ninth review of patient profiles (which is a deidentified chronological listing Revision, Clinical Modification; ICD‐10, International Statistical Classification of medical events and drug prescriptions and is used to detect exclu- of Diseases and Related Health Problems, Tenth Revision; ICPC, International sion diagnoses missed by the electronic algorithm and to provide an Classification of Primary Care. initial assignment of case status), medical record abstraction of rele- not validated per se, but the specific codes identifying the primary ALI vant clinical data by trained health care professionals, and review of definition were included in the secondary ALI definition, which abstracted data and case adjudication by trained physicians. However, underwent validation. The algorithm used to identify potential cases local adaptations were required in Denmark and SIDIAP to reflect data 968 FORNS ET AL. TABLE 2 Positive predictive values (PPVs) of study ALI definitions and of overall specific and nonspecific codes used to identify potential acute liver injury (ALI) cases (nonevaluable cases not included) EpiChron SIDIAP Denmark a b a b Total TP PPV, % (95% CI) Total TP PPV, % (95% CI) Totala TP PPV, % (95% CI)b Secondary ALI definition 31 20 64.5 (45.4‐80.8) 20 8 40.0 (19.1‐63.9) 213 150 70.4 (63.8‐76.5) c Specific codes 19 16 84.2 (60.4‐96.6) 10 6 60.0 (26.2‐87.8) 50 37 74.0 (59.7‐85.4) Nonspecific codes 12 4 33.3 (9.9‐65.1) 10 2 20.0 (2.5‐55.6) 163 113 69.3 (61.6‐76.3) Tertiary ALI definition 134 34 25.4 (18.3‐33.6) 2,242 172 7.7 (6.6‐8.9) 443 208 47.0 (42.2‐51.7) Specific codes 18 15 83.3 (58.6‐96.4) 46 16 34.8 (21.4‐50.2) 73 50 68.5 (56.6‐78.9) Nonspecific codes 116 19 16.4 (10.2‐24.4) 2,196 156 7.1 (6.1‐8.3) 370 158 42.7 (37.6‐47.9) Abbreviations: CI, confidence interval; SIDIAP, Information System for Research in Primary Care; TP, true positives. a Total of evaluable cases. Nonevaluable cases for the secondary and tertiary ALI definitions were 9 and 104 in EpiChron, 14 and 584 in SIDIAP, and 28 and 66 in Denmark. b PPV was calculated as PPV = confirmed cases / (true positives + false positives). Results are presented as positive predictive values (%) and their 95% CIs. c Equivalent to the PPVs for the study primary ALI definition (specific hospital discharge codes). including nurses, clinical pharmacists and physicians, received training on the validation processes. In EpiChron, for quality control purposes, patient profiles of a random sample of 10 potential cases were reviewed independently by a second physician and a random sample of 25% of the confirmed cases and of 10 inpatient nonevaluable cases also were reviewed by a second physician. In SIDIAP, for the tertiary ALI definition, an electronic algorithm evaluated all potential cases, and 10% of them were also evaluated manually by trained profes- sionals blinded to the study exposure. A very high level of agreement (kappa statistic equal to or larger than 0.95) between the algorithm and the manual reviewers was obtained before the algorithm was gen- eralized; agreement between the two clinician reviewers was also assessed (kappa statistic = 1). Similarly, in Denmark, an algorithm was created to evaluate potential cases. Trained physicians manually reviewed 50 potential cases, all of which were also reviewed using the automated algorithm. All potential cases were evaluated using FIGURE 1 Definition of the study ALI definition algorithmsa a ALI definition refers to the case‐identifying algorithms only. By the automated algorithm only after the kappa measuring the agree- definition, the secondary ALI definition in the analysis included only ment between manual review and the algorithm reached 1. cases confirmed after validation [Colour figure can be viewed at wileyonlinelibrary.com] 2.5 | Statistical analyses availability and/or local regulations (Supplementary eTable S2). In Denmark, patient profiles were not reviewed due to the very limited Validity of the electronic algorithms and individual codes used to iden- clinical information available. Also, primary care data were not tify potential cases of ALI for the secondary and tertiary ALI defini- available. Finally, patients with study exclusion criteria not identified tions were assessed by calculating the overall PPV of the algorithm, by hospital codes were excluded during either the abstraction or the the overall PPVs of the specific and nonspecific codes, and the PPV review of the abstracted information from medical records. In SIDIAP, of each individual code. PPVs for the primary ALI definition were source hospital medical records were not accessible; therefore, patient indirectly calculated through the specific codes of the secondary ALI profile review relied only on liver enzyme results available from pri- definition. The PPV was calculated as true positives / (true posi- mary care and yielded the final case classifications in this database. tives + false positives). In a sensitivity analysis, nonevaluable cases Cases were reviewed both by trained physicians for all secondary were included in the PPV denominator. ALI definition potential cases and by an electronic algorithm for The PPVs were computed with 95% confidence intervals (CIs) for the tertiary ALI definition due to the large number of identified binomial proportions by the exact method using Stata software20— potential cases. version 12 at EpiChron and version 14 at Denmark. At SIDIAP, SAS Several quality control checks and measures were performed. All statistical software (version 9.4; SAS Institute, Inc; Cary, North the health care professionals at each site involved in the validation, Carolina) and R software version 3.3.1 were used. FORNS ET AL. 969 3 | RESULTS Regarding the tertiary ALI definition, which includes the total num- ber of cases for all ALI definitions (see Figure 1), more than 70% of The number of users of antidepressants and the final number of new true positives in Denmark and SIDIAP and 56% of true positives in users (after applying inclusion/exclusion criteria) in the three data- EpiChron were females. Overall, the age group with the highest num- bases in which validation of potential cases was conducted are ber of true positives was patients 80 years and older, followed by included Supplementary eTable S3. In EpiChron, SIDIAP, and patients aged 50 to 79 years (Supplementary eTable S4). Denmark, 59, 34, and 489 potential cases of the secondary ALI defini- The overall PPVs for the algorithm used to identify potential cases tion, respectively, were identified; and 268, 2826, and 1008 potential of the secondary ALI definition were 65% (95% CI, 45%‐81%) in cases of the tertiary ALI definition were identified. Then, 31, 20, and EpiChron, 40% (95% CI, 19%‐64%) in SIDIAP, and 70% (95% CI, 213 potential cases of the secondary ALI definition were considered 64%‐77%) in Denmark (Table 2). As discussed in the Methods section, evaluable cases; and 134, 2242, and 443 potential cases of the tertiary the primary ALI definition was indirectly validated through the specific ALI definition were considered evaluable cases. Of them, 20, 8, and hospital discharge codes used in the secondary ALI definition, for which 150 cases of the secondary ALI definition and 34, 172, and 208 cases the overall PPVs were 84% (95% CI, 60%‐97%) in EpiChron, 60% (95% of the tertiary ALI definition were confirmed (true positives) after CI, 26%‐88%) in SIDIAP, and 74% (95% CI, 60%‐85%) in Denmark. The validation (Figure 2). overall PPVs for the specific codes were higher than those for the FIGURE 2 Flowchart with the flow of potential cases through the case validation process: Secondary (regular font) and tertiary (italics) ALI definitions [Colour figure can be viewed at wileyonlinelibrary.com] Note: In each cell, the first number refers to secondary ALI definitions, and the second number refers to tertiary ALI definitions.a One hundred fifteen patients did not undergo further validation due to the lack of additional hospital data for those cases. Among them, eight patients were excluded based on the presence of exclusion or censoring criteria and did not undergo further validation.b One hundred seven patients identified on ambulatory codes and with lack of additional hospital data were directly adjudicated during the patient profile phase. Among them, three were classified as true positives, 69 as false positives, and 35 were considered nonevaluable.c Patients with study exclusion criteria not identified by hospital codes were excluded during the abstraction or review of medical records. 970 FORNS ET AL. nonspecific codes in all data sources (Table 2). In EpiChron and SIDIAP, had the highest PPV among all individual specific or nonspecific hospi- the individual specific code 570.x (acute and subacute necrosis of liver) tal discharge codes. had the highest PPV, while the code 573.3 (hepatitis unspecified) cap- For the tertiary ALI definition, the overall PPVs were 25% (95% CI, tured the highest proportion of true positives (Table 3). In Denmark, 18%‐34%) in EpiChron, 8% (95% CI, 7%‐9%) in SIDIAP, and 47% (95% the individual specific codes K71.2 (toxic liver disease with acute hepa- CI, 42%‐52%) in Denmark. As observed for the secondary ALI defini- titis) and K71.6 (toxic liver disease with hepatitis, not elsewhere speci- tion, we observed higher PPVs for specific than nonspecific codes in fied) obtained the highest PPVs and captured the highest proportion of all data sources (Table 2). Among the individual specific codes, 570.x true positives (Table 4). None of the nonspecific codes captured more (acute and subacute necrosis of liver) had the highest PPV in EpiChron than two true positives in EpiChron and SIDIAP (Table 3). Conversely, and SIDIAP (Table 3 and Supplementary eTable S5). In Denmark, code in Denmark, the individual nonspecific code R17 (unspecified jaundice, K71.2 (toxic liver disease with acute hepatitis) had the highest PPV excludes neonatal) contributed the largest number of true positives and among specific codes (Table 4). Among the nonspecific codes, 782.4 TABLE 3 Positive predictive values (PPVs) of specific and nonspecific codes used to identify potential acute liver injury (ALI) cases: Secondary (regular font) and tertiary (italics) ALI definitions in data sources using ICD‐9‐CM codes (nonevaluable cases not included) EpiChron SIDIAP a Total TP PPV, % (95% CI) Total TP PPV, % (95% CI)a Specific codes 570.x Acute and subacute necrosis of liver Secondary ALI definition 5 5 100.0 (47.82‐100.0) 3 3 100.0 (29.2‐100.0) Tertiary ALI definition 5 5 100.0 (47.8‐100.0) 1 1 100.0 (2.5‐100.0) 572.2 Hepatic coma Secondary ALI definition 1 0 0.0 (0.0‐97.5) 0 ‐ ‐ Tertiary ALI definition 1 0 0 (0‐97.5) 0 ‐ ‐ 573.3 Hepatitis unspecified Secondary ALI definition 13 11 84.6 (54.6‐98.1) 7 3 42.9 (9.9‐81.6) Tertiary ALI definition 12 10 83.3 (51.6‐97.9) 4 3 75.0 (19.4‐99.4) Nonspecific codes 573.8 Other specified disorders of liver Secondary ALI definition 9 2 22.2 (2.8‐60.0) 6 0 0.0 (0.0‐45.9) Tertiary ALI definition 9 2 22.2 (2.8‐60.0) 5 0 0.0 (0.0‐52.2) 573.9 Unspecified disorders of liver Secondary ALI definition 1 0 0.0 (0.0‐97.5) 0 0 ‐ Tertiary ALI definition 0 0 ‐ 0 0 ‐ 782.4 Jaundice, unspecified, not of newborn Secondary ALI definition 1 1 100 (2.5‐100) 2 2 100 (15.8‐100) Tertiary ALI definition 1 1 100 (2.5‐100) 2 2 100 (15.8‐100) V42.7 Liver transplant Secondary ALI definition 0 ‐ ‐ 0 ‐ ‐ Tertiary ALI definition 0 ‐ ‐ 0 ‐ ‐ 790.4 Nonspecific elevation of transaminase or LDH Secondary ALI definition 1 1 100.0 (2.5‐100.0) 2 0 0.0 (0.0‐84.2) Tertiary ALI definition 1 1 100.0 (2.5‐100.0) 1 0 0.0 (0.0‐97.5) 789.1 Hepatomegaly Secondary ALI definition 0 ‐ ‐ 0 0 ‐ Tertiary ALI definition 0 ‐ ‐ 0 0 ‐ Abbreviations: CI, confidence interval; ICD‐9‐CM, International Classification of Diseases, Ninth Revision, Clinical Modification; LDH, lactic acid dehydroge- nase; TP, true positives. Note: PPVs for the ICPC codes used to define cases for the tertiary ALI definition in EpiChron are presented in eTable 5. a PPV was calculated as PPV = confirmed cases / (true positives + false positives). Results are presented as positive predictive values (%) and their 95% CIs. FORNS ET AL. 971 TABLE 4 Positive predictive values (PPVs) of specific and nonspecific codes used to identify potential acute liver injury (ALI) cases: Secondary (regular font) and tertiary (italics) ALI definitions in data sources using ICD‐10‐CM codes (nonevaluable cases not included) SIDIAPa Denmarkb Total TP PPV, % (95% CI)c Total TP PPV, % (95% CI)c Specific codes K71.0 Toxic liver disease with cholestasis Secondary ALI definition n<5 n<5 50.0 (1.3–98.7) Tertiary ALI definition 0 ‐ ‐ 5 n<5 60.0 (14.7‐94.7) K71.1 Toxic liver disease with hepatic necrosis Secondary ALI definition 5 n<5 40.0 (5.3‐85.3) Tertiary ALI definition 0 ‐ ‐ 6 n<5 33.3 (4.3‐77.7) K71.2 Toxic liver disease with acute hepatitis Secondary ALI definition 9 8 88.9 (51.8‐99.7) Tertiary ALI definition 0 ‐ ‐ 13 12 92.3 (64.0‐99.8) K71.6 Toxic liver disease with hepatitis, not elsewhere classified Secondary ALI definition 8 7 87.5 (47.3‐99.7) Tertiary ALI definition 5 2 40.0 (5.3‐85.3) 9 8 88.9 (51.8‐99.7) K71.9 Toxic liver disease, unspecified Secondary ALI definition 5 n<5 80.0 (28.4‐99.5) Tertiary ALI definition 1 0 0.0 (0.0‐97.5) 12 6 50.0 (21.1‐78.9) K72.0 Acute and subacute hepatic failure Secondary ALI definition 7 6 85.7 (42.1‐99.6) Tertiary ALI definition 3 2 66.7 (9.4‐99.2) 9 8 88.9 (51.8‐99.7) K72.9 Hepatic failure, unspecified Secondary ALI definition 10 6 60.0 (26.2‐87.8) Tertiary ALI definition 8 1 12.5 (0.3‐52.7) 13 7 53.8 (25.1‐80.8) K75.9 Inflammatory liver disease, unspecified Secondary ALI definition n<5 n<5 66.7 (9.4‐99.2) Tertiary ALI definition 23 7 30.4 (13.2‐52.9) 5 n<5 60.0 (14.7‐94.7) K76.2 Central hemorrhagic necrosis of liver Secondary ALI definition n<5 n<5 100 (2.5‐100) Tertiary ALI definition 0 ‐ ‐ n<5 n<5 100 (2.5‐100) Nonspecific codes K76.8 Other specified diseases of liver Secondary ALI definition 16 n<5 6.3 (0.2‐30.2) Tertiary ALI definition 111 1 0.9 (0.0‐4.9) 35 n<5 11.4 (3.2‐26.7) K76.9 Liver disease, unspecified Secondary ALI definition 30 15 50.0 (31.3‐68.7) Tertiary ALI definition 116 11 9.5 (4.8‐16.3) 107 33 30.8 (22.3‐40.5) R17 Unspecified jaundice, excludes neonatal Secondary ALI definition 79 75 94.9 (87.5‐98.6) Tertiary ALI definition 57 20 35.1 (22.9‐48.9) 90 82 91.1 (83.2‐96.1) R16.0 Hepatomegaly, not elsewhere classified Secondary ALI definition 7 n<5 42.9 (9.9‐81.6) Tertiary ALI definition 52 3 5.8 (1.2‐15.9) 12 n<5 25.0 (5.5‐57.2) (Continues) 972 FORNS ET AL. TABLE 4 (Continued) SIDIAPa Denmarkb Total TP PPV, % (95% CI)c Total TP PPV, % (95% CI)c R16.2 Hepatomegaly with splenomegaly, not elsewhere classified Secondary ALI definition n<5 n<5 75.0 (19.4‐99.4) Tertiary ALI definition 0 ‐ ‐ 6 n<5 50.0 (11.8‐88.2) R74.0 Nonspecific elevation of transaminase and LDH Secondary ALI definition 27 16 59.3 (38.8‐77.6) Tertiary ALI definition 1,852 119 6.4 (5.4‐7.6) 120 33 27.5 (19.7‐36.4) Z94.4 Liver transplant Secondary ALI definition 0 ‐ ‐ Tertiary ALI definition 0 ‐ ‐ 0 ‐ ‐ Abbreviations: CI, confidence interval; ICD‐10‐CM, International Classification of Diseases, Tenth Revision, Clinical Modification; LDH, lactic acid dehydroge- nase; TP, true positives. a In SIDIAP, ICD‐10 codes were used only for the outpatient codes of the study tertiary ALI definition. b Due to data protection policies in Denmark, the exact number of cases could not be provided when the number of cases was less than five. c PPV was calculated as PPV = confirmed cases / (true positives + false positives). Results are presented as positive predictive values (%) and their 95% CIs. (jaundice, unspecified, not of newborn) had the highest PPV in both the studies might be explained by the low prevalence of ALI or DILI. EpiChron and SIDIAP, although it had a low number of confirmed In addition, a different list of diagnosis codes, laboratory threshold cases (one and two true positives in EpiChron and SIDIAP, respec- criteria, and study drugs might be the cause of the differences tively). In Denmark, ICD‐10 code R17 (unspecified jaundice, excludes between studies. When we compared our study with previous studies neonatal) had the highest PPV (91%) and contributed the largest validating ALI definitions, we observed that our study differed from number of true positives. In SIDIAP, the same code used to identify these previous studies in different ways: Bui et al6 did not exclude primary care diagnoses had the second highest PPV, and it was also patients with hepatic, biliary, or pancreatic diseases or cancer; Lo Re the second highest contributor of true positives. Regarding code et al3 included only cases of severe ALI; Udo et al5 validated cases R74.0 (nonspecific elevation of transaminase or LDH), in SIDIAP, it of idiopathic ALI only; and Traversa et al4 validated cases of ALI asso- was the code with the highest number of true positives, although it ciated with the use of nonsteroidal antiinflammatory drugs. In addi- had a low PPV (6%). tion, there are differences in the type of data sources: the Bui et al6 In the sensitivity analysis including nonevaluable cases in the and Lo Re et al3 studies were conducted in claims databases including denominator of the PPV calculation, the overall PPVs for all study inpatient and outpatient encounters, prescriptions, and laboratory ALI definitions and for both specific and nonspecific codes were tests. The Traversa et al4 and Udo et al5 studies were conducted in smaller than those for the main PPV analysis in all data sources (see hospital databases in a way similar to the Danish component of our Supplementary eTables S6 and S7). study. There are also differences in the ALI definition used in previous studies compared with the criteria used in our study, which were based on Aithal criteria.19 4 | DISCUSSION Positive predictive values obtained in the present study for the ICD‐9 specific codes 573.3 (hepatitis unspecified) and 570.x (acute We observed consistently higher overall PPVs for specific ALI codes and subacute necrosis of liver) and specific ICD‐10 codes K71.2 versus nonspecific codes and higher overall PPVs for hospital dis- (toxic liver disease with acute hepatitis) and K71.6 (toxic liver disease charge codes versus outpatient codes. The identification of ALI cases with hepatitis, not elsewhere specified) were in line with previous based on hospital discharge specific codes, considered as the primary studies. In Udo et al,5 the code 573.3 had a PPV of 80%. In Bui ALI definition in this study, resulted in higher PPVs when compared et al,6 the PPV for individual code 570.x was 84% and for 573.3 3-6 with most previously described algorithms. was 76%, while the PPV for the algorithm including codes 570.x, In contrast to the present study, previous studies conducted to val- 572.2 (hepatic coma), or 573.3 was 74%. In Lo Re et al,3 the PPVs 3-5 6 idate ALI cases have reported PPVs below 60%, or around 75%. A for individual codes ranged from 6.5% to 54.3%, the combination recently published systematic review and meta‐analysis including 29 of codes 570.x with 572.8 (sequelae of liver disease; hepatic failure) studies validating ALI or drug‐induced liver injury (DILI) (25 of them had a PPV of 100%, and code 570.x in combination with 572.2 had a presenting PPVs) showed a pooled PPV estimate for ALI of 13.4% PPV of 67%. In addition, the authors calculated PPVs including 21 (95% CI, 6.1%‐22.8%) and for DILI of 15.3% (95% CI, 9.5%‐22.2%). patients with preexisting liver disease, and the PPVs were higher The authors of that study suggested that the low PPVs observed in when compared with the subset of the population that excluded FORNS ET AL. 973 those patients.3 In two studies validating drug‐induced ALI definition in SIDIAP. In Denmark, only outpatient potential cases (DILI),22,23 code 573.3 (hepatitis unspecified) was the highest con- from hospital outpatient clinics could be identified (primary care data tributor of DILI cases. were not available). This is probably the reason why the difference in In the present study, the nonspecific code for unspecified jaundice PPVs between specific and nonspecific codes was smaller in (R17) obtained high PPVs, and it was the highest contributor of true Denmark than in the other data sources, and it would also explain positives in Denmark. In EpiChron and SIDIAP databases, the ICD‐9‐ the higher PPVs obtained in Denmark for the secondary and tertiary CM code 782.4 (jaundice, unspecified, not of newborn) had high PPVs ALI definitions compared with the two Spanish data sources. For for the secondary ALI definition (hospitalized cases), although the some codes, the number of cases was low, resulting in wide CIs for number of true positives was one and two cases, respectively. In the PPV. The present study has also other limitations. First, we did SIDIAP, the ICD‐10 code for unspecified jaundice used in the tertiary not conduct validation of false positives, and therefore negative pre- ALI definition to validate hospitalized and outpatient cases was the dictive values could not be estimated. Second, the PPVs obtained in second contributor of true positives and had the second‐highest the present study apply only to patients using the study antidepres- PPV, although it was low (35%). Potential explanations for this sant drugs who did not have preexisting liver disease or risk factors discrepancy in the results for unspecified jaundice code between for developing ALI. Third, PPVs are dependent on the ALI case defi- Denmark and Spanish data sources could be the following: (a) in nition used. In the present study, we used the definition created by Denmark, only hospitalized and outpatient cases from hospital outpa- Aithal et al,19 but there are other case definitions that could be tient clinics are validated; and (b) in Denmark, exclusion criteria not used24,25 and PPVs could have been different with those other case identified previously were applied, if identified, during either the definition criteria. Finally, PPVs are dependent on ALI prevalence. abstraction or the review of the abstracted information from medical Therefore, the PPVs observed in our study might not apply directly records. These reasons may reduce the presence of false positives to patient populations with characteristics different from those and justify the high PPV observed for this code in Denmark compared included in the present study or to studies using different case with Spanish data sources. Results observed in Denmark also contrast definitions. with those in a previous study,23 which reported that the nonspecified code for unspecified jaundice identified only a small proportion of DILI 5 | CO NC LUSIO NS cases (5% of the 265 cases in Shin et al23 vs 39% of the 208 cases of the tertiary ALI definition confirmed in Denmark observed in our The PPVs obtained in this study apply to patients using antidepres- study), but the differences when validating ALI or DILI cases must be sants without preexisting liver disease or risk factors for ALI. Future taken into account. In addition, the study by Shin et al23 was not studies evaluating ALI in these and similar data sources should priori- restricted to hospital cases as it was in Denmark, and thus the preva- tize use of hospital discharge and specific codes to maximize validity. lence of true ALI in populations including outpatient primary care Moreover, case‐identifying algorithms should include hospital ICD cases should be lower, which would explain the differences observed codes for unspecified jaundice. In studies including nonspecific codes between the two studies. and outpatient cases, case validation is essential. ET HIC S ST AT EME NT 4.1 | Strengths and limitations RTI International institutional review board approval to conduct the In terms of number of validated cases, the present validation study study was granted on 04 August 2015. The following data source‐spe- represents one of the largest efforts performed in Europe to validate cific approvals were obtained: in EpiChron, Ethics committee approval ALI cases identified in automated health care databases, using case‐ was obtained from the Comité Etico de Investigación Clínica de identifying algorithms, and confirmed according to consensus criteria Aragón (07 October 2015) and the Spanish Agency of Medicines and based on the presence of elevated liver enzyme levels in blood. In Medical Devices (AEMPS) (08 September 2015). In SIDIAP, ethics addition, to the best of our knowledge, this study is the first to vali- committee approval was obtained from the IDIAP Jordi Gol Ethics date ICD‐10 codes related to ALI. However, the results obtained in Committee (23 December 2015). In Denmark, the authorization of the present study must be evaluated in the context of its limitations. the Danish Data Protection Agency was granted on 15 June 2015. An important limitation of this study is that, although the ALI defini- The Danish Health Authority provided authorization to access medical tions were consistent across data sources and based on blood liver records, granted on 29 March 2016. enzyme levels, the approach to the evaluation of potential cases was adapted to the type of information and local resources available ACKNOWLEDGEMENTS for the validation efforts, which may have impacted our findings. In Carla Franzoni, BSc1, for the management of the project and coordina- SIDIAP, the validation was partial for all potential cases (inpatient tion of activities between Research Partners in SIDIAP, EpiCHron and and outpatient), based only on liver enzyme results from primary Denmark, RTI‐HS and Servier teams. care, and no hospital medical records to validate hospital cases were Morten Olesen2, for his contribution to the validation activities in available. That could explain the lowest PPV for the secondary ALI Denmark. 974 FORNS ET AL. OPEN, Odense Patient data Explorative Network, Odense version of the protocol. Authors Alexandra Prados‐Torres, Beatriz University Hospital, Odense Denmark, for his contribution to the Poblador‐Plou, Maria Giner‐Soriano, Rosa Morros, Jordi Cortés, Anton validation activities in Denmark. Pottegård, Jesper Hallas, and Maja Hellfritzsch undertook the statisti- cal analysis of the different data sources. Joan Forns, Miguel Cainzos‐ CONF LICT OF INT E RE ST Achirica, Susana Perez‐Gutthann, and Manel Pladevall wrote the first draft of the manuscript. All authors contributed to and have approved Manel Pladevall, Joan Forns, Miguel Cainzos‐Achirica, Jordi the final manuscript. Castellsagué, and Susana Perez‐Gutthann are employees of RTI Health, a unit of RTI International, a nonprofit organization that ORCID conducts work for government, public, and private organizations, including pharmaceutical companies. Joan Forns https://orcid.org/0000-0002-1066-0358 Alexandra Prados‐Torres and Beatriz Poblador‐Plou are members Miguel Cainzos‐Achirica https://orcid.org/0000-0002-3805-9089 of the EpiChron Research Group on Chronic Diseases of the Aragon Jesper Hallas https://orcid.org/0000-0002-8097-8708 Health Sciences Institute (IACS), ascribed to IIS Aragón, and do not Maria Giner‐Soriano https://orcid.org/0000-0003-3750-9233 have any conflict of interest with this project. Anton Pottegård https://orcid.org/0000-0001-9314-5679 Maria Giner‐Soriano, Rosa Morros, and Jordi Cortés worked on other projects funded by pharmaceutical companies in their institution RE FE RE NC ES that were not related to this study and without personal profit. 1. Hussaini SH, Farrington EA. Idiosyncratic drug‐induced liver injury: an Anton Pottegård reports participation in research projects funded update on the 2007 overview. Expert Opin Drug Saf. 2014;13(1): 67‐81. https://doi.org/10.1517/14740338.2013.828032 by Alcon, Almirall, Astellas, AstraZeneca, and Servier, all with funds 2. Lee WM, Stravitz RT, Larson AM. Introduction to the revised American paid to the institution where he was employed (no personal fees) Association for the Study of Liver Diseases Position Paper on acute and with no relation to the work reported in this paper. liver failure 2011. Hepatology. 2012;55(3):965‐967. https://doi.org/ Jesper Hallas has participated in research projects funded by 10.1002/hep.25551 Novartis, Pfizer, Menarini, MSD, Nycomed, LEO Pharma, Almirall, 3. Lo Re V 3rd, Haynes K, Goldberg D, et al. Validity of diagnostic codes Servier, Astellas, and Alkabello with grants paid to the institution to identify cases of severe acute liver injury in the US Food and Drug where he was employed. He has personally received fees for teaching Administration's mini‐sentinel distributed database. Pharmacoepidemiol Drug Saf. 2013;22(8):861‐872. https://doi.org/10.1002/pds.3470 or consulting from the Danish Association of Pharmaceutical Manufacturers and from Pfizer and Menarini. 4. Traversa G, Bianchi C, Da Cas R, Abraha I, Menniti‐Ippolito F, Venegoni M. Cohort study of hepatotoxicity associated with nimesulide and Maja Hellfritzsch has received speaker honorarium fees from other non‐steroidal anti‐inflammatory drugs. BMJ. 2003;327(7405): Bristol‐Myers Squibb and Pfizer and a travel grant from LEO Pharma. 18‐22. https://doi.org/10.1136/bmj.327.7405.18 Nicolas Deltour and Emmanuelle Jacquot are employees of Les 5. Udo R, Maitland‐van der Zee AH, Egberts TC, et al. Validity of diagnos- Laboratoires Servier. tic codes and laboratory measurements to identify patients with idiopathic acute liver injury in a hospital database. Pharmacoepidemiol FUND ING Drug Saf. 2016;25(Suppl 1):21‐28. https://doi.org/10.1002/pds.3824 This study was funded by Les Laboratoires Servier under a contract 6. Bui CL, Kaye JA, Castellsague J, et al. Validation of acute liver injury cases in a population‐based cohort study of oral antimicrobial users. granting independent publication rights to the research team. Curr Drug Saf. 2014;9(1):23‐28. 7. Pladevall‐Vila M. Post‐authorisation safety study of agomelatine and DATA AVAILABILITY the risk of hospitalisation for acute liver injury 16/01/2018 2018. http://www.encepp.eu/encepp/viewResource.htm?id=12730 The data sets used for this study are owned by each of the individual (accessed 2 March 2018). research center or by the government data custodians from which the 8. Prados‐Torres A, Poblador‐Plou B, Gimeno‐Miguel A, et al. Cohort pro- research centers obtained access to the data at IACS (Spain), SIDIAP file: the epidemiology of chronic diseases and multimorbidity. The (SIDIAP), BIPS (Germany), Karolinska Institutet (Sweden), and South- EpiChron cohort study. Int J Epidemiol. 2018;47(2):382‐384f. https:// doi.org/10.1093/ije/dyx259 ern Denmark University (Denmark). Researchers desiring access to the data sets would be required to obtain permission from research 9. SIDIAP. Database. General details. 2014. http://www.sidiap.org/index. php/en (accessed 27 September 2016). center and/or data custodians at each country. Researchers desiring access to the code used to analyze that data would be required to 10. Schmidt M, Schmidt SA, Sandegaard JL, Ehrenstein V, Pedersen L, Sorensen HT. The Danish National Patient Registry: a review of obtain permission from the research centers and the study sponsor. content, data quality, and research potential. Clin Epidemiol. 2015;7:449‐490. https://doi.org/10.2147/clep.s91125 AUT HORS' CONT R IBUT IONS 11. Pottegard A, Schmidt SA, Wallach‐Kildemoes H, Sorensen HT, Hallas J, Schmidt M. Data resource profile: the Danish National Prescription Authors Manel Pladevall, Jordi Castellsagué, Emmanuelle Jacquot, Registry. Int J Epidemiol. 2016. https://doi.org/10.1093/ije/dyw213 Nicolas Deltour, and Susana Perez‐Gutthann planned the study. All 12. Pigeot I, Ahrens W. Establishment of a pharmacoepidemiological data- authors made contributions to the final design and final approved base in Germany: methodological potential, scientific value and FORNS ET AL. 975 practical limitations. Pharmacoepidemiol Drug Saf. 2008;17(3):215‐223. 21. Tan EH, Low EXS, Dan YY, Tai BC. Systematic review and meta‐ https://doi.org/10.1002/pds.1545 analysis of algorithms used to identify drug‐induced liver injury (DILI) 13. Jobski K, Kollhorst B, Garbe E, Schink T. The risk of ischemic cardio‐ in health record databases. Liver Int. 2018;38(4):742‐753. https://doi. and cerebrovascular events associated with oxycodone‐naloxone and org/10.1111/liv.13646 other extended‐release high‐potency opioids: a nested case‐control 22. Jinjuvadia K, Kwan W, Fontana RJ. Searching for a needle in a hay- study. Drug Saf. 2017;40(6):505‐515. https://doi.org/10.1007/ stack: use of ICD‐9‐CM codes in drug‐induced liver injury. Am J s40264‐017‐0511‐8 Gastroenterol. 2007;102(11):2437‐2443. https://doi.org/10.1111/ 14. Jobski K, Schmedt N, Kollhorst B, Krappweis J, Schink T, Garbe E. j.1572‐0241.2007.01456.x Characteristics and drug use patterns of older antidepressant initiators 23. Shin J, Hunt CM, Suzuki A, Papay JI, Beach KJ, Cheetham TC. in Germany. Eur J Clin Pharmacol. 2017;73(1):105‐113. https://doi.org/ Characterizing phenotypes and outcomes of drug‐associated liver 10.1007/s00228‐016‐2145‐7 injury using electronic medical record data. Pharmacoepidemiol Drug 15. Ludvigsson JF, Andersson E, Ekbom A, et al. External review and vali- Saf. 2013;22(2):190‐198. https://doi.org/10.1002/pds.3388 dation of the Swedish national inpatient register. BMC Public Health. 24. Bénichou C. Criteria of drug‐induced liver disordersReport of an inter- 2011;11(1):450. https://doi.org/10.1186/1471‐2458‐11‐450 national consensus meeting. J Hepatol. 1990;11(2):272‐276. 16. Wettermark B, Hammar N, Fored CM, et al. The new Swedish pre- 25. FDA. Drug‐Induced Liver Injury: Premarketing Clinical Evaluation. US scribed Drug register—opportunities for pharmacoepidemiological Food and Drug Administration, 2009. https://www.fda.gov/down- research and experience from the first six months. Pharmacoepidemiol loads/guidances/UCM174090.pdf (accessed 1 June 2018). Drug Saf. 2007;16(7):726‐735. https://doi.org/10.1002/pds.1294 17. Timmer A, Kappen S, de Sordi D, Pladevall M, Perez‐Gutthann S, SUPPORTING INFORMATION Jacquot E, Deltour N, Schink T. Validity of hospital ICD‐10‐GM codes Additional supporting information may be found online in the to identify acute liver injury. Presented at the 34th International Con- Supporting Information section at the end of the article. ference on Pharmacoepidemiology and Therapeutic Risk Management; August 22–26 2018. Prague, Czech Republic.p. 262. 18. Pladevall‐Vila M, Pottegard A, Schink T, et al. Risk of acute liver injury in agomelatine and other antidepressant users in four European coun- How to cite this article: Forns J, Cainzos‐Achirica M, tries: a cohort and nested case‐control study using automated health Hellfritzsch M, et al. Validity of ICD‐9 and ICD‐10 codes used data sources. CNS Drugs. 2019;33(4):383‐395. https://doi.org/ 10.1007/s40263‐019‐00611‐9 to identify acute liver injury: A study in three European data 19. Aithal GP, Watkins PB, Andrade RJ, et al. Case definition and pheno- sources. Pharmacoepidemiol Drug Saf. 2019;28:965–975. type standardization in drug‐induced liver injury. Clin Pharmacol Ther. https://doi.org/10.1002/pds.4803 2011;89(6):806‐815. https://doi.org/10.1038/clpt.2011.58 20. StataCorp LP. Stata User's Guide: Release 14. College Station, Tex: StataCorp LP; 2015.