29b2debe6b278b9581d3e5f54ef772ed applsci-15-00146-v2.pdf 85e63dede7ded7704af2422be551c090dcae9c88 applsci-15-00146-v2.pdf 656ceea97e98e6bd29958c6f153e447af2a7a2bbfe3590c581c25ae55cfe93aa applsci-15-00146-v2.pdf Title: Use of Attention Maps to Enrich Discriminability in Deep Learning Prediction Models Using Longitudinal Data from Electronic Health Records Subject: Background: In predictive modelling, particularly in fields such as healthcare, the importance of understanding the model's behaviour rivals, if not surpasses, that of discriminability. To this end, attention mechanisms have been included in deep learning models for years. However, when comparing different models, the one with the best discriminability is usually chosen without considering the clinical plausibility of their predictions. Objective: In this work several attention-based deep learning architectures with increasing degrees of complexity were designed and compared aiming to study the balance between discriminability and plausibility with architecture complexity when working with longitudinal data from Electronic Health Records (EHRs). Methods: We developed four deep learning-based architectures with attention mechanisms that were progressively more complex to handle longitudinal data from EHRs. We evaluated their discriminability and resulting attention maps and compared them amongst architectures and different input processing approaches. We trained them on 10 years of data from EHRs from Catalonia (Spain) and evaluated them using a 5-fold cross-validation to predict 1-year all-cause mortality in a subsample of 500,000 people over 65 years of age. Results: Generally, the simplest architectures led to the best overall discriminability, slightly decreasing with complexity by up to 8.7%. However, the attention maps resulting from the simpler architectures were less informative and less clinically plausible compared to those from more complex architectures. Moreover, the latter could give attention weights both in the time and feature domains. Conclusions: Our results suggest that discriminability and more informative and clinically plausible attention maps do not always go together. Given the preferences within the healthcare field for enhanced explainability, establishing a balance with discriminability is imperative. Keywords: attention mechanism; clinical plausibility; discriminability; electronic health record; recurrent neural network; longitudinal data Author: Lucía A. Carrasco-Ribelles, Margarita Cabrera-Bean, Jose Llanes-Jurado and Concepción Violán Creator: LaTeX with hyperref Producer: pdfTeX-1.40.25 CreationDate: Tue Jan 7 07:26:22 2025 CET ModDate: Tue Jan 7 07:29:04 2025 CET Custom Metadata: no Metadata Stream: no Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 14 Encrypted: no Page size: 595.276 x 841.89 pts (A4) Page rot: 0 File size: 815165 bytes Optimized: no PDF version: 1.7 name type encoding emb sub uni object ID ------------------------------------ ----------------- ---------------- --- --- --- --------- LGBFIK+URWPalladioL-Roma Type 1 Custom yes yes yes 10 0 VTDPVL+URWPalladioL-Bold Type 1 Custom yes yes yes 16 0 PTRBZS+URWPalladioL-Ital Type 1 Custom yes yes yes 21 0 GIGFZE+CMR10 Type 1 Builtin yes yes yes 53 0 LELDCK+PazoMath-BoldItalic Type 1 Builtin yes yes yes 58 0 ZUZWTT+CMSY10 Type 1 Builtin yes yes yes 63 0 QDTWCG+MSBM10 Type 1 Builtin yes yes yes 68 0 CIDFont+F1 CID TrueType Identity-H yes no yes 75 0 CIDFont+F2 CID TrueType Identity-H yes no yes 86 0 CIDFont+F3 CID TrueType Identity-H yes no yes 97 0 CIDFont+F4 CID TrueType Identity-H yes no yes 108 0 FPRLPS+Calibri CID TrueType Identity-H yes yes yes 132 0 FPRSSH+FranklinGothicBook CID TrueType Identity-H yes yes yes 139 0 Helvetica Type 1 Custom no no no 161 0 Helvetica Type 1 Custom no no no 171 0 NICHLG+PalatinoLinotype TrueType WinAnsi yes yes no 204 0 Jhove (Rel. 1.28.0, 2023-05-18) Date: 2025-07-12 02:08:08 CEST RepresentationInformation: applsci-15-00146-v2.pdf ReportingModule: PDF-hul, Rel. 1.12.4 (2023-03-16) LastModified: 2025-07-11 18:09:34 CEST Size: 815165 Format: PDF Version: 1.7 Status: Well-Formed and valid SignatureMatches: PDF-hul MIMEtype: application/pdf PDFMetadata: Objects: 409 FreeObjects: 1 IncrementalUpdates: 0 DocumentCatalog: PageLayout: SinglePage PageMode: UseNone Outlines: Item: Title: Introduction Destination: section.1 Item: Title: Related Work Destination: section.2 Item: Title: Materials and Methods Destination: section.3 Children: Item: Title: Proposed Architectures Destination: subsection.3.1 Children: Item: Title: Input Destination: subsubsection.3.1.1 Item: Title: Baseline Architecture Without Attention (Approach 0) Destination: subsubsection.3.1.2 Item: Title: Basic Architecture with Time Attention (Approach 1) Destination: subsubsection.3.1.3 Item: Title: Architecture with Domain-Specific Attention (Approach 2) Destination: subsubsection.3.1.4 Item: Title: Architecture with Hierarchical Attention (Approach 3) Destination: subsubsection.3.1.5 Item: Title: Evaluation Destination: subsection.3.2 Children: Item: Title: Study Design, Participants, and Data Source Destination: subsubsection.3.2.1 Item: Title: Model Development and Discriminability Evaluation Destination: subsubsection.3.2.2 Item: Title: Results Destination: section.4 Children: Item: Title: Discriminability Analysis Destination: subsection.4.1 Item: Title: Attention Maps' Analysis Destination: subsection.4.2 Item: Title: Conclusions Destination: section.5 Item: Title: References Destination: section.6 Info: Title: Use of Attention Maps to Enrich Discriminability in Deep Learning Prediction Models Using Longitudinal Data from Electronic Health Records Author: Lucía A. Carrasco-Ribelles, Margarita Cabrera-Bean, Jose Llanes-Jurado and Concepción Violán Subject: Background: In predictive modelling, particularly in fields such as healthcare, the importance of understanding the model's behaviour rivals, if not surpasses, that of discriminability. To this end, attention mechanisms have been included in deep learning models for years. However, when comparing different models, the one with the best discriminability is usually chosen without considering the clinical plausibility of their predictions. Objective: In this work several attention-based deep learning architectures with increasing degrees of complexity were designed and compared aiming to study the balance between discriminability and plausibility with architecture complexity when working with longitudinal data from Electronic Health Records (EHRs). Methods: We developed four deep learning-based architectures with attention mechanisms that were progressively more complex to handle longitudinal data from EHRs. We evaluated their discriminability and resulting attention maps and compared them amongst architectures and different input processing approaches. We trained them on 10 years of data from EHRs from Catalonia (Spain) and evaluated them using a 5-fold cross-validation to predict 1-year all-cause mortality in a subsample of 500,000 people over 65 years of age. Results: Generally, the simplest architectures led to the best overall discriminability, slightly decreasing with complexity by up to 8.7%. However, the attention maps resulting from the simpler architectures were less informative and less clinically plausible compared to those from more complex architectures. Moreover, the latter could give attention weights both in the time and feature domains. Conclusions: Our results suggest that discriminability and more informative and clinically plausible attention maps do not always go together. Given the preferences within the healthcare field for enhanced explainability, establishing a balance with discriminability is imperative. Keywords: attention mechanism; clinical plausibility; discriminability; electronic health record; recurrent neural network; longitudinal data Creator: LaTeX with hyperref Producer: pdfTeX-1.40.25 CreationDate: Tue Jan 07 07:26:22 CET 2025 ModDate: Tue Jan 07 07:29:04 CET 2025 ID: 0x7a783935f18595e3d18545f1ccb9ad8b, 0x7a783935f18595e3d18545f1ccb9ad8b Filters: FilterPipeline: FlateDecode Fonts: Type1: Font: BaseFont: VTDPVL+URWPalladioL-Bold FontSubset: true FirstChar: 3 LastChar: 243 FontDescriptor: FontName: VTDPVL+URWPalladioL-Bold Flags: Symbolic FontBBox: -152, -301, 1000, 935 FontFile: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: QDTWCG+MSBM10 FontSubset: true FirstChar: 82 LastChar: 82 FontDescriptor: FontName: QDTWCG+MSBM10 Flags: Symbolic FontBBox: -55, -420, 2343, 920 FontFile: true ToUnicode: true Font: BaseFont: PTRBZS+URWPalladioL-Ital FontSubset: true FirstChar: 39 LastChar: 121 FontDescriptor: FontName: PTRBZS+URWPalladioL-Ital Flags: Symbolic FontBBox: -170, -305, 1010, 941 FontFile: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: GIGFZE+CMR10 FontSubset: true FirstChar: 40 LastChar: 61 FontDescriptor: FontName: GIGFZE+CMR10 Flags: Symbolic FontBBox: -40, -250, 1009, 750 FontFile: true ToUnicode: true Font: BaseFont: LGBFIK+URWPalladioL-Roma FontSubset: true FirstChar: 2 LastChar: 243 FontDescriptor: FontName: LGBFIK+URWPalladioL-Roma Flags: Symbolic FontBBox: -166, -283, 1021, 943 FontFile: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: LELDCK+PazoMath-BoldItalic FontSubset: true FirstChar: 97 LastChar: 97 FontDescriptor: FontName: LELDCK+PazoMath-BoldItalic Flags: Symbolic FontBBox: -64, -272, 868, 740 FontFile: true ToUnicode: true Font: BaseFont: ZUZWTT+CMSY10 FontSubset: true FirstChar: 2 LastChar: 62 FontDescriptor: FontName: ZUZWTT+CMSY10 Flags: Symbolic FontBBox: -29, -960, 1116, 775 FontFile: true ToUnicode: true Pages: Page: Sequence: 1 Annotations: Annotation: Subtype: Link Rect: 50, 177, 152, 186 Annotation: Subtype: Link Rect: 34, 165, 76, 174 Annotation: Subtype: Link Rect: 36, 75, 152, 84 Annotation: Subtype: Link Rect: 34, 63, 89, 72 Annotation: Subtype: Link Rect: 34, 773, 193, 809 Annotation: Subtype: Link Rect: 515, 773, 560, 809 Annotation: Subtype: Link Rect: 170, 643, 181, 656 Annotation: Subtype: Link Rect: 297, 643, 309, 656 Annotation: Subtype: Link Rect: 401, 643, 412, 656 Annotation: Subtype: Link Rect: 546, 643, 557, 656 Annotation: Subtype: Link Rect: 422, 27, 560, 37 Annotation: Subtype: Link Rect: 34, 356, 87, 375 Page: Sequence: 2 Annotations: Annotation: Subtype: Link Rect: 333, 661, 340, 670 ActionDest: 223 Annotation: Subtype: Link Rect: 341, 661, 348, 670 ActionDest: 223 Annotation: Subtype: Link Rect: 287, 633, 294, 642 ActionDest: 223 Annotation: Subtype: Link Rect: 510, 633, 517, 642 ActionDest: 223 Annotation: Subtype: Link Rect: 297, 591, 304, 600 ActionDest: 223 Annotation: Subtype: Link Rect: 350, 549, 357, 558 ActionDest: 226 Annotation: Subtype: Link Rect: 196, 522, 203, 531 ActionDest: 226 Annotation: Subtype: Link Rect: 337, 521, 344, 531 ActionDest: 226 Annotation: Subtype: Link Rect: 460, 521, 467, 531 ActionDest: 226 Annotation: Subtype: Link Rect: 470, 521, 482, 531 ActionDest: 226 Annotation: Subtype: Link Rect: 211, 508, 223, 517 ActionDest: 226 Annotation: Subtype: Link Rect: 431, 508, 443, 517 ActionDest: 226 Annotation: Subtype: Link Rect: 485, 508, 497, 517 ActionDest: 226 Annotation: Subtype: Link Rect: 294, 480, 306, 489 ActionDest: 226 Annotation: Subtype: Link Rect: 309, 480, 321, 489 ActionDest: 226 Annotation: Subtype: Link Rect: 260, 466, 272, 475 ActionDest: 226 Annotation: Subtype: Link Rect: 383, 452, 390, 461 ActionDest: 223 Annotation: Subtype: Link Rect: 269, 438, 281, 447 ActionDest: 226 Annotation: Subtype: Link Rect: 281, 438, 293, 447 ActionDest: 226 Annotation: Subtype: Link Rect: 294, 438, 306, 447 ActionDest: 226 Annotation: Subtype: Link Rect: 309, 438, 321, 447 ActionDest: 226 Annotation: Subtype: Link Rect: 207, 424, 219, 433 ActionDest: 226 Annotation: Subtype: Link Rect: 219, 424, 231, 433 ActionDest: 226 Annotation: Subtype: Link Rect: 231, 424, 243, 433 ActionDest: 226 Annotation: Subtype: Link Rect: 200, 396, 212, 405 ActionDest: 226 Annotation: Subtype: Link Rect: 293, 354, 305, 363 ActionDest: 226 Annotation: Subtype: Link Rect: 350, 200, 357, 209 ActionDest: 223 Annotation: Subtype: Link Rect: 180, 172, 192, 181 ActionDest: 226 Annotation: Subtype: Link Rect: 192, 172, 204, 181 ActionDest: 226 Annotation: Subtype: Link Rect: 204, 172, 216, 181 ActionDest: 226 Annotation: Subtype: Link Rect: 204, 145, 215, 154 ActionDest: 226 Annotation: Subtype: Link Rect: 283, 144, 295, 154 ActionDest: 226 Annotation: Subtype: Link Rect: 533, 144, 545, 154 ActionDest: 226 Annotation: Subtype: Link Rect: 545, 145, 557, 154 ActionDest: 226 Annotation: Subtype: Link Rect: 544, 117, 556, 126 ActionDest: 226 Page: Sequence: 3 Annotations: Annotation: Subtype: Link Rect: 197, 758, 209, 767 ActionDest: 226 Annotation: Subtype: Link Rect: 446, 744, 458, 754 ActionDest: 229 Annotation: Subtype: Link Rect: 458, 744, 470, 754 ActionDest: 229 Annotation: Subtype: Link Rect: 341, 689, 348, 698 ActionDest: 226 Annotation: Subtype: Link Rect: 343, 675, 355, 684 ActionDest: 229 Annotation: Subtype: Link Rect: 355, 675, 367, 684 ActionDest: 229 Annotation: Subtype: Link Rect: 358, 661, 370, 670 ActionDest: 226 Annotation: Subtype: Link Rect: 299, 647, 311, 656 ActionDest: 229 Annotation: Subtype: Link Rect: 312, 647, 324, 656 ActionDest: 229 Annotation: Subtype: Link Rect: 544, 591, 556, 600 ActionDest: 226 Annotation: Subtype: Link Rect: 342, 550, 354, 559 ActionDest: 226 Annotation: Subtype: Link Rect: 347, 522, 359, 531 ActionDest: 226 Annotation: Subtype: Link Rect: 530, 508, 541, 517 ActionDest: 229 Annotation: Subtype: Link Rect: 544, 508, 556, 517 ActionDest: 229 Annotation: Subtype: Link Rect: 239, 480, 251, 489 ActionDest: 229 Annotation: Subtype: Link Rect: 397, 466, 409, 475 ActionDest: 229 Annotation: Subtype: Link Rect: 257, 438, 269, 447 ActionDest: 229 Annotation: Subtype: Link Rect: 216, 348, 223, 360 ActionDest: 50 Annotation: Subtype: Link Rect: 264, 295, 276, 304 ActionDest: 226 Annotation: Subtype: Link Rect: 492, 268, 504, 277 ActionDest: 226 Page: Sequence: 4 Annotations: Annotation: Subtype: Link Rect: 271, 411, 283, 420 ActionDest: 226 Annotation: Subtype: Link Rect: 329, 394, 344, 406 ActionDest: 124 Annotation: Subtype: Link Rect: 399, 339, 406, 351 ActionDest: 50 Annotation: Subtype: Link Rect: 377, 216, 399, 228 ActionDest: 50 Annotation: Subtype: Link Rect: 215, 175, 222, 186 ActionDest: 50 Annotation: Subtype: Link Rect: 229, 177, 241, 186 ActionDest: 226 Annotation: Subtype: Link Rect: 251, 175, 258, 186 ActionDest: 50 Annotation: Subtype: Link Rect: 265, 177, 277, 186 ActionDest: 229 Annotation: Subtype: Link Rect: 286, 175, 293, 186 ActionDest: 50 Annotation: Subtype: Link Rect: 322, 175, 329, 186 ActionDest: 50 Page: Sequence: 5 Annotations: Annotation: Subtype: Link Rect: 370, 618, 377, 630 ActionDest: 124 Annotation: Subtype: Link Rect: 213, 368, 219, 379 ActionDest: 50 Annotation: Subtype: Link Rect: 413, 275, 435, 287 ActionDest: 124 Annotation: Subtype: Link Rect: 473, 275, 480, 287 ActionDest: 124 Annotation: Subtype: Link Rect: 506, 247, 528, 259 ActionDest: 50 Annotation: Subtype: Link Rect: 203, 153, 215, 162 ActionDest: 229 Page: Sequence: 6 Annotations: Annotation: Subtype: Link Rect: 550, 700, 557, 712 ActionDest: 148 Annotation: Subtype: Link Rect: 544, 619, 556, 628 ActionDest: 229 Annotation: Subtype: Link Rect: 333, 554, 344, 562 ActionDest: 229 Annotation: Subtype: Link Rect: 303, 469, 315, 478 ActionDest: 229 Annotation: Subtype: Link Rect: 265, 367, 277, 376 ActionDest: 229 Annotation: Subtype: Link Rect: 518, 251, 530, 260 ActionDest: 229 Annotation: Subtype: Link Rect: 280, 168, 292, 177 ActionDest: 229 Page: Sequence: 7 Annotations: Annotation: Subtype: Link Rect: 213, 723, 220, 735 ActionDest: 151 Annotation: Subtype: Link Rect: 302, 668, 309, 679 ActionDest: 50 Annotation: Subtype: Link Rect: 476, 654, 482, 665 ActionDest: 50 Annotation: Subtype: Link Rect: 416, 600, 436, 613 ActionDest: 50 Annotation: Subtype: Link Rect: 438, 600, 458, 613 ActionDest: 124 Annotation: Subtype: Link Rect: 297, 547, 304, 559 ActionDest: 50 Annotation: Subtype: Link Rect: 297, 538, 304, 549 ActionDest: 50 Annotation: Subtype: Link Rect: 297, 529, 304, 540 ActionDest: 50 Annotation: Subtype: Link Rect: 297, 520, 304, 531 ActionDest: 50 Annotation: Subtype: Link Rect: 202, 358, 224, 370 ActionDest: 124 Annotation: Subtype: Link Rect: 280, 261, 302, 273 ActionDest: 124 Annotation: Subtype: Link Rect: 521, 109, 528, 121 ActionDest: 154 Page: Sequence: 8 Annotations: Annotation: Subtype: Link Rect: 220, 304, 227, 316 ActionDest: 164 Annotation: Subtype: Link Rect: 246, 304, 253, 316 ActionDest: 178 Page: Sequence: 9 Annotations: Annotation: Subtype: Link Rect: 521, 233, 532, 241 ActionDest: 229 Page: Sequence: 10 Annotations: Annotation: Subtype: Link Rect: 521, 263, 532, 271 ActionDest: 229 Page: Sequence: 11 Annotations: Annotation: Subtype: Link Rect: 244, 341, 256, 350 ActionDest: 229 Annotation: Subtype: Link Rect: 536, 201, 560, 212 Annotation: Subtype: Link Rect: 165, 188, 380, 199 Page: Sequence: 12 Annotations: Annotation: Subtype: Link Rect: 356, 530, 560, 541 Annotation: Subtype: Link Rect: 165, 517, 176, 528 Annotation: Subtype: Link Rect: 168, 504, 428, 515 Annotation: Subtype: Link Rect: 59, 204, 96, 214 Annotation: Subtype: Link Rect: 102, 204, 138, 214 Annotation: Subtype: Link Rect: 170, 179, 208, 189 Annotation: Subtype: Link Rect: 214, 179, 250, 189 Annotation: Subtype: Link Rect: 187, 153, 224, 163 Annotation: Subtype: Link Rect: 230, 153, 266, 163 Annotation: Subtype: Link Rect: 365, 114, 402, 125 Annotation: Subtype: Link Rect: 140, 77, 177, 87 Page: Sequence: 13 Annotations: Annotation: Subtype: Link Rect: 59, 731, 96, 742 Annotation: Subtype: Link Rect: 153, 693, 191, 703 Annotation: Subtype: Link Rect: 237, 552, 274, 563 Annotation: Subtype: Link Rect: 366, 488, 404, 499 Annotation: Subtype: Link Rect: 314, 450, 352, 461 Annotation: Subtype: Link Rect: 390, 386, 426, 397 Annotation: Subtype: Link Rect: 340, 347, 377, 358 Annotation: Subtype: Link Rect: 300, 322, 337, 333 Annotation: Subtype: Link Rect: 233, 296, 270, 307 Annotation: Subtype: Link Rect: 386, 271, 424, 282 Annotation: Subtype: Link Rect: 316, 245, 354, 256 Annotation: Subtype: Link Rect: 266, 207, 304, 218 Annotation: Subtype: Link Rect: 271, 169, 308, 180 Annotation: Subtype: Link Rect: 132, 143, 169, 154 Annotation: Subtype: Link Rect: 299, 118, 337, 129 Annotation: Subtype: Link Rect: 158, 79, 196, 90 Page: Sequence: 14 Annotations: Annotation: Subtype: Link Rect: 157, 744, 194, 754 Annotation: Subtype: Link Rect: 200, 744, 237, 754 Annotation: Subtype: Link Rect: 326, 718, 363, 729 Annotation: Subtype: Link Rect: 370, 718, 406, 729 Annotation: Subtype: Link Rect: 118, 680, 155, 690 Annotation: Subtype: Link Rect: 271, 654, 308, 665 Annotation: Subtype: Link Rect: 353, 628, 390, 639 Annotation: Subtype: Link Rect: 435, 590, 473, 601 Annotation: Subtype: Link Rect: 465, 526, 502, 537 Annotation: Subtype: Link Rect: 477, 513, 514, 524 Annotation: Subtype: Link Rect: 312, 475, 350, 486 Annotation: Subtype: Link Rect: 59, 438, 96, 448 Annotation: Subtype: Link Rect: 102, 438, 138, 448 Annotation: Subtype: Link Rect: 295, 411, 560, 422 Annotation: Subtype: Link Rect: 426, 360, 463, 371 Annotation: Subtype: Link Rect: 470, 360, 506, 371 Annotation: Subtype: Link Rect: 394, 336, 431, 346 Annotation: Subtype: Link Rect: 438, 336, 474, 346 Annotation: Subtype: Link Rect: 186, 296, 223, 307 Checksum: 0126da7b Type: CRC32 Checksum: 29b2debe6b278b9581d3e5f54ef772ed Type: MD5 Checksum: 85e63dede7ded7704af2422be551c090dcae9c88 Type: SHA-1 Checksum: 656ceea97e98e6bd29958c6f153e447af2a7a2bbfe3590c581c25ae55cfe93aa Type: SHA-256