a5ee8d748b7bdfb9e6e43c152af2cf4a appsci_a2022v12n5p2298.pdf 2c143701a524bdd40c76472e10a7bee7a6d83521 appsci_a2022v12n5p2298.pdf 4c0e7b83467d23b2b89efc5df726a971e7632614698303fbdd1e4bb04e395d2e appsci_a2022v12n5p2298.pdf Title: Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals Subject: The commercial flightdeck is a naturally multi-tasking work environment, one in which interruptions are frequent come in various forms, contributing in many cases to aviation incident reports. Automatic characterization of pilots' workloads is essential to preventing these kind of incidents. In addition, minimizing the physiological sensor network as much as possible remains both a challenge and a requirement. Electroencephalogram (EEG) signals have shown high correlations with specific cognitive and mental states, such as workload. However, there is not enough evidence in the literature to validate how well models generalize in cases of new subjects performing tasks with workloads similar to the ones included during the model's training. In this paper, we propose a convolutional neural network to classify EEG features across different mental workloads in a continuous performance task test that partly measures working memory and working memory capacity. Our model is valid at the general population level and it is able to transfer task learning to pilot mental workload recognition in a simulated operational environment. Keywords: cognitive states; mental workload; EEG analysis; neural networks; multimodal data fusion Author: Aura Hernández-Sabaté, José Yauri, Pau Folch, Miquel Àngel Piera, Carles Sánchez and Debora Gil Creator: LaTeX with hyperref Producer: pdfTeX-1.40.21 CreationDate: Wed Feb 23 10:16:30 2022 CET ModDate: Thu Feb 24 02:30:06 2022 CET Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 14 Encrypted: no Page size: 595.276 x 841.89 pts (A4) Page rot: 0 File size: 1119336 bytes Optimized: no PDF version: 1.7 name type encoding emb sub uni object ID ------------------------------------ ----------------- ---------------- --- --- --- --------- BNWVQK+URWPalladioL-Bold Type 1 Custom yes yes no 10 0 LEDLEA+URWPalladioL-Roma Type 1 Custom yes yes no 15 0 QXWBBO+URWPalladioL-Ital Type 1 Custom yes yes no 19 0 MCBCCI+SourceSansRoman-Regular Type 1C Custom yes yes no 27 0 MCBCCM+SourceSansRoman.613wght Type 1C Custom yes yes no 31 0 MCBCCO+SourceSansRoman-Bold Type 1C Custom yes yes no 35 0 CRVMKC+TimesNewRomanPS-BoldItalicMT Type 1C WinAnsi yes yes no 47 0 NJAOJC+CMSY10 Type 1 Builtin yes yes no 69 0 GZSQLS+PazoMath-Italic Type 1 Builtin yes yes no 89 0 RFLZJB+CMR10 Type 1 Builtin yes yes no 101 0 Jhove (Rel. 1.22.1, 2019-04-17) Date: 2022-06-16 02:27:58 CEST RepresentationInformation: appsci_a2022v12n5p2298.pdf ReportingModule: PDF-hul, Rel. 1.12.1 (2019-04-17) LastModified: 2022-03-03 12:57:21 CET Size: 1119336 Format: PDF Version: 1.7 Status: Well-Formed and valid SignatureMatches: PDF-hul MIMEtype: application/pdf PDFMetadata: Objects: 259 FreeObjects: 1 IncrementalUpdates: 0 DocumentCatalog: PageLayout: SinglePage PageMode: UseNone Outlines: Item: Title: Introduction Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@46daef40 Item: Title: Related Work Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@12f41634 Item: Title: Data Annotation and Ground Truth Generation Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@13c27452 Children: Item: Title: Dual N-Back Test Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@262b2c86 Item: Title: Flight Simulations Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@371a67ec Item: Title: Workload Recognition Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@5ed828d Children: Item: Title: Extracting Input Data from EEG Signals Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@50d0686 Item: Title: Network Architectures Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@7a3d45bd Item: Title: Experimental Design Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@1e7c7811 Children: Item: Title: Training and Validation Using N-Back-Test Data Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@77ec78b9 Item: Title: Task Transfer Verification Using Flight Simulator Data Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@1a3869f4 Item: Title: Results Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@a38d7a3 Children: Item: Title: Training and Validation Using N-Back-Test Data Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@77f99a05 Item: Title: Task Transfer Verification Using Flight Simulator Data Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@63440df3 Item: Title: Conclusions Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@3aeaafa6 Item: Title: References Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@76a3e297 Info: Title: Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals Author: Aura Hernández-Sabaté, José Yauri, Pau Folch, Miquel Àngel Piera, Carles Sánchez and Debora Gil Subject: The commercial flightdeck is a naturally multi-tasking work environment, one in which interruptions are frequent come in various forms, contributing in many cases to aviation incident reports. Automatic characterization of pilots' workloads is essential to preventing these kind of incidents. In addition, minimizing the physiological sensor network as much as possible remains both a challenge and a requirement. Electroencephalogram (EEG) signals have shown high correlations with specific cognitive and mental states, such as workload. However, there is not enough evidence in the literature to validate how well models generalize in cases of new subjects performing tasks with workloads similar to the ones included during the model's training. In this paper, we propose a convolutional neural network to classify EEG features across different mental workloads in a continuous performance task test that partly measures working memory and working memory capacity. Our model is valid at the general population level and it is able to transfer task learning to pilot mental workload recognition in a simulated operational environment. Keywords: cognitive states; mental workload; EEG analysis; neural networks; multimodal data fusion Creator: LaTeX with hyperref Producer: pdfTeX-1.40.21 CreationDate: Wed Feb 23 10:16:30 CET 2022 ModDate: Thu Feb 24 02:30:06 CET 2022 ID: 0x1894f9e436ebe6634a02dce18b17d234, 0x1894f9e436ebe6634a02dce18b17d234 Filters: FilterPipeline: FlateDecode FilterPipeline: DCTDecode Images: Image: NisoImageMetadata: FormatName: image/jpg CompressionScheme: JPEG ImageWidth: 900 ImageHeight: 700 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Image: NisoImageMetadata: FormatName: image/png ImageWidth: 376 ImageHeight: 287 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 1506 ImageHeight: 513 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 2364 ImageHeight: 720 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 810 ImageHeight: 599 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 806 ImageHeight: 613 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 806 ImageHeight: 646 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Image: NisoImageMetadata: FormatName: image/png ImageWidth: 810 ImageHeight: 631 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Filter: FlateDecode Fonts: Type1: Font: BaseFont: QXWBBO+URWPalladioL-Ital FontSubset: true FirstChar: 2 LastChar: 121 FontDescriptor: FontName: QXWBBO+URWPalladioL-Ital Flags: Symbolic FontBBox: -170, -305, 1010, 941 FontFile: true EncodingDictionary: Differences: true Font: BaseFont: NJAOJC+CMSY10 FontSubset: true FirstChar: 2 LastChar: 6 FontDescriptor: FontName: NJAOJC+CMSY10 Flags: Symbolic FontBBox: -29, -960, 1116, 775 FontFile: true Font: BaseFont: RFLZJB+CMR10 FontSubset: true FirstChar: 43 LastChar: 61 FontDescriptor: FontName: RFLZJB+CMR10 Flags: Symbolic FontBBox: -40, -250, 1009, 750 FontFile: true Font: BaseFont: GZSQLS+PazoMath-Italic FontSubset: true FirstChar: 97 LastChar: 113 FontDescriptor: FontName: GZSQLS+PazoMath-Italic Flags: Symbolic FontBBox: -70, -277, 902, 733 FontFile: true Font: BaseFont: BNWVQK+URWPalladioL-Bold FontSubset: true FirstChar: 3 LastChar: 233 FontDescriptor: FontName: BNWVQK+URWPalladioL-Bold Flags: Symbolic FontBBox: -152, -301, 1000, 935 FontFile: true EncodingDictionary: Differences: true Font: BaseFont: LEDLEA+URWPalladioL-Roma FontSubset: true FirstChar: 2 LastChar: 243 FontDescriptor: FontName: LEDLEA+URWPalladioL-Roma Flags: Symbolic FontBBox: -166, -283, 1021, 943 FontFile: true EncodingDictionary: Differences: true Pages: Page: Sequence: 1 Annotations: Annotation: Subtype: Link Rect: 34, 323, 149, 332 Annotation: Subtype: Link Rect: 34, 164, 96, 186 Annotation: Subtype: Link Rect: 126, 90, 154, 99 Annotation: Subtype: Link Rect: 34, 78, 147, 87 Annotation: Subtype: Link Rect: 34, 756, 175, 792 Annotation: Subtype: Link Rect: 515, 756, 560, 792 Annotation: Subtype: Link Rect: 162, 658, 173, 671 Annotation: Subtype: Link Rect: 234, 658, 245, 671 Annotation: Subtype: Link Rect: 406, 658, 417, 671 Annotation: Subtype: Link Rect: 501, 658, 512, 671 Annotation: Subtype: Link Rect: 287, 247, 294, 255 ActionDest: 123 Annotation: Subtype: Link Rect: 376, 221, 383, 230 ActionDest: 123 Annotation: Subtype: Link Rect: 461, 221, 467, 230 ActionDest: 123 Annotation: Subtype: Link Rect: 213, 209, 220, 218 ActionDest: 123 Annotation: Subtype: Link Rect: 397, 209, 404, 218 ActionDest: 123 Annotation: Subtype: Link Rect: 305, 108, 312, 117 ActionDest: 123 Annotation: Subtype: Link Rect: 338, 83, 345, 92 ActionDest: 123 Annotation: Subtype: Link Rect: 120, 17, 258, 28 Annotation: Subtype: Link Rect: 411, 17, 560, 28 Annotation: Subtype: Link Rect: 34, 396, 86, 415 Page: Sequence: 2 Annotations: Annotation: Subtype: Link Rect: 247, 723, 254, 732 ActionDest: 123 Annotation: Subtype: Link Rect: 360, 698, 367, 707 ActionDest: 123 Annotation: Subtype: Link Rect: 368, 698, 380, 707 ActionDest: 123 Annotation: Subtype: Link Rect: 529, 648, 541, 657 ActionDest: 123 Annotation: Subtype: Link Rect: 544, 647, 556, 657 ActionDest: 123 Annotation: Subtype: Link Rect: 455, 570, 462, 582 ActionDest: 66 Annotation: Subtype: Link Rect: 307, 557, 314, 569 ActionDest: 76 Annotation: Subtype: Link Rect: 513, 557, 520, 569 ActionDest: 79 Annotation: Subtype: Link Rect: 494, 544, 501, 556 ActionDest: 98 Annotation: Subtype: Link Rect: 298, 532, 305, 544 ActionDest: 105 Annotation: Subtype: Link Rect: 553, 532, 560, 544 ActionDest: 120 Annotation: Subtype: Link Rect: 217, 471, 224, 481 ActionDest: 123 Annotation: Subtype: Link Rect: 224, 472, 236, 481 ActionDest: 123 Annotation: Subtype: Link Rect: 237, 471, 249, 481 ActionDest: 123 Annotation: Subtype: Link Rect: 318, 434, 330, 443 ActionDest: 123 Annotation: Subtype: Link Rect: 250, 409, 262, 418 ActionDest: 123 Annotation: Subtype: Link Rect: 262, 409, 274, 418 ActionDest: 123 Annotation: Subtype: Link Rect: 416, 371, 428, 380 ActionDest: 123 Annotation: Subtype: Link Rect: 428, 371, 440, 380 ActionDest: 123 Annotation: Subtype: Link Rect: 529, 334, 541, 342 ActionDest: 123 Annotation: Subtype: Link Rect: 544, 333, 556, 342 ActionDest: 123 Annotation: Subtype: Link Rect: 480, 258, 492, 267 ActionDest: 123 Annotation: Subtype: Link Rect: 371, 245, 383, 255 ActionDest: 123 Annotation: Subtype: Link Rect: 223, 220, 230, 229 ActionDest: 123 Annotation: Subtype: Link Rect: 231, 221, 243, 229 ActionDest: 123 Annotation: Subtype: Link Rect: 243, 221, 255, 229 ActionDest: 123 Annotation: Subtype: Link Rect: 473, 195, 485, 204 ActionDest: 123 Annotation: Subtype: Link Rect: 269, 158, 281, 167 ActionDest: 123 Annotation: Subtype: Link Rect: 495, 145, 507, 154 ActionDest: 123 Annotation: Subtype: Link Rect: 437, 95, 449, 104 ActionDest: 123 Annotation: Subtype: Link Rect: 449, 95, 461, 104 ActionDest: 123 Annotation: Subtype: Link Rect: 461, 95, 473, 104 ActionDest: 126 Page: Sequence: 3 Annotations: Annotation: Subtype: Link Rect: 498, 735, 510, 744 ActionDest: 126 Annotation: Subtype: Link Rect: 195, 723, 206, 732 ActionDest: 126 Annotation: Subtype: Link Rect: 290, 723, 301, 732 ActionDest: 126 Annotation: Subtype: Link Rect: 368, 723, 380, 732 ActionDest: 126 Annotation: Subtype: Link Rect: 545, 723, 556, 732 ActionDest: 126 Annotation: Subtype: Link Rect: 225, 685, 232, 694 ActionDest: 123 Annotation: Subtype: Link Rect: 287, 647, 299, 657 ActionDest: 126 Annotation: Subtype: Link Rect: 202, 409, 213, 418 ActionDest: 126 Annotation: Subtype: Link Rect: 393, 353, 405, 363 ActionDest: 126 Annotation: Subtype: Link Rect: 336, 228, 348, 237 ActionDest: 126 Page: Sequence: 4 Annotations: Annotation: Subtype: Link Rect: 469, 295, 481, 304 ActionDest: 126 Page: Sequence: 5 Annotations: Annotation: Subtype: Link Rect: 420, 519, 432, 528 ActionDest: 126 Annotation: Subtype: Link Rect: 217, 378, 224, 390 ActionDest: 79 Page: Sequence: 6 Annotations: Annotation: Subtype: Link Rect: 397, 685, 409, 694 ActionDest: 126 Annotation: Subtype: Link Rect: 307, 635, 319, 644 ActionDest: 126 Annotation: Subtype: Link Rect: 247, 572, 259, 581 ActionDest: 123 Annotation: Subtype: Link Rect: 505, 572, 517, 581 ActionDest: 126 Annotation: Subtype: Link Rect: 275, 346, 287, 355 ActionDest: 126 Annotation: Subtype: Link Rect: 300, 217, 307, 230 ActionDest: 93 Annotation: Subtype: Link Rect: 326, 217, 333, 230 ActionDest: 93 Page: Sequence: 7 Page: Sequence: 8 Annotations: Annotation: Subtype: Link Rect: 297, 431, 309, 440 ActionDest: 126 Annotation: Subtype: Link Rect: 440, 210, 447, 222 ActionDest: 76 Page: Sequence: 9 Annotations: Annotation: Subtype: Link Rect: 217, 680, 224, 691 ActionDest: 105 Annotation: Subtype: Link Rect: 246, 680, 253, 691 ActionDest: 105 Annotation: Subtype: Link Rect: 275, 187, 282, 199 ActionDest: 108 Annotation: Subtype: Link Rect: 303, 187, 310, 199 ActionDest: 108 Annotation: Subtype: Link Rect: 222, 124, 229, 136 ActionDest: 113 Annotation: Subtype: Link Rect: 249, 124, 257, 136 ActionDest: 113 Page: Sequence: 10 Page: Sequence: 11 Page: Sequence: 12 Annotations: Annotation: Subtype: Link Rect: 411, 431, 423, 440 ActionDest: 126 Annotation: Subtype: Link Rect: 225, 83, 427, 94 Page: Sequence: 13 Annotations: Annotation: Subtype: Link Rect: 470, 632, 507, 643 Annotation: Subtype: Link Rect: 170, 609, 208, 620 Annotation: Subtype: Link Rect: 519, 597, 557, 608 Annotation: Subtype: Link Rect: 413, 586, 450, 597 Annotation: Subtype: Link Rect: 520, 563, 557, 574 Annotation: Subtype: Link Rect: 151, 506, 188, 516 Annotation: Subtype: Link Rect: 279, 482, 317, 493 Annotation: Subtype: Link Rect: 183, 459, 220, 470 Annotation: Subtype: Link Rect: 412, 436, 450, 447 Annotation: Subtype: Link Rect: 383, 414, 421, 424 Annotation: Subtype: Link Rect: 95, 345, 133, 355 Annotation: Subtype: Link Rect: 501, 298, 539, 309 Annotation: Subtype: Link Rect: 467, 264, 504, 275 Annotation: Subtype: Link Rect: 403, 241, 441, 252 Annotation: Subtype: Link Rect: 507, 149, 545, 160 Annotation: Subtype: Link Rect: 230, 69, 268, 79 Page: Sequence: 14 Annotations: Annotation: Subtype: Link Rect: 333, 722, 371, 733 Annotation: Subtype: Link Rect: 357, 699, 395, 710 Annotation: Subtype: Link Rect: 182, 653, 219, 664 Annotation: Subtype: Link Rect: 418, 629, 455, 640 Annotation: Subtype: Link Rect: 280, 573, 318, 583 Annotation: Subtype: Link Rect: 205, 527, 242, 537 Annotation: Subtype: Link Rect: 201, 491, 239, 502 Annotation: Subtype: Link Rect: 254, 445, 291, 456 Annotation: Subtype: Link Rect: 335, 411, 372, 422 Annotation: Subtype: Link Rect: 520, 376, 557, 387 Annotation: Subtype: Link Rect: 286, 330, 323, 341 Checksum: 4dc2a4ab Type: CRC32 Checksum: a5ee8d748b7bdfb9e6e43c152af2cf4a Type: MD5 Checksum: 2c143701a524bdd40c76472e10a7bee7a6d83521 Type: SHA-1