66243a053d225b44b999dbf494b903b5 appsci_a2021v12n2p610.pdf 0d4e4e66269999f6abbb086b0af0c3c0a6f1c564 appsci_a2021v12n2p610.pdf 663f432732cb4f9690a0c61a49d0be88c412c6f7897f010cb12181aa19110203 appsci_a2021v12n2p610.pdf Title: Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning Subject: Safety is the primary concern when it comes to air traffic. In-flight safety between Unmanned Aircraft Vehicles (UAVs) is ensured through pairwise separation minima, utilizing conflict detection and resolution methods. Existing methods mainly deal with pairwise conflicts, however, due to an expected increase in traffic density, encounters with more than two UAVs are likely to happen. In this paper, we model multi-UAV conflict resolution as a multiagent reinforcement learning problem. We implement an algorithm based on graph neural networks where cooperative agents can communicate to jointly generate resolution maneuvers. The model is evaluated in scenarios with 3 and 4 present agents. Results show that agents are able to successfully solve the multi-UAV conflicts through a cooperative strategy. Keywords: UTM; UAS; machine learning; artificial intelligence; multi-UAS cooperative control; multiagent reinforcement learning Author: Ralvi Isufaj, Marsel Omeri and Miquel Angel Piera Creator: LaTeX with hyperref Producer: pdfTeX-1.40.21 CreationDate: Mon Jan 10 08:02:53 2022 CET ModDate: Mon Jan 10 10:06:34 2022 CET Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 15 Encrypted: no Page size: 595.276 x 841.89 pts (A4) Page rot: 0 File size: 531758 bytes Optimized: no PDF version: 1.7 name type encoding emb sub uni object ID ------------------------------------ ----------------- ---------------- --- --- --- --------- YNXEJU+URWPalladioL-Bold Type 1 Custom yes yes no 10 0 SCWGZG+URWPalladioL-Roma Type 1 Custom yes yes no 15 0 ZDINCL+URWPalladioL-Ital Type 1 Custom yes yes no 19 0 MCBCCI+SourceSansRoman-Regular Type 1C Custom yes yes no 27 0 MCBCCM+SourceSansRoman.613wght Type 1C Custom yes yes no 31 0 MCBCCO+SourceSansRoman-Bold Type 1C Custom yes yes no 35 0 CRVMKC+TimesNewRomanPS-BoldItalicMT Type 1C WinAnsi yes yes no 47 0 SOSTRQ+CMR10 Type 1 Builtin yes yes no 72 0 SUGRDO+PazoMath-Italic Type 1 Builtin yes yes no 76 0 YDRXXO+CMSY10 Type 1 Builtin yes yes no 80 0 EVXUBX+PazoMath Type 1 Builtin yes yes no 84 0 MZKJYP+MSBM10 Type 1 Builtin yes yes no 88 0 LHRDPA+CMEX10 Type 1 Builtin yes yes no 104 0 RLYTRW+CMMI10 Type 1 Builtin yes yes no 108 0 Jhove (Rel. 1.22.1, 2019-04-17) Date: 2023-02-22 02:35:08 CET RepresentationInformation: appsci_a2021v12n2p610.pdf ReportingModule: PDF-hul, Rel. 1.12.1 (2019-04-17) LastModified: 2022-02-23 12:53:53 CET Size: 531758 Format: PDF Version: 1.7 Status: Well-Formed and valid SignatureMatches: PDF-hul MIMEtype: application/pdf PDFMetadata: Objects: 318 FreeObjects: 11 IncrementalUpdates: 0 DocumentCatalog: PageLayout: SinglePage PageMode: UseNone Outlines: Item: Title: Introduction Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@44c8afef Item: Title: Related Work Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@7b69c6ba Item: Title: Theoretical Background Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@46daef40 Children: Item: Title: Reinforcement Learning Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@12f41634 Item: Title: Multiagent Reinforcement Learning Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@13c27452 Item: Title: Graph Convolutional Reinforcement Learning Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@262b2c86 Item: Title: Experimental Setup Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@371a67ec Children: Item: Title: Compound Conflicts Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@5ed828d Item: Title: Traffic as a Graph Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@50d0686 Item: Title: Training Environment Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@7a3d45bd Children: Item: Title: State Space Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@1e7c7811 Item: Title: Action Space Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@77ec78b9 Item: Title: Reward Function Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@1a3869f4 Item: Title: Simulation Environment Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@a38d7a3 Item: Title: Data Generation Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@77f99a05 Item: Title: Simulation Results Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@63440df3 Children: Item: Title: Conflict Resolution Performance Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@3aeaafa6 Item: Title: Agent Behavior Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@76a3e297 Item: Title: Conclusions and Future Work Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@4d3167f4 Item: Title: References Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@ed9d034 Info: Title: Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning Author: Ralvi Isufaj, Marsel Omeri and Miquel Angel Piera Subject: Safety is the primary concern when it comes to air traffic. In-flight safety between Unmanned Aircraft Vehicles (UAVs) is ensured through pairwise separation minima, utilizing conflict detection and resolution methods. Existing methods mainly deal with pairwise conflicts, however, due to an expected increase in traffic density, encounters with more than two UAVs are likely to happen. In this paper, we model multi-UAV conflict resolution as a multiagent reinforcement learning problem. We implement an algorithm based on graph neural networks where cooperative agents can communicate to jointly generate resolution maneuvers. The model is evaluated in scenarios with 3 and 4 present agents. Results show that agents are able to successfully solve the multi-UAV conflicts through a cooperative strategy. Keywords: UTM; UAS; machine learning; artificial intelligence; multi-UAS cooperative control; multiagent reinforcement learning Creator: LaTeX with hyperref Producer: pdfTeX-1.40.21 CreationDate: Mon Jan 10 08:02:53 CET 2022 ModDate: Mon Jan 10 10:06:34 CET 2022 ID: 0x97da67437b88dcd6143dc2911b37411a, 0x97da67437b88dcd6143dc2911b37411a Filters: FilterPipeline: FlateDecode Fonts: Type1: Font: BaseFont: YDRXXO+CMSY10 FontSubset: true FirstChar: 0 LastChar: 106 FontDescriptor: FontName: YDRXXO+CMSY10 Flags: Symbolic FontBBox: -29, -960, 1116, 775 FontFile: true Font: BaseFont: ZDINCL+URWPalladioL-Ital FontSubset: true FirstChar: 2 LastChar: 150 FontDescriptor: FontName: ZDINCL+URWPalladioL-Ital Flags: Symbolic FontBBox: -170, -305, 1010, 941 FontFile: true EncodingDictionary: Differences: true Font: BaseFont: EVXUBX+PazoMath FontSubset: true FirstChar: 165 LastChar: 229 FontDescriptor: FontName: EVXUBX+PazoMath Flags: Symbolic FontBBox: -40, -283, 878, 946 FontFile: true Font: BaseFont: SOSTRQ+CMR10 FontSubset: true FirstChar: 40 LastChar: 93 FontDescriptor: FontName: SOSTRQ+CMR10 Flags: Symbolic FontBBox: -40, -250, 1009, 750 FontFile: true Font: BaseFont: MZKJYP+MSBM10 FontSubset: true FirstChar: 66 LastChar: 82 FontDescriptor: FontName: MZKJYP+MSBM10 Flags: Symbolic FontBBox: -55, -420, 2343, 920 FontFile: true Font: BaseFont: LHRDPA+CMEX10 FontSubset: true FirstChar: 40 LastChar: 40 FontDescriptor: FontName: LHRDPA+CMEX10 Flags: Symbolic FontBBox: -24, -2960, 1454, 772 FontFile: true Font: BaseFont: YNXEJU+URWPalladioL-Bold FontSubset: true FirstChar: 3 LastChar: 121 FontDescriptor: FontName: YNXEJU+URWPalladioL-Bold Flags: Symbolic FontBBox: -152, -301, 1000, 935 FontFile: true EncodingDictionary: Differences: true Font: BaseFont: SUGRDO+PazoMath-Italic FontSubset: true FirstChar: 97 LastChar: 119 FontDescriptor: FontName: SUGRDO+PazoMath-Italic Flags: Symbolic FontBBox: -70, -277, 902, 733 FontFile: true Font: BaseFont: RLYTRW+CMMI10 FontSubset: true FirstChar: 60 LastChar: 60 FontDescriptor: FontName: RLYTRW+CMMI10 Flags: Symbolic FontBBox: -32, -250, 1048, 750 FontFile: true Font: BaseFont: SCWGZG+URWPalladioL-Roma FontSubset: true FirstChar: 2 LastChar: 241 FontDescriptor: FontName: SCWGZG+URWPalladioL-Roma Flags: Symbolic FontBBox: -166, -283, 1021, 943 FontFile: true EncodingDictionary: Differences: true Pages: Page: Sequence: 1 Annotations: Annotation: Subtype: Link Rect: 34, 307, 149, 316 Annotation: Subtype: Link Rect: 34, 159, 96, 181 Annotation: Subtype: Link Rect: 126, 86, 154, 95 Annotation: Subtype: Link Rect: 34, 74, 147, 83 Annotation: Subtype: Link Rect: 34, 756, 175, 792 Annotation: Subtype: Link Rect: 515, 756, 560, 792 Annotation: Subtype: Link Rect: 95, 656, 106, 670 Annotation: Subtype: Link Rect: 174, 656, 186, 670 Annotation: Subtype: Link Rect: 298, 656, 309, 670 Annotation: Subtype: Link Rect: 549, 359, 556, 368 ActionDest: 167 Annotation: Subtype: Link Rect: 231, 247, 238, 255 ActionDest: 167 Annotation: Subtype: Link Rect: 462, 221, 469, 230 ActionDest: 167 Annotation: Subtype: Link Rect: 333, 209, 340, 218 ActionDest: 167 Annotation: Subtype: Link Rect: 531, 171, 538, 180 ActionDest: 167 Annotation: Subtype: Link Rect: 194, 133, 201, 142 ActionDest: 167 Annotation: Subtype: Link Rect: 527, 133, 534, 142 ActionDest: 167 Annotation: Subtype: Link Rect: 535, 133, 542, 142 ActionDest: 167 Annotation: Subtype: Link Rect: 486, 108, 493, 117 ActionDest: 167 Annotation: Subtype: Link Rect: 494, 108, 501, 117 ActionDest: 167 Annotation: Subtype: Link Rect: 117, 13, 254, 23 Annotation: Subtype: Link Rect: 411, 13, 560, 23 Annotation: Subtype: Link Rect: 34, 380, 86, 399 Page: Sequence: 2 Annotations: Annotation: Subtype: Link Rect: 245, 723, 252, 732 ActionDest: 167 Annotation: Subtype: Link Rect: 252, 723, 259, 732 ActionDest: 167 Annotation: Subtype: Link Rect: 338, 685, 350, 694 ActionDest: 167 Annotation: Subtype: Link Rect: 529, 660, 541, 669 ActionDest: 167 Annotation: Subtype: Link Rect: 544, 660, 556, 669 ActionDest: 170 Annotation: Subtype: Link Rect: 343, 635, 355, 644 ActionDest: 170 Annotation: Subtype: Link Rect: 356, 635, 368, 644 ActionDest: 170 Annotation: Subtype: Link Rect: 429, 585, 441, 594 ActionDest: 170 Annotation: Subtype: Link Rect: 553, 572, 560, 581 ActionDest: 66 Annotation: Subtype: Link Rect: 500, 547, 507, 556 ActionDest: 167 Annotation: Subtype: Link Rect: 496, 510, 508, 519 ActionDest: 170 Annotation: Subtype: Link Rect: 212, 271, 219, 283 ActionDest: 66 Annotation: Subtype: Link Rect: 259, 271, 266, 283 ActionDest: 69 Annotation: Subtype: Link Rect: 199, 261, 206, 270 ActionDest: 98 Annotation: Subtype: Link Rect: 199, 246, 206, 258 ActionDest: 115 Annotation: Subtype: Link Rect: 282, 246, 289, 258 ActionDest: 149 Annotation: Subtype: Link Rect: 544, 185, 556, 194 ActionDest: 170 Annotation: Subtype: Link Rect: 261, 160, 273, 169 ActionDest: 170 Annotation: Subtype: Link Rect: 379, 160, 391, 169 ActionDest: 170 Page: Sequence: 3 Annotations: Annotation: Subtype: Link Rect: 227, 736, 239, 744 ActionDest: 170 Annotation: Subtype: Link Rect: 543, 723, 555, 732 ActionDest: 170 Annotation: Subtype: Link Rect: 252, 685, 264, 694 ActionDest: 170 Annotation: Subtype: Link Rect: 252, 673, 263, 682 ActionDest: 170 Annotation: Subtype: Link Rect: 252, 635, 264, 644 ActionDest: 170 Annotation: Subtype: Link Rect: 266, 610, 278, 619 ActionDest: 170 Annotation: Subtype: Link Rect: 333, 585, 345, 594 ActionDest: 170 Annotation: Subtype: Link Rect: 230, 560, 242, 569 ActionDest: 170 Annotation: Subtype: Link Rect: 412, 547, 424, 556 ActionDest: 170 Annotation: Subtype: Link Rect: 421, 384, 433, 393 ActionDest: 170 Annotation: Subtype: Link Rect: 247, 308, 259, 317 ActionDest: 170 Annotation: Subtype: Link Rect: 262, 308, 273, 317 ActionDest: 170 Annotation: Subtype: Link Rect: 291, 258, 303, 267 ActionDest: 170 Annotation: Subtype: Link Rect: 467, 245, 478, 254 ActionDest: 170 Page: Sequence: 4 Annotations: Annotation: Subtype: Link Rect: 472, 673, 484, 682 ActionDest: 170 Annotation: Subtype: Link Rect: 323, 419, 335, 429 ActionDest: 170 Annotation: Subtype: Link Rect: 306, 369, 318, 378 ActionDest: 170 Annotation: Subtype: Link Rect: 364, 357, 376, 366 ActionDest: 170 Annotation: Subtype: Link Rect: 406, 269, 418, 278 ActionDest: 170 Annotation: Subtype: Link Rect: 287, 105, 299, 114 ActionDest: 170 Page: Sequence: 5 Annotations: Annotation: Subtype: Link Rect: 469, 672, 481, 681 ActionDest: 170 Annotation: Subtype: Link Rect: 289, 584, 301, 594 ActionDest: 170 Annotation: Subtype: Link Rect: 486, 584, 498, 594 ActionDest: 170 Annotation: Subtype: Link Rect: 330, 572, 342, 581 ActionDest: 170 Annotation: Subtype: Link Rect: 450, 572, 462, 581 ActionDest: 170 Annotation: Subtype: Link Rect: 199, 534, 211, 543 ActionDest: 170 Annotation: Subtype: Link Rect: 212, 534, 224, 543 ActionDest: 170 Annotation: Subtype: Link Rect: 505, 522, 517, 531 ActionDest: 170 Annotation: Subtype: Link Rect: 381, 296, 393, 305 ActionDest: 173 Annotation: Subtype: Link Rect: 335, 271, 347, 280 ActionDest: 170 Annotation: Subtype: Link Rect: 347, 271, 359, 280 ActionDest: 173 Page: Sequence: 6 Annotations: Annotation: Subtype: Link Rect: 215, 403, 226, 412 ActionDest: 173 Annotation: Subtype: Link Rect: 415, 328, 427, 337 ActionDest: 173 Page: Sequence: 7 Annotations: Annotation: Subtype: Link Rect: 543, 698, 555, 707 ActionDest: 170 Annotation: Subtype: Link Rect: 521, 509, 532, 518 ActionDest: 170 Annotation: Subtype: Link Rect: 455, 97, 467, 105 ActionDest: 173 Page: Sequence: 8 Annotations: Annotation: Subtype: Link Rect: 235, 682, 242, 694 ActionDest: 112 Annotation: Subtype: Link Rect: 531, 522, 543, 531 ActionDest: 173 Annotation: Subtype: Link Rect: 544, 522, 556, 531 ActionDest: 173 Page: Sequence: 9 Annotations: Annotation: Subtype: Link Rect: 303, 660, 315, 669 ActionDest: 170 Annotation: Subtype: Link Rect: 315, 660, 327, 669 ActionDest: 173 Annotation: Subtype: Link Rect: 328, 660, 340, 669 ActionDest: 173 Annotation: Subtype: Link Rect: 344, 606, 505, 618 Annotation: Subtype: Link Rect: 368, 543, 488, 555 Annotation: Subtype: Link Rect: 217, 518, 224, 530 ActionDest: 115 Page: Sequence: 10 Annotations: Annotation: Subtype: Link Rect: 229, 733, 236, 745 ActionDest: 126 Annotation: Subtype: Link Rect: 385, 361, 392, 373 ActionDest: 115 Annotation: Subtype: Link Rect: 212, 323, 219, 335 ActionDest: 126 Annotation: Subtype: Link Rect: 552, 166, 559, 177 ActionDest: 137 Page: Sequence: 11 Annotations: Annotation: Subtype: Link Rect: 257, 407, 264, 419 ActionDest: 149 Annotation: Subtype: Link Rect: 285, 407, 292, 419 ActionDest: 149 Page: Sequence: 12 Page: Sequence: 13 Annotations: Annotation: Subtype: Link Rect: 499, 393, 560, 404 Annotation: Subtype: Link Rect: 165, 382, 249, 393 Annotation: Subtype: Link Rect: 59, 282, 96, 292 Annotation: Subtype: Link Rect: 357, 246, 394, 257 Annotation: Subtype: Link Rect: 440, 223, 560, 234 Annotation: Subtype: Link Rect: 56, 212, 294, 223 Annotation: Subtype: Link Rect: 274, 200, 560, 211 Annotation: Subtype: Link Rect: 56, 189, 107, 200 Annotation: Subtype: Link Rect: 229, 166, 266, 177 Annotation: Subtype: Link Rect: 277, 131, 314, 142 Annotation: Subtype: Link Rect: 509, 120, 560, 131 Annotation: Subtype: Link Rect: 56, 108, 441, 119 Annotation: Subtype: Link Rect: 461, 62, 498, 73 Page: Sequence: 14 Annotations: Annotation: Subtype: Link Rect: 287, 687, 324, 698 Annotation: Subtype: Link Rect: 334, 664, 372, 675 Annotation: Subtype: Link Rect: 478, 572, 516, 583 Annotation: Subtype: Link Rect: 104, 550, 141, 560 Annotation: Subtype: Link Rect: 104, 527, 142, 537 Annotation: Subtype: Link Rect: 215, 480, 252, 491 Annotation: Subtype: Link Rect: 151, 239, 188, 249 Annotation: Subtype: Link Rect: 201, 215, 239, 226 Annotation: Subtype: Link Rect: 449, 204, 486, 215 Annotation: Subtype: Link Rect: 189, 100, 226, 111 Page: Sequence: 15 Annotations: Annotation: Subtype: Link Rect: 213, 629, 250, 640 Annotation: Subtype: Link Rect: 345, 583, 382, 594 Checksum: fedf79f3 Type: CRC32 Checksum: 66243a053d225b44b999dbf494b903b5 Type: MD5 Checksum: 0d4e4e66269999f6abbb086b0af0c3c0a6f1c564 Type: SHA-1