149a3d0982c7cb6b3285e2da29459e59 pmc_9487607.pdf 032701195b795b32fc2805dd78edb0275bf8f614 pmc_9487607.pdf d7efd6d4dc8cd3c10dd14f43f056608540b7a4dd87b1abd6c28a98451f46701b pmc_9487607.pdf Title: Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data Subject: DOI: 10.1093/bib/bbac343, Briefings in Bioinformatics, 23(5), 5, 0 20xx. Abstract: Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at . Keywords: Author: Creator: LaTeX with hyperref package Producer: Acrobat Distiller 22.0 (Windows) CreationDate: Sat Sep 17 05:27:47 2022 CEST ModDate: Sat Sep 17 05:27:47 2022 CEST Tagged: yes UserProperties: no Suspects: no Form: none JavaScript: no Pages: 12 Encrypted: no Page size: 595.289 x 782.365 pts Page rot: 0 File size: 1619204 bytes Optimized: yes PDF version: 1.7 name type encoding emb sub uni object ID ------------------------------------ ----------------- ---------------- --- --- --- --------- HKPPJH+CaeciliaLTStd-Bold Type 1C Custom yes yes yes 1949 0 HKPPKI+CaeciliaLTStd-Light Type 1C Custom yes yes yes 1950 0 HKPPNI+SFRB0700 Type 1C Custom yes yes yes 1910 0 HKPPOJ+CaeciliaLTStd-LightItalic Type 1C Custom yes yes yes 1951 0 HLAACI+CaeciliaLTStd-Roman Type 1C Custom yes yes yes 1952 0 HLAAHH+CaeciliaLTStd-Italic Type 1C Custom yes yes yes 1953 0 HLABKB+SFRM0800 Type 1C Custom yes yes yes 1911 0 HLADFI+CaeciliaLTStd-Heavy Type 1C Custom yes yes yes 1954 0 HLAFBO+MTSYN Type 1C Custom yes yes yes 83 0 HLAGBJ+MTMI Type 1C Custom yes yes yes 84 0 Jhove (Rel. 1.22.1, 2019-04-17) Date: 2023-07-06 02:32:35 CEST RepresentationInformation: pmc_9487607.pdf ReportingModule: PDF-hul, Rel. 1.12.1 (2019-04-17) LastModified: 2022-10-10 02:26:20 CEST Size: 1619204 Format: PDF Version: 1.7 Status: Well-Formed, but not valid SignatureMatches: PDF-hul ID: PDF-HUL-122 ErrorMessage: edu.harvard.hul.ois.jhove.module.pdf.PdfInvalidException: Invalid indirect destination - referenced object 'myrefsref8' cannot be found Offset: 1432849 ID: PDF-HUL-122 ErrorMessage: edu.harvard.hul.ois.jhove.module.pdf.PdfInvalidException: Invalid indirect destination - referenced object 'myrefsref9' cannot be found Offset: 1432849 ID: PDF-HUL-122 ErrorMessage: edu.harvard.hul.ois.jhove.module.pdf.PdfInvalidException: Invalid indirect destination - referenced object 'myrefsref14' cannot be found Offset: 1432849 MIMEtype: application/pdf Profile: Linearized PDF PDFMetadata: Objects: 1989 FreeObjects: 1 IncrementalUpdates: 1 DocumentCatalog: ViewerPreferences: HideToolbar: false HideMenubar: false HideWindowUI: false FitWindow: true CenterWindow: false DisplayDocTitle: true NonFullScreenPageMode: UseNone Direction: L2R ViewArea: CropBox ViewClip: CropBox PrintArea: CropBox PageClip: CropBox PageLayout: SinglePage PageMode: UseOutlines Language: en Outlines: Item: Title: Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@6a1aab78 Children: Item: Title: Introduction Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@462d5aee Item: Title: Materials and methods Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@69b0fd6f Item: Title: Results Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@757942a1 Item: Title: Discussion and conclusions Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@4a87761d Item: Title: Key Points Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@66d1af89 Item: Title: Acknowledgments Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@8646db9 Item: Title: Funding Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@37374a5e Item: Title: Code, software and data availability Destination: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject@4671e53b Info: Title: Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data Author: Subject: DOI: 10.1093/bib/bbac343, Briefings in Bioinformatics, 23(5), 5, 0 20xx. Abstract: Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at . Keywords: Creator: LaTeX with hyperref package Producer: Acrobat Distiller 22.0 (Windows) CreationDate: Sat Sep 17 05:57:47 CEST 2022 ModDate: Sat Sep 17 05:57:47 CEST 2022 ID: 0x9df52b4d154a4c636f7752ca4fb4c470, 0xa3d17c60281e50c86f3bcbf0dfc7d1e4 Filters: FilterPipeline: FlateDecode FilterPipeline: DCTDecode Images: Image: NisoImageMetadata: FormatName: image/jpg CompressionScheme: JPEG ImageWidth: 2050 ImageHeight: 1620 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Image: NisoImageMetadata: FormatName: image/jpg CompressionScheme: JPEG ImageWidth: 2050 ImageHeight: 1936 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Image: NisoImageMetadata: FormatName: image/jpg CompressionScheme: JPEG ImageWidth: 2050 ImageHeight: 1822 ColorSpace: RGB BitsPerSample: 8 BitsPerSampleUnit: integer Fonts: Type1: Font: BaseFont: HLAACI+CaeciliaLTStd-Roman FontSubset: true FirstChar: 40 LastChar: 246 FontDescriptor: FontName: HLAACI+CaeciliaLTStd-Roman Flags: Serif, Symbolic FontBBox: -167, -270, 1136, 926 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HLAAHH+CaeciliaLTStd-Italic FontSubset: true FirstChar: 66 LastChar: 122 FontDescriptor: FontName: HLAAHH+CaeciliaLTStd-Italic Flags: Serif, Symbolic, Italic FontBBox: -166, -270, 1053, 915 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HLADFI+CaeciliaLTStd-Heavy FontSubset: true FirstChar: 44 LastChar: 124 FontDescriptor: FontName: HLADFI+CaeciliaLTStd-Heavy Flags: Serif, Symbolic, ForceBold FontBBox: -170, -277, 1146, 970 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HLAFBO+MTSYN FontSubset: true FirstChar: 2 LastChar: 138 FontDescriptor: FontName: HLAFBO+MTSYN Flags: Symbolic FontBBox: 0, -954, 1043, 900 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HLAGBJ+MTMI FontSubset: true FirstChar: 2 LastChar: 60 FontDescriptor: FontName: HLAGBJ+MTMI Flags: Serif, Symbolic, Italic FontBBox: 0, -213, 987, 680 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HKPPNI+SFRB0700 FontSubset: true FirstChar: 2 LastChar: 2 FontDescriptor: FontName: HKPPNI+SFRB0700 Flags: Serif, Symbolic FontBBox: -224, -373, 1732, 955 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HLABKB+SFRM0800 FontSubset: true FirstChar: 2 LastChar: 2 FontDescriptor: FontName: HLABKB+SFRM0800 Flags: Serif, Symbolic FontBBox: -203, -320, 1554, 938 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HKPPJH+CaeciliaLTStd-Bold FontSubset: true FirstChar: 40 LastChar: 246 FontDescriptor: FontName: HKPPJH+CaeciliaLTStd-Bold Flags: Serif, Symbolic, ForceBold FontBBox: -166, -273, 1141, 938 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HKPPKI+CaeciliaLTStd-Light FontSubset: true FirstChar: 37 LastChar: 243 FontDescriptor: FontName: HKPPKI+CaeciliaLTStd-Light Flags: Serif, Symbolic FontBBox: -164, -266, 1130, 908 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true Font: BaseFont: HKPPOJ+CaeciliaLTStd-LightItalic FontSubset: true FirstChar: 45 LastChar: 121 FontDescriptor: FontName: HKPPOJ+CaeciliaLTStd-LightItalic Flags: Serif, Symbolic, Italic FontBBox: -166, -267, 1033, 909 FontFile3: true EncodingDictionary: Differences: true ToUnicode: true XMP: application/pdf Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data Sidorczuk KatarzynaGagat PrzemysławPietluch FilipKała JakubRafacz DominikBąkała LauraSłowik JadwigaKolenda RafałRödiger StefanFingerhut Legana C.H.W.Cooke Ira R.Mackiewicz PawełBurdukiewicz Michał DOI: 10.1093/bib/bbac343, Briefings in Bioinformatics, 23(5), 5, 0 20xx. Abstract: Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at . antimicrobial peptidesbenchmarksmachine learningnegative samplingpredictionreproducibility © The Author(s) 2022. Published by Oxford University Press. True Pages: Page: Label: 1 Annotations: Annotation: Subtype: Link Rect: 382, 46, 525, 55 Annotation: Subtype: Link Rect: 127, 605, 139, 617 Annotation: Subtype: Link Rect: 215, 605, 228, 617 Annotation: Subtype: Link Rect: 284, 605, 296, 617 Annotation: Subtype: Link Rect: 339, 605, 351, 617 Annotation: Subtype: Link Rect: 415, 605, 427, 617 Annotation: Subtype: Link Rect: 479, 605, 491, 617 Annotation: Subtype: Link Rect: 105, 590, 117, 602 Annotation: Subtype: Link Rect: 173, 590, 185, 602 Annotation: Subtype: Link Rect: 244, 590, 256, 602 Annotation: Subtype: Link Rect: 353, 590, 365, 602 Annotation: Subtype: Link Rect: 413, 590, 425, 602 Annotation: Subtype: Link Rect: 497, 590, 509, 602 Annotation: Subtype: Link Rect: 127, 575, 139, 587 Annotation: Subtype: Link Rect: 388, 419, 531, 429 Page: Label: 2 Annotations: Annotation: Subtype: Link Rect: 176, 670, 181, 676 Destination: 67 Annotation: Subtype: Link Rect: 184, 670, 189, 676 Destination: 67 Annotation: Subtype: Link Rect: 257, 659, 262, 665 Destination: 67 Annotation: Subtype: Link Rect: 266, 659, 270, 665 Destination: 67 Annotation: Subtype: Link Rect: 266, 637, 271, 643 Destination: 67 Annotation: Subtype: Link Rect: 276, 637, 280, 643 Destination: 67 Annotation: Subtype: Link Rect: 140, 528, 144, 533 Destination: 67 Annotation: Subtype: Link Rect: 148, 528, 156, 533 Destination: 76 Annotation: Subtype: Link Rect: 150, 484, 159, 489 Destination: 76 Annotation: Subtype: Link Rect: 163, 484, 171, 489 Destination: 76 Annotation: Subtype: Link Rect: 136, 418, 144, 423 Destination: 76 Annotation: Subtype: Link Rect: 148, 418, 157, 423 Destination: 76 Annotation: Subtype: Link Rect: 136, 362, 145, 368 Destination: 76 Annotation: Subtype: Link Rect: 162, 319, 170, 324 Destination: 76 Annotation: Subtype: Link Rect: 73, 263, 82, 269 Destination: 76 Annotation: Subtype: Link Rect: 170, 132, 179, 137 Destination: 76 Annotation: Subtype: Link Rect: 210, 109, 219, 115 Destination: 76 Annotation: Subtype: Link Rect: 247, 109, 255, 115 Destination: 76 Annotation: Subtype: Link Rect: 44, 99, 53, 104 Destination: 76 Annotation: Subtype: Link Rect: 94, 99, 103, 104 Destination: 76 Annotation: Subtype: Link Rect: 152, 98, 160, 104 Destination: 76 Annotation: Subtype: Link Rect: 173, 77, 182, 82 Destination: 76 Annotation: Subtype: Link Rect: 377, 667, 385, 673 Destination: 76 Annotation: Subtype: Link Rect: 497, 634, 506, 640 Destination: 76 Annotation: Subtype: Link Rect: 305, 580, 314, 585 Destination: 76 Annotation: Subtype: Link Rect: 320, 580, 328, 585 Destination: 76 Annotation: Subtype: Link Rect: 385, 580, 394, 585 Destination: 76 Annotation: Subtype: Link Rect: 510, 580, 518, 585 Destination: 76 Annotation: Subtype: Link Rect: 414, 569, 423, 574 Destination: 76 Annotation: Subtype: Link Rect: 504, 569, 513, 574 Destination: 76 Annotation: Subtype: Link Rect: 518, 569, 526, 574 Destination: 76 Annotation: Subtype: Link Rect: 418, 558, 427, 563 Destination: 76 Annotation: Subtype: Link Rect: 431, 558, 439, 563 Destination: 76 Annotation: Subtype: Link Rect: 429, 547, 437, 552 Destination: 76 Annotation: Subtype: Link Rect: 465, 437, 474, 442 Destination: 76 Annotation: Subtype: Link Rect: 479, 437, 488, 442 Destination: 76 Annotation: Subtype: Link Rect: 346, 382, 355, 388 Destination: 76 Annotation: Subtype: Link Rect: 472, 382, 480, 388 Destination: 76 Annotation: Subtype: Link Rect: 485, 338, 512, 344 Destination: 4 Annotation: Subtype: Link Rect: 486, 84, 516, 92 Destination: 28 Page: Label: 3 Annotations: Annotation: Subtype: Link Rect: 162, 691, 167, 699 Destination: 76 Annotation: Subtype: Link Rect: 192, 691, 197, 699 Destination: 76 Annotation: Subtype: Link Rect: 202, 691, 207, 699 Destination: 76 Annotation: Subtype: Link Rect: 222, 691, 227, 699 Destination: 76 Annotation: Subtype: Link Rect: 262, 691, 267, 699 Destination: 76 Annotation: Subtype: Link Rect: 322, 516, 326, 524 Destination: 76 Annotation: Subtype: Link Rect: 292, 691, 297, 699 Destination: 76 Annotation: Subtype: Link Rect: 331, 691, 336, 699 Destination: 76 Annotation: Subtype: Link Rect: 361, 691, 366, 699 Destination: 76 Annotation: Subtype: Link Rect: 391, 691, 396, 699 Destination: 76 Annotation: Subtype: Link Rect: 421, 691, 426, 699 Destination: 76 Annotation: Subtype: Link Rect: 431, 691, 436, 699 Destination: 76 Page: Label: 4 Annotations: Annotation: Subtype: Link Rect: 149, 678, 176, 684 Destination: 25 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 179, 677, 190, 685 Annotation: Subtype: Link Rect: 157, 588, 187, 597 Destination: 28 Annotation: Subtype: Link Rect: 263, 514, 272, 519 Destination: 76 Annotation: Subtype: Link Rect: 204, 459, 213, 464 Destination: 76 Annotation: Subtype: Link Rect: 218, 459, 227, 464 Destination: 76 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 40, 402, 287, 423 Annotation: Subtype: Link Rect: 160, 327, 168, 333 Destination: 76 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 83, 305, 119, 313 Annotation: Subtype: Link Rect: 139, 295, 165, 301 Destination: 4 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 236, 272, 270, 280 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 273, 272, 283, 279 Annotation: Subtype: Link Rect: 77, 119, 103, 125 Destination: 4 Annotation: Subtype: Link Rect: 305, 623, 314, 629 Destination: 76 Annotation: Subtype: Link Rect: 477, 580, 504, 586 Destination: 4 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 507, 579, 518, 586 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 520, 579, 531, 586 Annotation: Subtype: Link Rect: 393, 492, 402, 497 Destination: 76 Annotation: Subtype: Link Rect: 406, 492, 415, 497 Destination: 76 Annotation: Subtype: Link Rect: 477, 459, 503, 465 Destination: 4 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 506, 456, 543, 466 Annotation: Subtype: Link Rect: 305, 393, 332, 399 Destination: 4 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 335, 390, 371, 400 Annotation: Subtype: Link Rect: 373, 306, 382, 311 Destination: 76 Annotation: Subtype: Link Rect: 431, 284, 458, 290 Destination: 4 Annotation: Subtype: Link Rect: 515, 251, 542, 257 Destination: 4 Annotation: Subtype: Link Rect: 369, 218, 378, 223 Destination: 76 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 372, 182, 409, 192 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 411, 184, 421, 192 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 304, 171, 340, 181 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 403, 149, 440, 159 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 442, 151, 452, 159 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 363, 107, 396, 115 Annotation: Subtype: Link Rect: 505, 86, 531, 92 Destination: 25 Page: Label: 5 Annotations: Annotation: Subtype: Link Rect: 161, 691, 166, 699 Destination: 76 Annotation: Subtype: Link Rect: 201, 691, 206, 699 Destination: 76 Annotation: Subtype: Link Rect: 261, 691, 266, 699 Destination: 76 Annotation: Subtype: Link Rect: 291, 691, 296, 699 Destination: 76 Annotation: Subtype: Link Rect: 301, 691, 306, 699 Destination: 76 Annotation: Subtype: Link Rect: 321, 691, 326, 699 Destination: 80 Annotation: Subtype: Link Rect: 331, 691, 336, 699 Destination: 80 Annotation: Subtype: Link Rect: 341, 691, 346, 699 Destination: 76 Annotation: Subtype: Link Rect: 351, 691, 356, 699 Destination: 80 Annotation: Subtype: Link Rect: 361, 691, 366, 699 Destination: 80 Annotation: Subtype: Link Rect: 440, 691, 445, 699 Destination: 76 Annotation: Subtype: Link Rect: 450, 691, 455, 699 Destination: 76 Page: Label: 6 Annotations: Annotation: Subtype: Link Rect: 119, 132, 146, 139 Destination: 25 Annotation: Subtype: Link Rect: 347, 220, 373, 226 Destination: 25 Annotation: Subtype: Link Rect: 305, 154, 332, 160 Destination: 25 Annotation: Subtype: Link Rect: 454, 122, 481, 128 Destination: 25 Annotation: Subtype: Link Rect: 429, 78, 455, 84 Destination: 25 Page: Label: 7 Annotations: Annotation: Subtype: Link Rect: 143, 711, 169, 717 Destination: 25 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 239, 579, 272, 587 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 101, 500, 139, 510 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 140, 502, 156, 509 Annotation: Subtype: Link Rect: 48, 457, 78, 465 Destination: 49 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 92, 458, 107, 465 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 126, 458, 141, 465 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 218, 370, 251, 378 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 207, 304, 240, 312 Annotation: Subtype: Link Rect: 85, 84, 115, 92 Destination: 49 Annotation: Subtype: Link Rect: 334, 665, 363, 673 Destination: 49 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 388, 644, 423, 652 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 425, 644, 440, 652 Annotation: Subtype: Link Rect: 311, 523, 341, 531 Destination: 49 Annotation: Subtype: Link Rect: 331, 424, 360, 432 Destination: 49 Annotation: Subtype: Link Rect: 334, 327, 342, 333 Destination: 80 Annotation: Subtype: Link Rect: 513, 304, 543, 312 Destination: 49 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 386, 226, 423, 236 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 307, 214, 554, 237 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 321, 217, 332, 225 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 461, 215, 498, 225 Annotation: Subtype: Link Rect: 311, 62, 341, 70 Destination: 49 Page: Label: 8 Annotations: Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 160, 172, 196, 182 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 409, 172, 446, 182 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 304, 161, 342, 171 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 344, 163, 354, 171 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 485, 161, 523, 171 Annotation: Subtype: Link Rect: 305, 140, 335, 148 Destination: 60 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 503, 119, 543, 127 Page: Label: 9 Annotations: Annotation: Subtype: Link Rect: 214, 194, 243, 202 Destination: 49 Annotation: Subtype: Link Rect: 208, 150, 238, 158 Destination: 60 Annotation: Subtype: Link Rect: 50, 97, 59, 103 Destination: 80 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 140, 64, 179, 72 Annotation: Subtype: Link Rect: 183, 63, 213, 71 Destination: 49 Annotation: Subtype: Link Rect: 225, 65, 229, 70 Destination: 49 Annotation: Subtype: Link Rect: 240, 65, 244, 70 Destination: 60 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 249, 64, 264, 71 Annotation: Subtype: Link Rect: 413, 139, 443, 147 Destination: 60 Annotation: Subtype: Link Rect: 523, 75, 532, 81 Destination: 76 Page: Label: 10 Annotations: Annotation: Subtype: Link Rect: 44, 709, 74, 717 Destination: 28 Annotation: Subtype: Link Rect: 224, 676, 254, 684 Destination: 49 Annotation: Subtype: Link Rect: 265, 678, 269, 684 Destination: 49 Annotation: Subtype: Link Rect: 279, 678, 283, 684 Destination: 60 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 41, 666, 56, 674 Annotation: Subtype: Link Rect: 210, 426, 218, 432 Destination: 80 Annotation: Subtype: Link Rect: 224, 426, 233, 432 Destination: 80 Annotation: Subtype: Link Rect: 212, 393, 221, 399 Destination: 80 Annotation: Subtype: Link Rect: 72, 709, 72, 709 Destination: 1 Annotation: Subtype: Link Rect: 199, 326, 234, 334 Annotation: Subtype: Link Rect: 274, 295, 282, 300 Destination: 80 Annotation: Subtype: Link Rect: 196, 64, 205, 70 Destination: 80 Annotation: Subtype: Link Rect: 344, 346, 525, 356 Annotation: Subtype: Link Rect: 432, 335, 547, 345 Annotation: Subtype: Link Rect: 302, 324, 420, 334 Annotation: Subtype: Link Rect: 347, 313, 490, 323 Annotation: Subtype: Link Rect: 307, 264, 313, 269 Destination: 1 Annotation: Subtype: Link Rect: 307, 231, 313, 236 Destination: 1 Annotation: Subtype: Link Rect: 307, 198, 313, 203 Destination: 1 Annotation: Subtype: Link Rect: 307, 176, 313, 181 Destination: 1 Annotation: Subtype: Link Rect: 307, 110, 313, 116 Destination: 1 Annotation: Subtype: Link Rect: 307, 88, 313, 94 Destination: 1 Page: Label: 11 Annotations: Annotation: Subtype: Link Rect: 52, 722, 58, 727 Destination: 1 Annotation: Subtype: Link Rect: 52, 689, 58, 694 Destination: null Annotation: Subtype: Link Rect: 52, 656, 58, 661 Destination: null Annotation: Subtype: Link Rect: 48, 623, 58, 629 Destination: 1 Annotation: Subtype: Link Rect: 48, 590, 58, 596 Destination: 1 Annotation: Subtype: Link Rect: 48, 558, 58, 563 Destination: 1 Annotation: Subtype: Link Rect: 48, 525, 58, 530 Destination: 1 Annotation: Subtype: Link Rect: 48, 492, 58, 497 Destination: null Annotation: Subtype: Link Rect: 48, 459, 58, 464 Destination: 1 Annotation: Subtype: Link Rect: 48, 426, 58, 431 Destination: 1 Annotation: Subtype: Link Rect: 48, 404, 58, 409 Destination: 1 Annotation: Subtype: Link Rect: 48, 371, 58, 377 Destination: 1 Annotation: Subtype: Link Rect: 48, 338, 58, 344 Destination: 1 Annotation: Subtype: Link Rect: 48, 295, 58, 300 Destination: 1 Annotation: Subtype: Link Rect: 48, 262, 58, 267 Destination: 1 Annotation: Subtype: Link Rect: 48, 229, 58, 234 Destination: 1 Annotation: Subtype: Link Rect: 48, 207, 58, 212 Destination: 1 Annotation: Subtype: Link Rect: 48, 163, 58, 168 Destination: 1 Annotation: Subtype: Link Rect: 48, 130, 58, 135 Destination: 1 Annotation: Subtype: Link Rect: 48, 97, 58, 103 Destination: 1 Annotation: Subtype: Link Rect: 48, 75, 58, 81 Destination: 1 Annotation: Subtype: Link Rect: 48, 53, 58, 59 Destination: 1 Annotation: Subtype: Link Rect: 309, 700, 319, 705 Destination: 1 Annotation: Subtype: Link Rect: 309, 667, 319, 672 Destination: 1 Annotation: Subtype: Link Rect: 309, 634, 319, 640 Destination: 1 Annotation: Subtype: Link Rect: 309, 601, 319, 607 Destination: 1 Annotation: Subtype: Link Rect: 309, 569, 319, 574 Destination: 1 Annotation: Subtype: Link Rect: 309, 536, 319, 541 Destination: 1 Annotation: Subtype: Link Rect: 309, 503, 319, 508 Destination: 1 Annotation: Subtype: Link Rect: 309, 470, 319, 475 Destination: 1 Annotation: Subtype: Link Rect: 309, 448, 319, 453 Destination: 1 Annotation: Subtype: Link Rect: 309, 415, 319, 420 Destination: 1 Annotation: Subtype: Link Rect: 309, 393, 319, 398 Destination: 7 Annotation: Subtype: Link Rect: 309, 360, 319, 366 Destination: 7 Annotation: Subtype: Link Rect: 309, 327, 319, 333 Destination: 4 Annotation: Subtype: Link Rect: 309, 295, 319, 300 Destination: 4 Annotation: Subtype: Link Rect: 309, 262, 319, 267 Destination: 4 Annotation: Subtype: Link Rect: 309, 229, 319, 234 Destination: 4 Annotation: Subtype: Link Rect: 309, 196, 319, 201 Destination: 4 Annotation: Subtype: Link Rect: 309, 174, 319, 179 Destination: 4 Annotation: Subtype: Link Rect: 309, 152, 319, 157 Destination: 4 Annotation: Subtype: Link Rect: 309, 130, 319, 135 Destination: 4 Annotation: Subtype: Link Rect: 365, 94, 524, 104 Annotation: Subtype: Link Rect: 309, 86, 319, 92 Destination: 25 Annotation: Subtype: Link Rect: 309, 53, 319, 59 Destination: 25 Page: Label: 12 Annotations: Annotation: Subtype: Link Rect: 42, 700, 52, 705 Destination: 25 Annotation: Subtype: Link Rect: 42, 667, 52, 672 Destination: 25 Annotation: Subtype: Link Rect: 42, 634, 52, 640 Destination: 25 Annotation: Subtype: Link Rect: 42, 601, 52, 607 Destination: 25 Annotation: Subtype: Link Rect: 303, 722, 313, 727 Destination: 33 Annotation: Subtype: Link Rect: 303, 689, 313, 694 Destination: 60 Annotation: Subtype: Link Rect: 303, 645, 313, 651 Destination: 67 Annotation: Subtype: Link Rect: 303, 623, 313, 629 Destination: 67 Annotation: Subtype: Link Rect: 303, 601, 313, 607 Destination: 67 Checksum: 1bbe0593 Type: CRC32 Checksum: 149a3d0982c7cb6b3285e2da29459e59 Type: MD5 Checksum: 032701195b795b32fc2805dd78edb0275bf8f614 Type: SHA-1