Briefings in Bioinformatics, 2022, 23(5), 1–12 https://doi.org/10.1093/bib/bbac343 Problem Solving Protocol Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data † † Katarzyna Sidorczuk , Przemysław Gagat , Filip Pietluch , Jakub Kała , Dominik Rafacz , Laura Bąkała , Jadwiga Słowik , Rafał Kolenda , Stefan Rödiger , Legana C.H.W. Fingerhut , Ira R. Cooke , Paweł Mackiewicz and Michał Burdukiewicz Corresponding author: Michał Burdukiewicz; E-mail: michalburdukiewicz@gmail.com † Katarzyna Sidorczuk and Przemys?aw Gagat contributed equally to this work. Abstract Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at http://BioGenies.info/AMPBenchmark. Keywords: antimicrobial peptides, benchmarks, machine learning, negative sampling, prediction, reproducibility Katarzyna Sidorczuk received the M.Sc. degree in biotechnology from the University of Wrocław, Poland, in 2019. She is currently pursuing the Ph.D. degree in biological sciences at the University of Wrocław. Her research focuses on bioinformatics and machine learning approaches for the analysis and prediction of peptide functions, protein targeting sequences and bacterial adhesins. Przemysław Gagat is an Assistant Professor in the Faculty of Biotechnology at the University of Wrocław in Poland, where he received his B.Sc., (2006) M.Sc. (2008) and PhD (2014) in biotechnology. He has been involved in research projects in the field of bioinformatics, molecular phylogeny, molecular biology and evolution. His research interests include particularly endosymbioses of primary and complex plastids, and recently antimicrobial and anticancer peptides. Filip Pietluch graduated with both a Masters and a Bachelors degree in Biotechnology from the University of Wrocław, Poland. He is currently a PhD student at the University of Wrocław, exploring phylogenetics and AMPs evolution. His main research topics are in computational and evolutionary biology. Jakub Kał,a received the M. Sc. Eng. degree in Data Science from the Warsaw University of Technology in 2021. His main research focuses on applications of machine learning in protein function prediction. Dominik Rafacz is a M.Sc. student with his major in Data Science at the Warsaw University of Technology. He has contributed to several projects related to predicting properties of proteins. He is an active open-source developer, author and coauthor of several packages for the R programming language, including a framework for biological sequences storage and processing. Laura Bąkała is another Data Science major at the Warsaw University of Technology, as well as a M.Sc. student. She has worked on a few open-source R packages together with Dominik Rafacz, among those being the aforementioned framework for handling biological sequences. Jadwiga Słowik received her master’s in data science from Warsaw University of Technology. She is passionate about software development and competitive programming. Rafał Kolenda graduated in veterinary medicine from Wrocław University of Environmental and Life Sciences, Poland, in 2013. He received Dr. med. vet. degree from Freie Universität, Berlin, Germany, in 2018. He is currently working as a research associate at the Department of Biochemistry and Molecular Biology of Wrocław University of Environmental and Life Sciences, Poland. In his research, he uses molecular microbiology and omics techniques to investigate the pathogenic actions of bacteria. Moreover, he is involved in the development of new tools for Escherichia coli adhesiome analysis. Stefan Rödiger is Privatdozent at the Brandenburg University of Technology. He studied pharamabiotechnology and forensics and received the doctoral degree from the Charité. His research includes personalized medicine, bioinformatics and pharmacology. Legana C H W Fingerhut graduated with a B.Sc degree in Ecology and Conservation and Zoology followed by a B.Biomed.Sc(Hons1) degree in Molecular Biology at James Cook University, Australia. She is currently a PhD candidate in bioinformatics examining antimicrobial peptide classification. She is the first author of the genome-wide antimicrobial peptide prediction R package, ampir. Ira R Cooke received his PhD from the Australian National University in 2006 and is now a senior lecturer in bioinformatics at James Cook University, Australia. He is interested in the evolution and function of small secreted proteins in marine taxa, especially cephalopods and corals. Paweı Mackiewicz is a professor at the University of Wrocław in Poland. He is an author and co-author of more than 180 scientific publications. He is involved in many interdisciplinary research projects in the field of bioinformatics, genomics, phylogenetics and molecular evolution. His research interests focus on endosymbiosis, genetic code optimization and evolution, computer simulation of genome evolution, as well as phylogeny at the level of genes, proteins, genomes and organisms including ancient DNA samples. Michał Burdukiewicz received Ph.D. in biochemistry from the University of Wrocław in 2019. He is currently working as a post-doc at the Institute of Biotechnology and Biomedicine at the Autonomous University of Barcelona and a research assistant in the Centre for Clinical Research at the Medical University of Białystok. His research interests cover machine learning applications in the functional analysis of peptides and proteins, focusing on amyloids. Moreover, he is co-developing tools for proteomics, mainly hydrogen-deuterium exchange monitored by mass spectrometry. Received: May 30, 2022. Revised: July 7, 2022. Accepted: July 25, 2022 © The Author(s) 2022. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. 2 | Sidorczuk et al. Introduction Since the databases seem to contain generally mature AMPs, and moreover the developers often restrict the sequence length Antimicrobial peptides (AMPs) are short polypeptides, generally in their data sets, the algorithms are mainly trained on mature composed of up to 50 amino acids that are widespread in all forms AMPs. Consequently, they are good at detecting mature AMPs but of life, from microorganisms, i.e. bacteria, archaeans and one- might have problems classifying longer sequences, including the celled eukaryotes, to multicellulars [1, 2]. In microorganisms, they precursor proteins [25]. participate in self-protection and microbial competition [3, 4]; in The issue of identification of precursor and longer AMPs can be multicellulars, they are part of the first line of defence against satisfactorily addressed because the data about these sequences microorganisms but also target viruses and cancer cells [5, 6]. are available in public databases, e.g. in UniProt [26]. The real Despite their diversity in the primary amino acid sequence, AMPs problem with AMP prediction lies with the negative data set as are rich in cationic and hydrophobic residues. The positive charge there are hardly any sequences annotated as non-AMPs. Interest- and hydrophobicity allow them to fold into amphipathic sec- ingly, the lack of reliable negative samples also concerns other ondary structures that preferentially disrupt negatively charged areas related to bioinformatics, e.g. prediction of disease genes microbial/cancer cell membranes but not the healthy eukaryotic [27, 28], microRNAs [29], bacterial virulence factors [30]; iden- ones; the latter contain stabilizing cholesterol and their outer tification of protein–protein [31], protein–RNA/DNA [32, 33] and leaf let is composed of neutral phospholipids. AMPs can trigger protein–drug interaction sites [32, 34]; as well as inferring protein transient membrane disruption by forming pores and micelliza- sequence-function relationships [35]. tion but, depending on the concentration, they may lead to cell In all these cases, the developers have to resort to: (i) one-class death by osmotic shock [7–10]. The alternative mechanisms of classification, (ii) positive-unlabelled learning or (iii) to somehow action, especially for the larger AMPs (about 100 amino acids long build a negative data set. In the first case, the model is trained or longer), include binding to specific cytosolic macromolecules on the positive sample, whereas in the second on the positive and thereby inhibiting synthesis of proteins, nucleic acids and and unlabelled data; the unlabelled set includes both positive components of the cell wall [11, 12]. and negative examples. These two approaches aim at solving AMPs have also been demonstrated to have lower selection the problem of the negative sample by either not using it at for resistance compared with traditional antibiotics. A traditional all or applying a wide variety of strategies to obtain negative antibiotic specifically targets a single enzyme but AMPs, most of cases from the unlabelled set based on the positive sample, e.g. all, interact non-specifically with many components of the cell using distance measures (for details, see [36, 37]). Interestingly, membrane. This makes it more difficult for bacteria to develop neither the one-class classification nor the positive-unlabelled resistance against them [13–15]. learning have attracted the attention of developers working on According to the World Health Organization, the antibiotic AMP prediction. The majority of them created their negative sets resistance is currently behind the death of at least 700 000 people by performing non-probability sampling on sequences deposited each year; however, the forecast of the death toll of 10 million in UniProt [26] or other databases (e.g. PDB [38]) though they do annually by 2050 makes the race for alternative therapeutics of not define it as such. In this approach, the negative examples are the utmost importance [16]. In light of their medical potential, selected on the basis of clearly defined criteria dictated by the AMPs are viewed as hopeful candidates for further experimental researcher, and these criteria represent a sampling method (for research. Consequently, we have recently observed a boom in details, see section Materials and methods and Table 1). In con- computational tools for AMP prediction with the machine learn- trast to positive-unlabelled learning, dividing UniProt sequences ing algorithms leading the way [17]. between AMPs and non-AMPs does not require any complex Traditionally, biological problems have first been approached methodology and is independent of the positive sample but for the by conventional, i.e. non-deep machine learning-based methods, length and number of sequences for some sampling methods. It such as random forests (RF) or support vector machines (SVM), is simply made by sequence filtering and then randomly selecting which were then followed by more complex deep learning algo- peptides for the final negative data set. For clarity purposes in this rithms [17]. In order to produce reliable predictions, the algo- article, the name of the sampling method is always preceded by rithms first require labelled training data to build a predictive an abbreviation: SM (sampling method), TSM (sampling method model. The training data include a positive and a negative data used to generate the training set) or BSM (sampling method used set, in our case AMPs and non-AMPs, respectively. In order to to generate the benchmark set) and colon, e.g. SM:AmpGram, make the sequences readable for machine learning, they have TSM:AmpGram and BSM:AmpGram, respectively. to be transformed into informative features (feature vectors) and The aim of this study was to elaborate on the impact of negative this process is known as feature extraction. Depending on the data sampling on model performance and benchmarking. We method of feature extraction, the obtained feature space may decided to explore this issue because each developer of an AMP require additional reduction, and consequently an appropriate predictor tested only one sampling method to build the optimal feature selection method is applied, e.g. for AmpGram the initial non-AMP class despite knowing that machine learning models feature set amounted to 33 620 n-grams (amino acid motifs of n heavily depend on the data sets they are trained on. They all elements) but was decreased with Quick Permutation Test to 13 overlooked the fact that various sampling methods could gener- 087 most informative descriptors [18]. ate statistically different samples, thereby affecting the predic- There are many databases with thousands of experimentally tive power of their models. Moreover, and more importantly, we validated AMP sequences, such as DBAASP [19], APD [20], CAMP investigated how machine learning architectures perform when [21], DRAMP [22] or dbAMP [23]; therefore, it is possible to cre- they are trained on a given data set but tested on a different ate a representative positive data set. However, the authors of one, a commonplace in model benchmarking (Figure 1A). This AMP classifiers, except for ampir [24], do not take into account particular issue is of vital importance not only for the comparison that there might be two types of sequences deposited in these of AMP predictors but for the evaluation of all machine learning databases: mature AMPs and precursor AMPs with cleavable N- models in general. We define the machine learning architecture terminal signal peptides; AMPs are mostly secretory proteins. as an approach to solve the problem of AMP prediction with Table 1. A comprehensive summary of the negative sampling methods implemented for AMP prediction Sampling method Excluded keywords Sequence lengths CD-HIT Additional filtering Balanced classes Reference Wang et al. secreted, antimicrobial similar to the length 0.7 no [41] distribution of positive dataset CS-AMPpred antimicrobial 16 - 90 0.4 yes [42] iAMP-2L antimicrobial, antibiotic, 5–100 0.4 no [43] fungicide, defensin Gabere&Noble (DAMPD dataset) antimicrobial equal to the length no localization: golgi, no [25] distribution of positive cytoplasm, endoplasmic dataset reticulum or mitochondria AMAP antimicrobial similar to the length 0.4 no [44] distribution of positive dataset AMPScanner V2 antimicrobial, antibiotic, equal to the length no localization: cytoplasm; yes [46] antiviral, fungicide, secreted distribution of positive removed sequences dataset similar to any AMPs with BLAT [45] dbAMP transmembrane, toxin, secreted, 10–100 0.4 no [23] defensin, antimicrobial, antibiotic, antiviral, fungicide Witten&Witten antimicrobial, antibiotic, equal to the length 0.4 localization: cytoplasm; yes [47] antiviral, fungicide, secreted distribution of positive only cysteine-free dataset substrings AmpGram antimicrobial, antibacterial, equal to the length no yes [18] antiviral, fungicide, secreted, distribution of positive transit peptide dataset ampir-mature antimicrobial 10–40 0.9 no [24] AMPlify antimicrobial, antibiotic, defence, equal to the length no removed potential AMPs∗ yes [48] defensin, bacteriocin, fungicide distribution of positive dataset ∗ UniProt sequences with keywords: antimicrobial, antibiotic, defence, defensin, bacteriocin, fungicide Benchmarks in antimicrobial peptide prediction 3 | 4 | Sidorczuk et al. all its parameters involved in the machine learning cycle. The cytosolic or additional location, especially experimentally veri- architectures for our study were developed based on published fied, was included to 65%, 70% and 98% for SM:Gabere&Noble, models that we were able to reuse or reimplement; some might SM:AMPScannerV2 and SM:Witten&Witten, respectively. slightly deviate from the original methods (for details, see section For five methods, SM:AmpGram, SM:AMPlify, SM: AMPScan- Materials and methods and Table 2, S2). For clarity purpose, the nerV2, SM:CS-AMPPred and SM:Witten&Witten, the number of name of a given architecture always begins with a letter ‘A’ and sequences in the negative sample depended on the positive one, colon, e.g. A:AmpGram. By the term machine learning model, we i.e. the data sets were balanced. Two methods: SM:AMAP and understand one specific instance of a given architecture, i.e. an SM:dbAMP generated only slightly imbalanced samples, the for- architecture trained on the same positive and one of negative mer due to the reduction of sequence redundancy with CD-HIT samples. Consequently, what we did was to generate 660 machine [40] at the end of the sampling process but the latter by acci- learning models using (i) 12 defined architectures, (ii) the same dent. The remaining methods produced imbalanced (SM:iAMP- positive training data set and (iii) 11 different negative sampling 2L, SM:Wang et. al) or highly imbalanced (SM:Gabere&Noble) methods each run five times (Figure 1B). To our knowledge, this sets with the predominance of non-AMPs. The exception was was the first kind of such a research project undertaken, and SM:ampir-mature with minority of non-AMPs (Table 1, S4, S5). moreover on such a scale. Five negative sampling methods: SM:AmpGram, SM: AMPlify, SM:AMPScannerV2, SM:Gabere&Noble and SM: Witten&Witten produced non-AMPs that exactly matched in length peptides and Materials and methods proteins contained in the positive set, and they were mostly up to Data sets 50 amino acids long though the sequence maximum length was To create the positive data set, we used DBAASP v3.0 [19], a 190 amino acids. The negative samples of SM:AMAP and SM:Wang manually curated database for experimentally verified peptides et. al were only similar in terms of length distribution to the AMP with antimicrobial properties. We selected sequences with set because of CD-HIT [39, 40] reduction at the end of sequence activity against Gram positive or Gram negative bacteria. Next, filtering. The SM:ampir-mature generated only short sequences, we removed those with non-standard amino acids or shorter and each sequence length within the range of 10 to 40 amino than five. We used CD-HIT version 4.8.1 [39, 40] to reduce the acids was approximately equally represented (Table 1, Figure S1). redundancy and eliminated sequences with the identity threshold The other sampling methods focused on longer peptides and higher than 90%. This threshold was most frequently used for proteins though at the same time they rejected sequences longer the reduction of positive data in the algorithms selected for than about 100 amino acids. These methods included SM:dbAMP, the reimplementation of negative sampling methods (Table SM:CS-AMPPred and SM:iAMP-2L and their sequence length dis- S1). In total, we obtained 5190 AMP sequences. To prevent the tribution resembled that of an upside-down isosceles triangle information leakage, the positive data set was split, before (Table 1, Figure S1). sampling the negative set, into a training sample (80%, 4151 All the negative data sampling methods generating sets with sequences) and a benchmark sample (20%, 1039 sequences). equal or similar length distribution to the positive sample selected The negative data set used for sampling was created using their non-AMPs from peptide/protein fragment or fragments, sequences available in the UniProt database. The reviewed protein while the other methods from uncut sequences of the negative sequences (563 972) and their annotations were downloaded from UniProt data set. the UniProtKB release 2020_06 [26]. In order to avoid overrepresentation of highly similar sequences We considered 26 methods of negative data sampling from in the non-AMP samples, we used the clustering algorithm CD- literature (Table S2) and selected 11 well-described ones for HIT version 4.8.1 [40] for seven sampling methods according to reuse/reimplementation (Table 1). Each method was run on the their description. We removed sequences above a certain identity negative data set of UniProt sequences five times to create five threshold, and mostly it was 40% (Table 1). replicates of the training and benchmark samples (Table S4, S5) Interestingly, SM:AMPlify, SM:AMPScanner V2 and SM:ampir- to verify their repeatability. Some selected methods required mature additionally verified whether the negative data set did not, modifications, e.g. removal of sequences with non-standard by chance, contain sequences from the positive data set (Table 1). amino acids to make them readable for all architectures. We This might arise as a result of: (i) non-AMPs generation from a also took measures to prevent information leakage between the protein fragment or fragments, and (ii) an improper/lack of anno- training and benchmark sets, especially for the sampling methods tation in UniProt [26] for sequences that are indeed antimicrobial. that did not depend on the positive data. The full description of The sampling methods generated sequences that greatly dif- changes is provided in the Supplementary Data. fered from those in the positive data set both in the amino The selected sampling methods use combinations of keywords acid composition (Figure S2–S6) and physicochemical properties to search the negative data set of UniProt sequences. Gener- (Figure S7). There were also some pronounced differences among ally the keywords are repetitive among the methods used. They the negative sets, but the five iterations of each method always allowed to filter out AMPs by naming their typical functions, e.g. produced similar samples (Figure S2–S8). ‘antimicrobial’, ‘antibacterial’, ‘antiviral’ and/or restrict the cellu- lar compartment to cytoplasm since AMPs are mostly secretory Model architectures proteins (Table 1). The latter is, however, unfortunate because the We considered 26 model architectures for the prediction of AMPs predictive models focus then on differences between cytoplasmic from literature (Table S2), and selected 12 for reimplementation, and secretory peptides instead of detecting AMPs, i.e. they neglect two deep learning and ten non-deep learning-based methods, the issue of (i) distinguishing AMPs from secretory non-AMPs and the latter represent mainly RF and SVM algorithms (Table 2). In (ii) cytosolic AMPs. The number of the filtered-out sequences was our research, we chose algorithms described in detail that could small if only the function keywords were considered, e.g. about be run locally and do not require any usage of web servers or 1% of the negative data set of UniProt sequences for SM:CS- other software for feature selection. Moreover, we focused on the AMPPred, SM:AMAP and SM:iAMP-2L, but increased when the classification task, i.e. the model ability to divide sequences into Table 2. A comprehensive summary of the implemented architectures for AMP prediction Type Architecture Year Feature space Feature extraction Feature Algorithm Reference size selection Deep learning-based AMPScanner V2 2018 128 Embedding none DNN with [46] architectures convolutional, maximal pooling and LSTM layers Deep-AmPEP30 2020 86 PseKRAAC none CNN with two [49] convolutional layers, two maximum pooling layers, and one fully connected hidden layer Non-deep learning-based CS-AMPpred 2012 9 α-helix, α-helix propensity, β-sheet, loop PCA SVM [42] architectures formation, charge, hydrophobicity, f lexibility, amphipathicity, hydrophobic moment iAMP-2L 2013 40 PseAAC none FKNN [43] SVM-LZ 2015 1000 LZ complexity pairwise similarity scores none sequence alignment [50] (blastp), SVM MLAMP 2016 30 PseAAC with grey model coefficients none RF [51] AmPEP 2018 23 Selected distribution descriptors none RF [52] AMAP 2019 20 AAC none SVM [44] AmPEPpy 2020 105 CTD none RF [53] MACREL 2020 22 relative position of the first occurrence of none RF [54] residues in three groups of amino acids defined by their free energy of transition in a peptide from a random coil in aqueous environment to an organized helical structure in a lipid phase, solvent accessibility, AAC, charge and solubility, instability, aliphaticity, propensity to bind to membranes, hydrophobicity AmpGram 2020 32 835–33 612 n-grams QuiPT RF [18] ampir 2020 27 PseAAC, amphiphilicity, hydrophobicity, none SVM [24] isoelectric point, molecular weight, net charge Benchmarks in antimicrobial peptide prediction 5 | 6 | Sidorczuk et al. Figure 1. Schematic representation of the traditional model benchmarking (A) and the methodology employed in comparing the impact of different negative data sampling methods on model performance (B). Models 1, 2 and 3 (colourful hexagons) were trained on data set A, B and C (colourful rectangles), respectively. Each data set was generated by an appropriate negative sampling method (white ovals) and a positive sample (blue rectangles). In the evaluation process, the models were compared only on the benchmark set C, built with the same method as the training set C, thereby introducing some bias in favour of Model 3 in the benchmark analysis (A). Architectures were developed based on published models, and they represent the algorithm with all its parameters involved in the machine learning cycle (white parallelograms). Each architecture was trained on the same positive data set (the white rectangle) and a negative sample was generated by one of the 11 negative sampling methods (white ovals) five times to verify the repeatability. The training and benchmark sample are indicated as blue and red rectangles, respectively. The models (orange hexagons) represent instances of architectures trained on given data sets and were validated on each benchmark sample. The results of model performance were indicated as white clouds (B). AMPs or non-AMPs. Consequently, we did not consider software occurrence frequency of one of the 20 amino acids in a peptide that is trained using MIC (minimum inhibitory concentration) sequence (Table 2). Other architectures such as A:iAMP-2L, values, e.g. Witten&Witten or multiclass models. However, if the A:MLAMP, A:ampir and A:Deep-AmPEP30 used features based on multiclass algorithm was composed of two models: one predicting pseudo-amino acid composition. Beyond the simple amino acid if a given sequence is or is not an AMP, and the second classifying counts, they included various physicochemical and structural an AMPs into functional groups, we did reimplement the first properties of amino acids to incorporate the information about model, e.g. for A:iAMP-2L and A:MLAMP. the sequence order; their feature space increased accordingly For each model, sequences in the data sets were transformed (Table 2). Four architectures, A:CS-AMPPred, A:MACREL, A:AmPEP (encoded) into features (descriptors) using an appropriate feature and A:AmPEPpy, used features based on structural and physic- extraction method (Table 2). In the case of non-deep learning ochemical properties of peptide sequences, e.g. their α-helix architectures, the features can be defined by the researcher, based propensity, charge and hydrophobicity (Table 2). A:SVM-LZ used on the knowledge of AMP properties, whereas the deep learning pairwise similarity scores, and A:AmpGram n-grams. In the case architectures can automatically learn high-level features from of A:AMPScanner V2 the feature information was extracted in the training data sets though A:Deep-AmPEP30 also employed the embedding layer, and then the obtained embeddings fed the developer-defined features. subsequent layers of the models (Table 2). The simplest feature extraction method was used by A:AMAP, Five non-deep machine learning architectures used feature and it was based on the amino acid composition. Consequently, selection for feature space optimization, but we implemented its feature space contained 20 descriptors, each ref lecting the it for only two architectures: A:CS-AMPpred and A:AmpGram. Benchmarks in antimicrobial peptide prediction | 7 The former used principal component analysis and the latter sampling methods is much greater than the AUC dispersal found Quick Permutation Test (Table 2). For A:AmPEP and A:ampir, we inside a single architecture or method. The MAD ratios amounted used the reduced feature space indicated by their developers to 1.29, 0.48 and 0.29 for architectures, training and benchmark without computing Pearson correlation coefficients and rigorous data sampling methods, respectively, and express in numbers the recursive feature elimination, respectively. For A:AmPEPpy, we relative AUC variation presented in the graphical form in the box abandoned the reduction of feature space by stepwise feature plots (Figure 2B–D). Moreover, to further verify the importance selection because according to the authors it did not improve the of the three components, we conducted pairwise Wilcoxon test predictive power of the model but only its size. for paired samples (Table S8–S10). The statistically significant Five of the selected architectures required only slight modifi- differences (after Bonferroni correction) were indicated for 86%, cations in their already available codes, and seven were imple- 60% and 62% comparisons for groups of architectures, training mented based on the information provided by the authors either and benchmark data sampling methods, respectively. in their articles or personal communication. A comprehensive Unquestionably, the greatest differences in AUC are associ- description of the implementations and the applied modifications ated with the architecture indicating that this component is are provided in the Supplementary Data, including Table S3. more important than the training and benchmark data sam- pling method for model performance. Among the five architec- tures with the median AUC value greater than 0.9, there were Results three using RF (A:AmpGram, A:MACREL and A:MLAMP), one SVM The impact of data sampling on benchmarks (A:ampir) and one deep learning algorithm (A:AMPScannerV2) In order to evaluate the impact of data sampling on bench- (Figure 2B). The results emphasize the power of RF-based archi- mark analysis, the receiver operating characteristic (ROC) curves tectures in tackling the problem of AMP prediction. The only RF were plotted (Figure S9–S69) and values of the area under the architecture that stood out with small median AUC value was ROC curve (AUC) were calculated for each of 660 models on A:AmPEP, which most probably results from the reduction of each benchmark data set and then averaged for the appropri- its feature space to 23; A:AmPEPy, a python implementation of ate architecture. The results of the analysis are presented in A:AmPEP, with the full feature set of 105 performed quite well. Figure 2(A), S70 and S71. They clearly indicate that all but two The best architecture was A:AmpGram with the median AUC architectures, A:AmPEP and A:iAMP-2L, performed much better value of 0.93 and the narrowest box indicating low variance of when the training and benchmark samples were generated by AUC obtained for various combination of training and benchmark the same sampling method. A:SVM-LZ, A:AmpGram and A:CS- data (Figure 2B). About 73% models of this architecture obtained AMPPred showed only small improvement of 2.3%, 3.6% and 4.4%, AUC greater than 0.9. The A:AmpGram performance suggests respectively; however, A:AmpGram by far outperformed the other that short amino acid motifs might have greater discriminatory architectures. The mean value of AUC for A:AMAP and A:MACREL power than global amino acid composition or physicochemical increased 7.5%, for A:ampir and A:AmPEPpy about 9.5% and for and structural properties, such as charge or tendency to form α- the rest architectures soared more than 10% (Table S6). A:AmPEP helices. The existence of well-conserved motifs typical for AMPs, and A:iAMP-2L were the only architectures, which preferred dis- e.g. lysine-tryptophan motifs, supports our observation. These similar sampling methods for training and benchmarking, but motifs cannot be replaced without reducing the antimicrobial both generally performed very poorly with mean AUC amounting propensity even if the global amino acid composition stays the to 0.65. The calculated differences were statistically significant for same [55]. all comparisons but for iAMP-2L and SVM-LZ (Kruskal–Wallis test We also noticed a certain trend in the distribution of with Bonferroni correction, P-value < 0.05, Table S7). AUC for the training data sampling methods (Figure 2C). The main conclusion from these analyses is that similar train- The AUC values calculated for TSM:AmpGram, TSM:AMPlify, ing and benchmark data set positively affect model performance. TSM:AMPScannerV2, TSM:Gabere&Noble and TSM:Wang et. al Accordingly, there was significant negative correlation between were generally higher than for TSM:AMAP, TSM:ampir-mature and mean AUC value and the difference in amino acid composi- TSM:Witten&Witten, and the lowest AUC values were for TSM:CS- tion between the training and benchmark sets, measured as the AMPPred, TSM:dbAMP and TSM:iAMP-2L. Interestingly, the first square root of the sum of the squared differences in the frequency five sampling methods produced data sets similar in terms of of individual amino acids (Spearman correlation coefficient, ρ length distribution (Figure S1), amino acid composition (Figure = −0.53, P-value < 2.2e-16). There was also smaller but still S2–S6) and physicochemical properties (Figure S7) that deviated significant negative correlation for mean AUC and the absolute from the other sets, especially those with the lowest median AUC. difference between median length of the sets (ρ = −0.44, P-value Given that there was significant negative correlation between < 2.2e-16). mean AUC value and the difference in amino acid composition and median length between the training and benchmark sets (see The impact of architecture, training and above), it is not surprising that architectures trained and bench- benchmark data sampling method on model marked on the five similar sampling methods outperformed the performance others. They simply were advantaged classifying benchmark To visualize which of the three components of the machine learn- sequences in accordance with our finding that similar training ing model, architecture, training or benchmark data sampling and benchmark sample positively affect model performance. method, bears the greatest importance for model performance, Contrary to the results presented in the prior paragraph, we compared box plots of AUC distribution for each of these the sampling methods that performed worse as training sets features (Figure 2B–D). The plots clearly indicate the greatest (SM:dbAMP, SM:iAMP-2L, SM:CS-AMPPred and also SM:AMAP) variation of AUC for data grouped according to the architecture. turned out with the highest AUC as benchmark samples We also calculated the ratio of between-group median absolute (Figure 2D). This can be explained by the fact that these methods deviation (MAD) to within-group MAD to verify if the AUC disper- generate sequences that not only differ from AMPs of the positive sal between different architectures or training/benchmark data set in the amino acid content and physicochemical properties but 8 | Sidorczuk et al. Figure 2. Model performance depending on the architecture and negative data sampling method used for training and benchmarking. The x-axis represents mean AUC for architectures trained and tested on sets generated by the same negative data sampling method. The y-axis represents mean AUC for architectures trained and tested on sets generated by different negative data sampling methods. The architectures on the right of the diagonal perform better when the training and benchmark sample are produced by the same method, while the architectures on the left when the methods are different (A). Box plots with median and interquartile range differences in AUC for architectures (B), training data set sampling method (C) and benchmark data set sampling method (D). are also generally much longer (Figure S1). The median values for similar in terms of length (Figure S1), amino acid composition sequences of SM:dbAMP, SM:iAMP-2L, SM:CS-AMPPred, SM:AMAP (Figure S2–S6) and physicochemical properties (Figure S7), they and the positive sample are: 79, 79, 72, 36 and 18, respectively. did affect the performance of our investigated architectures Accordingly, we found significant positive correlation between (Figure 3), especially A:iAMP-2L, A:AMPScannerV2 and A:Deep- mean AUC and differences in the median length of the benchmark AmPEP30, for which mean standard deviation (SD) of AUC value negative data sets and the benchmark positive sample (Spearman amounted to 0.035, 0.019 and 0.014, respectively (Table S11). correlation coefficient, ρ = 0.74, P-value = 8.63e-11). A:iAMP-2L generally resulted in poorly performing models and the majority of them were characterized by low repeatability. Repeatability of prediction for the replicates Similarly, A:Deep-AmPEP30 was not a very robust architecture of data sets and some models trained on the replicated data also produced To verify the repeatability of prediction, each architecture was significantly different AUC values, e.g. those trained on SM:ampir- trained and benchmarked on five replicates of the training and mature, SM:AMPScannerV2 and SM:AMPlify, and benchmarked benchmark sample. Despite the fact that the replicates were very on SM:CS-AMPPred, SM:dbAMP and SM:iAMP-2L. In contrast, Benchmarks in antimicrobial peptide prediction | 9 Figure 3. Architecture performance depending on the negative data sampling method used for training and benchmarking. Each of 12 heat maps represents an architecture, the x-axis and y-axis describe the training and benchmark method of negative data sampling, respectively. Each architecture was trained and benchmarked on five replicates of the training and benchmark sample. The mean value of AUC for the five replicates is indicated as shades of red, orange and yellow, and the standard deviation as black dots of varying sizes. The diagonals mark results for architectures trained and benchmarked on the data generated by the same sampling method. A:AMPScannerV2 was rather at the forefront of the investigated over deep learning architectures (A:AMPScannerV2 and A:Deep- architectures, especially if the training and benchmark set were AmPEP30) in tackling our data. generated by the same sampling method (Figure 2A, B). In the We also noticed a certain trend in the distribution of AUC case of A:AMPScannerV2, there is a clear pattern of overlapping ref lecting a previously formulated conclusion that similar train- low AUC and high SD distribution for models trained on replicated ing and benchmark data set positively affect model performance. data generated by SM:ampir-mature, SM:CS-AMPPred, SM:dbAMP, It was less noticeable for poorly performing architectures: A:iAMP- SM:iAMP-2L and also SM:Witten&Witten (Figure 3). It looks like 2L, A:AmPEP and A:SVM-LZ, and A:AmpGram representing the top these training sets were not enough for deep learning models to architecture in our studies (Figure 3). learn features necessary to classify AMPs and non-AMPs correctly. This result concurs with other works reporting that shallow models have similar performance to deep ones for AMP prediction Discussion and conclusions [56]. Machine learning represents the most cost-effective method for It is worth emphasizing that the most stable architectures novel AMP discovery. As a result, many computational tools for included also the best ones: A:AmpGram, A: MACREL and A:ampir, AMP prediction have been developed in recent years [17] and as well as A:AmPEPpy (Table S11, Figure 2A, 2B, 3, S70). Their each subsequent state-of-the-art model claims to outperform its mean SD of AUC value amounted to about 0.004. A:ampir imple- predecessors. As a rule, the state-of-the-art model is evaluated mented SVM and the rest RF algorithm indicating their superiority with other software on a benchmark sample generated by the 10 | Sidorczuk et al. same method that was also used to produce its training set (Figure 1A). According to the presented research, this is a source of statistically significant bias in favour of the state-of-the-art Key Points model because the more similar the training and benchmark • We review the performance of existing machine learning data set are the better the model performance (Figure 2A, 2B, 3, models for identification of AMPs. S70). Consequently, we came to logical conclusions that (i) all the • Our benchmark highlights the major methodological benchmark analyses that have been published for AMP prediction f law in the construction of benchmarks as the data tools are unfair and (ii) we do not know which model is the most sampling impacts the quality of AMP prediction. accurate. • We propose a solution for fair benchmarking of AMP- To provide researchers with reliable information about the predicting models. performance of AMP predictors, we created a web server AMP- Benchmark for fair benchmarking of AMP prediction models. Similarly to Kaggle, AMPBenchmark provides developers with public and private data sets for model training and validation that contain explicit and hidden data labels, respectively. The public Acknowledgments data sets are the same samples that were used in the presented We thank Henrik Nielsen (Technical University of Denmark) and research. AMPBenchmark allows users to upload the prediction Simon Rasmussen (University of Copenhagen) for their critical results for their AMP models, trained and benchmarked on the review of the draft and fruitful discussions. public data sets. It generates charts and tables comparing the performance of the uploaded architecture with those deposited in our database. The users can also upload prediction results for Funding their AMP models, trained and benchmarked on the private data This work was partially supported by the National Science Centre set, which is accessible after entering the e-mail address. The grant 2017/26/D/NZ8/00444 to P.G., the National Science Centre operator of AMPBenchmark will manually verify the results of the grant 2018/31/N/NZ2/01338 to K.S., the National Science Centre prediction and similarly provide charts and tables for comparative grant 2019/35/N/NZ8/03366 to F.P. and the IDUB against COVID- analysis. 19 project granted by Warsaw University of Technology under the Our study has also vital importance for the ongoing debate program Excellence Initiative: Research University (IDUB) to M.B. about the reproducibility crisis in science [57, 58]. In machine M. B. was partially supported by the Maria Zambrano grant funded learning research, reproducibility means obtaining the same by the European Union-NextGenerationEU. results to those presented in the original study using the same data and source code. Recently, Heil et al. [59] proposed three standards for computational reproducibility: bronze, silver and gold, ref lecting the time needed to recreate research. The minimal Code, software and data availability and most time-consuming bronze standard requires: (i) data, (ii) The code necessary to reproduce the whole analysis is avail- models and (iii) source code to be published and downloadable. able at https://github.com/BioGenies/NegativeDatasets. All From the 26 models for AMP prediction that we considered, only architectures are located at https://github.com/BioGenies/ eight met the minimal bronze standard (Table S2). This means NegativeDatasetsArchitectures. AMPBenchmark web server is that about 70% models represented non-reproducible work and available at http://BioGenies.info/AMPBenchmark. consequently are unreliable. Interestingly, this number is very consistent with the survey published in the journal Nature [58] indicating that more than 70% researchers failed to reproduce References other group’s experiments. Among the implemented models, five met the bronze standard: AmPEP, AmPEPpy, AmpGram, ampir and 1. Maróti G, Kereszt A, Kondorosi E, et al. Natural roles of antimi- MACREL, and AmpScannerV2 was accessible upon request. These crobial peptides in microbes, plants and animals. Res Microbiol architectures, with the exception of A:AmPEP, also represent the 2011;162(4):363–74. top architectures investigated though A:AmpGram was clearly 2. Magrone T, Russo MA, Jirillo E. Antimicrobial peptides: phylo- the most accurate and best at generalizing to other data sets. genic sources and biological activities. First of two parts. Curr The developers that do not reveal all the details necessary Pharm Des 2018;24(10):1043–53. to recreate their models, not to mention reuse them, shoulder 3. Raffatellu M. Learning from bacterial competition in the host to the blame for the lack of fair benchmarks for AMP prediction develop antimicrobials. Nat Med 2018;24(8):1097–1103. software. Consequently, progress in the field is slowed, mistrust 4. Garima Suneja, Sonam Nain, and Rakesh Sharma. Microbiome: to bioinformatics is spreading and resources that could have been A source of novel bioactive compounds and antimicrobial pep- allocated to other projects are wasted. Our study represents the tides. In: Tulasi Satyanarayana, Bhavdish Narain Johri, Subrata first unbiased approach to compare models for AMP prediction, Kumar Das (eds) Microbial Diversity in Ecosystem Sustainability and and moreover, we made reproducible another six model architec- Biotechnological Applications, pages 615–30. Springer Singapore, tures for further research. In total, we built a staggering number 2019. of 660 machine learning models from 12 architectures. Therefore, 5. Ahmed A, Siman-Tov G, Hall G, et al. Human antimicrobial pep- being fully aware of the difficulty of the task, we highly recom- tides as therapeutics for viral infections. Viruses 2019;11(8):704. mend all researchers to embrace the notion of fair benchmarking 6. Mookherjee N, Anderson MA, Haagsman HP, et al. Antimicrobial and reproducibility using AMPBenchmark web server and the host defence peptides: functions and clinical potential. Nat Rev recommendations provided by Heil et al. [59]. Drug Discov 2020;19(5):1–22. Benchmarks in antimicrobial peptide prediction | 11 7. Li J, Koh J-J, Liu S, et al. Membrane active antimicrobial pep- association in acute myeloid leukemia cancer. PLoS One tides: translating mechanistic insights to design. Front Neurosci 2019;14(12):e0226115. 2017;11:73. 29. Bugnon LA, Yones C, Milone DH, et al. Genome-wide discovery of 8. Travkova OG, Moehwald H, Brezesinski G. The interaction of pre-mirnas: comparison of recent approaches based on machine antimicrobial peptides with membranes. Adv Colloid Interface Sci learning. Brief Bioinformatics 2021;22(3):bbaa184. 2017;247:521–32. 30. Rentzsch R, Deneke C, Nitsche A, et al. Predicting bacterial 9. Kumar P, Kizhakkedathu JN, Straus SK. Antimicrobial peptides: virulence factors–evaluation of machine learning and negative diversity, mechanism of action and strategies to improve the data strategies. Brief. Bioinformatics 2020;21(5):1596–608. activity and biocompatibility in vivo. Biomolecules 2018;8(1):4. 31. Ben-Hur A, Noble WS. Choosing negative examples for the 10. Ahmed TAE, Hammami R. Recent insights into structure– prediction of protein-protein interactions. BMC Bioinform function relationships of antimicrobial peptides. J Food Biochem 2006;7(1):1–6. 2019;43(1):e12546. 32. Cheng Z, Zhou S, Guan J. Computationally predicting protein- 11. Le C-F, Fang C-M, Sekaran SD. Intracellular targeting mecha- RNA interactions using only positive and unlabeled examples. J nisms by antimicrobial peptides. Antimicrob Agents Chemother Bioinform Comput Biol 2015;13(03):1541005. 2017;61(4):e02340–16. 33. Le NQK, Do DT, Le QA, et al. A sequence-based prediction of 12. Li S, Wang Y, Xue Z, et al. The structure-mechanism relationship kruppel-like factors proteins using xgboost and optimized fea- and mode of actions of antimicrobial peptides: a review. Trends tures. Gene 2021;787:145643. Food Sci Technol 2021;109:103–15. 34. Hung TNK, Le NQK, Le NH, et al. An ai-based prediction model 13. Andersson DI, Hughes D, Kubicek-Sutherland JZ. Mechanisms for drug-drug interactions in osteoporosis and paget’s diseases and consequences of bacterial resistance to antimicrobial pep- from smiles. Molecular Informatics 2022;41(6):2100264. tides. Drug Resist Updat 2016;26:43–57. 35. Song H, Bremer BJ, Hinds EC, et al. Inferring protein sequence- 14. Lázár V, Martins A, Spohn R, et al. Antibiotic-resistant bacteria function relationships with large-scale positive-unlabeled show widespread collateral sensitivity to antimicrobial peptides. learning. Cell Syst 2021;12(1):92–101. Nat Microbiol 2018;3(6):718. 36. Bekker J, Davis J. Learning from positive and unlabeled data: a 15. Spohn R, Daruka L, Lázár V, et al. Integrated evolutionary anal- survey. Mach Learn 2020;109(4):719–60. ysis reveals antimicrobial peptides with limited resistance. Nat 37. Li F, Dong S, Leier A, et al. Positive-unlabeled learning in bioin- Commun 2019;10(1):1–13. formatics and computational biology: a brief review. Brief Bioin- 16. WHO et al. No time to wait: securing the future from drug-resistant formatics 2021. infections. Geneva, Switzerland: World Health Organization, 2019. 38. Berman HM, Westbrook J, Feng Z, et al. The protein data bank. 17. Xu J, Li F, Leier A, et al. Comprehensive assessment of machine Nucleic Acids Res 2000;28(1):235–42. learning-based methods for predicting antimicrobial peptides. 39. Li W, Godzik A. Cd-hit: a fast program for clustering and compar- Brief. Bioinformatics 2021;22(5):bbab083. ing large sets of protein or nucleotide sequences. Bioinformatics 18. Burdukiewicz M, Sidorczuk K, Rafacz D, et al. Proteomic screen- 2006;22(13):1658–9. ing for prediction and design of antimicrobial peptides with 40. Limin F, Niu B, Zhu Z, et al. CD-HIT: accelerated for clus- AmpGram. Int J Mol Sci 2020;21(12):4310. tering the next-generation sequencing data. Bioinformatics 19. Pirtskhalava M, Amstrong AA, Grigolava M, et al. DBAASP v3: 2012;28(23):3150–2. database of antimicrobial/cytotoxic activity and structure of 41. Wang P, Lele H, Liu G, et al. Prediction of antimicrobial peptides peptides as a resource for development of new therapeutics. based on sequence alignment and feature selection methods. Nucleic Acids Res 2021;49(D1):D288–97. PLoS One 2011;6(4):e18476. 20. Wang G, Li X, Wang Z. APD3: the antimicrobial peptide 42. Porto WF, Pires ÁS, Franco OL. CS-AMPPred: an updated SVM database as a tool for research and education. Nucleic Acids Res model for antimicrobial activity prediction in cysteine-stabilized 2016;44(D1):D1087–93. peptides. PLoS One 2012;7(12):e51444. 21. Waghu FH, Barai RS, Gurung P, et al. CAMPR3: a database on 43. Xuan Xiao P, Wang W-ZL, Jia J-H, et al. iAMP-2L: a two-level multi- sequences, structures and signatures of antimicrobial peptides. label classifier for identifying antimicrobial peptides and their Nucleic Acids Res 2016;44(D1):D1094–7. functional types. Anal Biochem 2013;436(2):168–77. 22. Kang X, Dong F, Cheng S, et al. DRAMP 2.0, an updated data 44. Gull S, Shamim N, Minhas F. AMAP: hierarchical multi-label pre- repository of antimicrobial peptides. Sci Data 2019;6(1):1–10. diction of biologically active and antimicrobial peptides. Comput 23. Jhong J-H, Chi Y-H, Li W-C, et al. dbAMP: an integrated resource Biol Med 2019;107:172–81. for exploring antimicrobial peptides with functional activities 45. James Kent W. BLAT-the BLAST-like alignment tool. Genome Res and physicochemical properties on transcriptome and proteome 2002;12(4):656–64. data. Nucleic Acids Res 2019;47(D1):D285–97. 46. Veltri D, Kamath U, Shehu A. Deep learning improves antimicro- 24. Fingerhut LCHW, Miller DJ, Strugnell JM, et al. Ampir: an R pack- bial peptide recognition. Bioinformatics 2018;34(16):2740–7. age for fast genome-wide prediction of antimicrobial peptides. 47. Witten J, Witten Z. Deep learning regression model for antimi- Bioinformatics 2020;36(21):5262–3. crobial peptide design. BioRxiv 2019;692681. 25. Gabere MN, Noble WS. Empirical comparison of web- 48. Li C, Sutherland D, Hammond SA, et al. AMPlify: attentive based antimicrobial peptide prediction tools. Bioinformatics deep learning model for discovery of novel antimicrobial pep- 2017;33(13):1921–9. tides effective against who priority pathogens. BMC Genomics 26. UniProt Consortium. Uniprot: the universal protein knowledge- 2020;23:77. https://doi.org/10.1186/s12864-022-08310-4. base in 2021. Nucleic Acids Res 2021;49(D1):D480–9. 49. Yan J, Bhadra P, Li A, et al. Deep-AmPEP30: improve short antimi- 27. Yang P, Li X-L, Mei J-P, et al. Positive-unlabeled learning for crobial peptides prediction with deep learning. Mol Ther Nucleic disease gene identification. Bioinformatics 2012;28(20):2640–7. Acids 2020;20:882–94. 28. Vasighizaker A, Sharma A, Dehzangi A. A novel one-class 50. Ng XY, Rosdi BA, Shahrudin S. Prediction of antimicrobial classification approach to accurately predict disease-gene peptides based on sequence alignment and support vector 12 | Sidorczuk et al. machine-pairwise algorithm utilizing LZ-complexity. Biomed Res 55. Gopal R, Seo CH, Song PI, et al. Effect of repetitive lysine- Int 2015;2015:212715. tryptophan motifs on the bactericidal activity of antimicrobial 51. Lin W, Dong X. Imbalanced multi-label learning for identifying peptides. Amino Acids 2013;44(2):645–60. antimicrobial peptides and their functional types. Bioinformatics 56. García-Jacas CR, Pinacho-Castellanos SA, García-González 2016;32(24):3745–52. LA, et al. Do deep learning models make a difference in 52. Bhadra P, Yan J, Li J, et al. AmPEP: sequence-based prediction of the identification of antimicrobial peptides? Brief Bioinform antimicrobial peptides using distribution patterns of amino acid 2022;23(3):bbac094. properties and random forest. Sci Rep 2018;8(1):1–10. 57. Ioannidis JPA. Why most published research findings are false. 53. Lawrence TJ, Carper DL, Spangler MK, et al. amPEPpy 1.0: a PLoS Med 2005;2(8):e124. portable and accurate antimicrobial peptide prediction tool. 58. Baker M. 1,500 scientists lift the lid on reproducibility. Nature Bioinformatics 2020;37(14):2058–60. 2016;533(7604):452–4. 54. Santos-Junior CD, Pan S, Zhao X-M, et al. MACREL: antimi- 59. Heil BJ, Hoffman MM, Markowetz F, et al. Reproducibility stan- crobial peptide screening in genomes and metagenomes. PeerJ dards for machine learning in the life sciences. Nat Methods 2020;8:e10555. 2021;18(10):1132–5.