González‑Pérez et al. Parasites & Vectors https://doi.org/10.1186/s13071-022-05324-5 (2022) 15:190 Parasites & Vectors RESEARCH Open Access A novel optical sensor system for the automatic classification of mosquitoes by genus and sex with high levels of accuracy María I. González‑Pérez1†, Bastian Faulhaber2†, Mark Williams2, Josep Brosa1, Carles Aranda1,3, Nuria Pujol1, Marta Verdún1, Pancraç Villalonga2, Joao Encarnação2, Núria Busquets1† and Sandra Talavera1*† Abstract Background: Every year, more than 700,000 people die from vector-borne diseases, mainly transmitted by mosqui‑ toes. Vector surveillance plays a major role in the control of these diseases and requires accurate and rapid taxo‑ nomical identification. New approaches to mosquito surveillance include the use of acoustic and optical sensors in combination with machine learning techniques to provide an automatic classification of mosquitoes based on their flight characteristics, including wingbeat frequency. The development and application of these methods could enable the remote monitoring of mosquito populations in the field, which could lead to significant improvements in vector surveillance. Methods: A novel optical sensor prototype coupled to a commercial mosquito trap was tested in laboratory conditions for the automatic classification of mosquitoes by genus and sex. Recordings of > 4300 laboratory-reared mosquitoes of Aedes and Culex genera were made using the sensor. The chosen genera include mosquito species that have a major impact on public health in many parts of the world. Five features were extracted from each recording to form balanced datasets and used for the training and evaluation of five different machine learning algorithms to achieve the best model for mosquito classification. Results: The best accuracy results achieved using machine learning were: 94.2% for genus classification, 99.4% for sex classification of Aedes, and 100% for sex classification of Culex. The best algorithms and features were deep neural network with spectrogram for genus classification and gradient boosting with Mel Frequency Cepstrum Coefficients among others for sex classification of either genus. Conclusions: To our knowledge, this is the first time that a sensor coupled to a standard mosquito suction trap has provided automatic classification of mosquito genus and sex with high accuracy using a large number of unique samples with class balance. This system represents an improvement of the state of the art in mosquito surveillance and encourages future use of the sensor for remote, real-time characterization of mosquito populations. † María I. Gonzalez-Perez and Bastian Faulhaber contributed equally to this work † Núria Busquets and Sandra Talavera contributed equally to this work *Correspondence: sandra.talavera@irta.cat 1 IRTA, Centre de Recerca en Sanitat Animal (CReSA, IRTA-UAB), Campus de la Universitat Autònoma de Barcelona, Bellaterra, Cerdanyola del Vallès, Spain Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​ mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 2 of 11 Keywords: Mosquito trap, Automatic classification, Optical sensor, Machine learning, Deep learning, Aedes, Culex, Genus and sex classification, Mosquito surveillance Background to entomological surveillance include novel optical sen- Approximately 80% of the world’s human population sors to sense the characteristics of flying mosquitoes and lives with the risk of one or more vector-borne diseases analysis methods including machine learning methods to (VBDs), and every year > 700,000 people die as a result enable classification of mosquitoes in near real-time [15– [1]. In an increasingly connected world, travel and trade 19], which is crucial for surveillance programs. contribute to the spread of VBDs. Furthermore, a global Since the 1940s, microphones have been used to warming scenario may lead to more favourable condi- sense the audible flight tones emitted by flying mosqui- tions for the survival and life cycle completion of the vec- toes, which may be associated with a particular mos- tors [2] and may affect their abundance and distribution quito genus, species or sex [20]. Acoustic methods are [3]. Mosquitoes (Diptera: Culicidae), particularly those still employed today in applications such as sound traps, belonging to Aedes, Anopheles and Culex genera, are one which emit species- and sex-specific sound frequencies of the deadliest vectors worldwide. Mosquito species to attract mosquitoes [21], and in classification systems can transmit diseases such as malaria, dengue, yellow such as those in which citizen scientists use their mobile fever, West Nile fever, Zika, chikungunya and others [4]. phones to record mosquitoes [22, 23]. However, it is hard According to World Health Organization directives [5] to obtain acceptable quality audio recordings of free-fly- and European Centre for Disease Prevention and Con- ing insects in the field because of the presence of back- trol guidelines [6, 7], appropriate surveillance methods ground noise [18]. To address this, optical methods have and indicators are needed to: determine the composition been employed in which a light source is used to illumi- and monitor changes in mosquito populations, identify nate the flying insect and a light sensor is used to detect the presence of new invasive species, monitor mosquito- the light reflected and scattered, or attenuated, by the borne diseases, quantify the transmission potential insect in flight [24–30]. The use of optical methods in this of vectors and enable the design of accurate control field began in 1955 when a photoelectric cell was used to programmes. detect the light modulation produced by a flying insect A range of insect trap types and methods are used in crossing its field of view [31]. In recent years, several regular monitoring and surveillance of immature and/ optoelectronic sensors have been developed and used in or adult mosquito populations [8]. Although immature conjunction with machine learning techniques to classify stage monitoring can be easier to set up, it is not useful flying mosquitoes, with promising levels of accuracy [16, for estimating adult abundance due to the lack of corre- 17, 32–34]. lation between egg, larval and pupal density indices and Variables known to condition mosquito wingbeat fun- adult indices [9]. Studies show that the seasonal variation damental frequency or its detection include taxonomy, in mosquito abundance is better represented by adult sex, parity status, size, age, environmental temperature trap monitoring than by other indices (e.g. House Index) or wind speed [35–39]. Historically, wingbeat frequency based on immature stages [10]. Therefore, adult mos- has been used as the only predictor variable for mosquito quito surveillance is generally the most widely applicable classification, but it appears insufficient on its own to dif- and accurate solution, especially for VBD risk assessment ferentiate between mosquito species, especially those of [11]. Many adult mosquito monitoring systems rely on the same genus [18]. This could limit field applications, traps using light, chemical attractants or C ­ O2 as a bait. where different mosquito species can coexist, with the Most traps include a suction fan to draw approaching possibility of overlap in wingbeat frequency distributions insects into a catch bag within the trap, and such types [40]. Efforts have been made in recent years to improve have been successfully used in many studies [12–14]. classification methods to distinguish among mosquito However, they require the catch bag to be periodically species, sex and even parity status [16, 17, 35]. In some collected in the field, followed by a time-consuming pro- cases, more advanced optical approaches have been used, cess of identification of the collected specimens by ento- for example to determine insect body and wing depo- mologists. The time delay between insect trapping and larization ratio, to improve the accuracy of classification analysis may limit the correct characterization of the [17]. temporal dynamics of mosquito populations. Such delays In addition to the selection of the proper predictor var- may also result in degradation of the insects in the catch iables and machine learning algorithms, the use of meta- bag because of desiccation or predation. New approaches data such as the climatic conditions, the spatiotemporal González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 3 of 11 localization and other ecological features accompanying iii. Culex pipiens, population of Gavà (2012), Barce- mosquito captures may also be relevant for remote mos- lona, Spain (41.3000°, 2.0167°). quito classification in the field [18, 33], since different mosquito species have different behaviour and ecological The mosquito populations were all reared under con- needs (e.g. geographical distribution, climatic range, cir- trolled environmental conditions in a climatic cham- cadian rhythm, and peaks of activity). According to new ber at a temperature of 28 °C and a relative humidity of paradigms of remote mosquito surveillance, wingbeat 80%, with a light:dark photoperiod of 12:12 h, except sensor information and metadata could be sent wirelessly for Cx. pipiens (with a light:dark photoperiod of 11:11 h in real time to a server using Internet of Things (IoT) tech- plus 1 h of dusk and 1 h of dawn). Culex pipiens and nology [41–43] with the potential to improve entomolog- Ae. albopictus were reared in a biosafety level 2 (BSL2) ical surveillance. laboratory and Ae. aegypti in a biosafety level 3 (BSL3) Currently, there is only one commercial optical sensor laboratory at IRTA-CReSA facilities. Larvae were main- product available for the remote monitoring of mosquito tained in plastic trays with 750 ml of dechlorinated tap populations [41]. It is called the BG-Counter (Biogents, water (renewed three times per week) and were fed with Germany), which according to the company, can distin- fish pellets (Goldfish Sticks-TETRA, Melle, Germany) guish mosquitoes from other insects and count mosqui- ad libitum. Pupae, upon appearance, were immediately toes. However, the sensor does not provide information placed in insect cages (BugDorm-1 Insect Rearing Cage about mosquito genus, species, sex or other attributes. W30 × D30 × H30 cm, MegaView Science, Talchung, In this study, we present the results of a prototype Taiwan). After metamorphosis, adults were fed with optical sensor, which is coupled to the entrance of a sucrose solution (10%) ad libitum. Females were not fed commercial mosquito trap. The trap is of a type widely with blood to avoid any body size or flight variation. For used for mosquito surveillance in the field and contains Aedes females, the sucrose solution was removed 24 h a suction fan. The fan causes the mosquitoes to pass before the sensor tests. For Cx. pipiens females, this was through the sensor more quickly and with a more per- done 48 h before to improve their affinity for the attract- turbed wingbeat compared to free flight conditions as ant used in the trap. described in another work [39]. For the present work, 4335 flights from mosquitoes of Aedes and Culex genera Sensor and trap description were recorded using the sensor. The three species for the The prototype sensor was designed and produced by study, Aedes albopictus, Aedes aegypti and Culex pipiens, Irideon SL (Barcelona, Spain) and was coupled to the were chosen because they are major vectors of arbovi- entrance of a commercial BG-Mosquitaire suction trap ruses, have a significant impact on public health and are from Biogents AG (Regensburg, Germany), as shown in a focus of vector surveillance and control programs in Fig. 1a. many parts of the world. A set of features were extracted The trap coupled to the sensor was placed from each recording and used to train a series of machine in an insect-rearing cage (BugDorm-4S4590 learning algorithms to determine which combination of W47.5 × D47.5 × H93.0 cm, MegaView Science, Tal- feature and algorithm gave the best performance in clas- chung, Taiwan). The trap was fitted with a sachet of BG- sifying mosquitoes by genus and sex. Whilst the scope of Sweetscent chemical attractant from Biogents AG. The this work is limited to the classification of genus (Aedes/ air flow generated by the fan was approximately 3 m/s in Culex) and sex (female/male), the inclusion of the two the downward direction. When a mosquito flies close to Aedes species in this study improves the genetic vari- the entrance funnel of the sensor, it may be sucked in by ability and permits future work on species classification the fan, detected by the sensor and then trapped in the using the data set from the present work. catch bag inside the body of the trap, as shown in Fig. 1b. The sensor contains an optical emitter panel and an optical receiver panel, which face each other through a Methods transparent flight tube with a diameter of 105 mm. The Mosquito rearing conditions optical emitter comprises a two-dimensional (2D) array As stated, three species of mosquitoes, from two genera, of 940-nm wavelength infrared light-emitting diodes were used to generate the dataset: (LEDs), and the optical receiver comprises a 2D array of 940-nm photodiodes. The optical sensor has an active i. Aedes albopictus, population of Sant Cugat del length of 70 mm in the downward direction. These ele- Vallès (2005), Barcelona, Spain (41.4667°, 2.0833°). ments are also shown in Fig. 1b. ii. Aedes aegypti, population of Paea (1994), Tahiti, The output of the optical receiver is amplified and French Polynesia (− 17.6889°, − 149.5869°). acquired by an analog to the digital converter (ADC) González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 4 of 11 Fig. 1 a Prototype sensor (top) fitted to a BG-Mosquitaire trap (bottom). b Side view diagram of sensor and trap to illustrate operation. The exterior of the sensor unit (1) is formed by an inlet tube with a diameter of approximately 100 mm (2), sensor housing (3) and outlet tube (4). The housing contains an optical emitter (5), which projects collimated beams of light through the transparent flight tube (6) and onto an optical receiver (7) to create a sensing zone (8) within the flight tube. The trap (9) contains a suction fan (10), a removable catch bag (11) made of textile mesh and a perforated lid (12). The fan produces a flow of air downward through the inlet tube, flight tube and catch bag and upward through the perforated lid as indicated by the blue arrows. An insect (13) which flies close to the entrance of the inlet tube may then be sucked downwards through the sensing zone where it will be recorded and then trapped in the catch bag. As the mosquito passes through the sensing zone it casts a shadow upon the optical receiver according to the so-called optical extinction mode of operation. As the insect flaps its wings within the sensing zone, the light falling on the optical receiver is modulated, giving rise to changes in the amplitude in the recorded waveform with a sampling frequency of 9603 samples per second. approached it and were sucked in to approximate field When a mosquito enters the sensing volume, it automati- conditions. cally triggers a recording of up to 1024 samples, i.e. of up Each recording corresponds to a different mosquito, i.e. to 107 ms duration. The duration of a typical mosquito trapped mosquitoes were not re-used to generate more flight is around 50 ms. The sensor automatically adds a recordings. Wingbeat files were tagged with species and timestamp to each recording, along with the measured sex class by the operator. After each experiment, the ambient temperature. wingbeat recordings were downloaded from the sensor and processed using a Python script to produce playable Data acquisition process and viewable audio files, as depicted in Fig. 2a. Wingbeat Mosquitoes from Aedes and Culex genera were anesthe- recordings were examined manually, and those deemed tized with carbon dioxide 48 and 72 h respectively before to be invalid, such as recordings containing more than each experiment. They were separated into groups by one mosquito or where a mosquito may have hit the wall species (Cx. pipiens, Ae. albopictus and Ae. aegypti) and of the flight tube, were excluded from the dataset. The sex (male, female). excluded recordings represented 2.3% of the data. Culex pipiens and Ae. albopictus were introduced The resulting dataset contained 4335 wingbeat record- into the insect rearing cage in batches of 20 individu- ings, comprising 2472 of Aedes genus (882 Ae. aegypti als to reduce the chance of multiple mosquitoes passing and 1590 Ae. albopictus) and 1863 of Culex genus (all through the sensor simultaneously. Batches of ten indi- Cx. pipiens). There were 1211 Aedes females, 1261 Aedes viduals were used for Ae. aegypti because of their greater males, 964 Culex females and 899 Culex males. Females affinity to the attractant. All mosquitoes were intro- were in an age range of 2 to 16 days old and males were in duced at a distance of 20 to 30 cm from the entrance of an age range of 2 to 9 days old. These age ranges provide the sensor to ensure that they could fly freely until they a representative variety in the dataset. González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 5 of 11 Fig. 2 a Example of a recorded mosquito flight with ADC sample number (0 to 1023) on the x-axis and amplitude on the y-axis, scaled to a range of [− 1, 1], which equates to the full-scale range of the ADC. A high pass filter in the optical receiver attenuates frequencies < 300 Hz to remove electronic offsets and low-frequency noise, which also attenuates the signal due to the body of the insect. Baseline correction has been applied by subtracting the average value of the recording from each data point in the recording. b Power spectral density (PSD) plot of a typical mosquito flight. The wingbeat fundamental peak is labelled as f1. The fundamental frequency is indicated by the vertical arrow and the fundamental peak power by the horizontal arrow. The various peaks to the right of f1 are harmonics of f1, i.e. at frequencies of 2*f1, 3*f1, etc. The power density has units of ­(units2/Hz) on a logarithmic (dB) scale. A level of 0 dB/Hz corresponds to a white noise signal time domain signal with a power density of 1.0 ­unit2/Hz. The fundamental peak power density levels in this study are typically < − 40 dB/Hz, i.e. < 1 × ­10–4 ­units2/Hz. The noise floor of the system, i.e. with sensor active but with no insect in the sensing zone, is < − 85 dB/Hz from 0 to 300 Hz and < − 90 dB/Hz from 300 Hz All recordings took place with the sensor and trap • Wingbeat fundamental frequency in Hertz (Hz) is located in the laboratory facilities of IRTA-CReSA during determined from the PSD as shown in Fig. 2b using daylight hours. The average ambient temperature meas- a peak search method. The wingbeat fundamental ured was 25.8 (standard deviation = 1.2 °C). frequency is the frequency at which a mosquito flaps its wings. It is characteristic of mosquito taxonomy and sex and varies depending on intrinsic variables Feature extraction of mosquito biology (size, age, parity status, mating The following five features were extracted from each behaviour) [16, 35, 36, 38] and environmental vari- wingbeat recording via the application of digital signal ables such as temperature [37]. The typical range of processing methods: mosquito wingbeat fundamental frequencies is 300 to 900 Hz [40]. • The power spectral density (PSD) shows the power • The fundamental peak power density (dB/Hz) (here- of the signal at different frequencies. It is calculated after referred to as fundamental peak power) is also using Welch’s method [44], in which the wingbeat determined from the PSD as shown in Fig. 2b and recording is divided into several overlapping seg- represents the peak power density of the sensor ments. A windowing function is applied to each output at the wingbeat fundamental frequency. It is of the segments and a series of periodograms is equivalent to the intensity of the sound produced by obtained by calculating the power spectrum of each a flying mosquito, typically ranging from 40 to 80 dB windowed segment. Finally, the periodograms are [46, 47]. averaged to give the PSD [45]. A PSD plot of a typical • The spectrogram is a series of spectra calculated mosquito recording is shown in Fig. 2b. from multiple overlapping segments of the wingbeat González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 6 of 11 recording. Each spectrum is generated by applying a two single-value features, fundamental frequency and Fourier transform to the segment to provide infor- fundamental peak power, for the three classifications are mation about the amplitude of the various frequency shown in Additional file 1: Fig. S2. components in the segment. The spectrogram repre- sents the variations of the frequency content of the Machine learning signal over time rather than an average for the whole The goal of the machine learning process was to com- signal as given by the PSD [48]. pare the performance of five selected machine learning • Mel Frequency Cepstral Coefficients (MFCCs) are algorithms using the features described above, in clas- calculated by converting the frequencies of a spectro- sifying mosquito genus and sex. A labelled dataset con- gram to the Mel scale and applying overlapping tri- sisting of the feature set was used to train, evaluate and angular filter banks before calculating the cepstrum compare the classification models. The following five by transforming the spectra to a logarithmic scale machine learning algorithms were used: logistic regres- and then applying an inverse Fourier transform [49]. sion (LR), gradient boosting (GB), random forests (RF), Please refer to Additional file 1: Text S1 and Fig. S1 support vector machines (SVM) and a fully connected for further details. deep neural network (DNN). These algorithms were cho- sen because of their widespread usage and good perfor- The PSDs have 257 values, generated using a window mance [50]. A brief overview of each algorithm is given in length of 512 samples. The spectrograms and MFCCs are Additional file 1: Text S2. Of these algorithms, the more obtained using nine segments of 512 samples; then, 16 complex ones, such as DNN or RF, were also used with Mel filter banks are applied to each spectrum to give a the single-value features (fundamental frequency and total of 144 values. All the MFCC coefficients are used. fundamental peak power) because they can model non- Each individual feature and one combined feature (fun- linearities, unlike LR. damental frequency and fundamental peak power) were Three classification tasks were performed: one genus used for the machine learning models. classification (Aedes/Culex) and two sex classifications A scatter plot of the wingbeat fundamental frequency (male/female), one for each genus (sex of Aedes, sex of and peak power features is shown in Fig. 3a for the entire Culex). The logic of the classification process is shown in dataset, in Fig. 3b for all Aedes samples and in Fig. 3c for Additional file 1: Fig. S3. all Culex samples. In Fig. 3a, which is coloured by genus, Balanced datasets, i.e. datasets that contained an equal a high degree of overlap between the genera is observed. number of samples in each class, were used to make an In Fig. 3b and c, which are coloured by sex, two clearly unbiased assessment. They were obtained by randomly separated clusters are observed. The distributions of the Fig. 3 a Scatterplot of wingbeat fundamental frequency and peak power for the full dataset showing Aedes genus in red and Culex in blue. b Scatter plot of wingbeat fundamental frequency and peak power for Aedes genus showing females in red and males in blue. c Scatter plot of wingbeat fundamental frequency and peak power for Culex genus showing females in red and males in blue González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 7 of 11 under-sampling the classes which had a higher number the models took days to weeks, but once done, each new of available samples. sample was classified in under 1 s. Model performance was assessed using the accuracy metric, which is calculated by dividing the number of Results correct predictions by the total number of predictions. Genus classification The accuracy metric is a simple evaluation metric, which In the genus classification, mosquitoes were classified makes it easy to interpret, and is appropriate when using into Aedes and Culex genus. A total of 2688 samples were balanced datasets. used comprising: 1344 Aedes (672 Ae. albopictus and The typical machine learning process consists of train- 672 Ae. aegypti) and 1344 Culex (all Cx. pipiens) with an ing, validation and testing. In the training phase, the equal number of males and females for each species. The model is fitted to the data with different configurations dataset was split 75%/25% into the training data set (2016 of the algorithm determined by hyperparameters, which samples) and the test set (672 samples). The accuracy can have a significant impact on performance. In the results for genus classification on the test set are shown validation phase, the performances of the models trained in Table 1 with the best performing algorithm for each with the different configurations are compared and the feature shown in bold. The best result for genus classifi- best one is selected. The testing phase assesses how well cation was obtained for the DNN algorithm trained on the model generalizes on previously unused data. A sche- the spectrogram feature, with an accuracy of 94.2%. matic overview of the training, validation and testing approach employed in this work is shown in Additional Sex classification of Aedes file 1: Fig. S4. In this classification, mosquitoes of the Aedes genus were Seventy-five percent of the recordings in each dataset classified into males and females. A total of 1344 samples were chosen randomly to create a training set for use in were used, comprising 672 females and 672 males, with the training and validation phase. Training and valida- each sex group comprising 336 Ae. aegypti and 336 Ae. tion were done using fourfold cross-validation, in which albopictus. The dataset was split 75%/25% into the train- the training set is split into four parts of equal size and ing data set (1008 samples) and the test set (336 samples). the model being optimized is trained on three of the The results for this classification on the test set are shown four parts and validated on the fourth part. This process in Table 2. The best performing algorithms for sex classi- is done four times using a different part of the training fication of Aedes were logistic regression trained on spec- set for the cross-validation in each iteration. The final trogram and MFCC, and gradient boosting trained on cross-validation score was obtained by averaging the four MFCC, with an accuracy of 99.4% in each case. cross-validation results. The model with the best cross- validation score was then selected for testing. Sex classification of Culex The remaining 25% of each dataset, i.e. that part which In this classification, mosquitoes of the Culex genus (all was not allocated to training and validation, was used to Cx. pipiens) were separated into males and females. A test the performance of the trained model. Since the data total of 1560 samples were used comprising 780 females in the test set are completely new to the model, accuracy and 780 males. The dataset was split 75%/25% into the results for the test set are an indication of how well the training data set (1170 samples) and the test set (390 model generalizes on new data, and good results cannot samples). The results for this classification on the test be attributed to overfitting of the model. Error analysis consists of analysing the training and validation accuracies obtained during the training and validation phase. If the training accuracy is considerably Table 1 Accuracy results for genus classification with best higher than the validation accuracy, it indicates overfit- results per feature indicated by a superscript letter ting, so more samples could help to improve the model. Feature Algorithm If, on the other hand, training and validation accuracies LR (%) GB (%) RF (%) SVM (%) DNN (%) have a similar low score, it indicates that the model is too simple and that more training data would probably not Fundamental frequency 55.2 67.3a 65.9 65.5 66.1 help. In this case, the model could possibly be improved Fundamental peak power 68.9 70.1a 69.6 69.8 70.0 by using a different algorithm which is able to learn more Fundamental frequency 70.1 77.7 77.2 77.2 77.8a complex relationships or to use more features. and peak power Programming was done in Python [51]. For model gen- PSD 84.8 92.3a 89.0 90.5 90.3 eration, scikit-learn [52], TensorFlow [53] and XGBoost Spectrogram 90.5 93.2 91.2 93.4 94.2a [54] were used. Regarding execution times, training of MFCC 89.3 93.2 a 90.2 93.0 93.2a González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 8 of 11 Table 2 Accuracy results for sex classification of Aedes with best how the results might be improved. The corresponding results per feature indicated by a superscript letter hyperparameters are listed in Additional file 1: Table S1. Feature Algorithm The best accuracy results were 94.2% for genus classifi- cation, 99.4% for sex classification of Aedes and 100% for LR (%) GB (%) RF (%) SVM (%) DNN (%) sex classification of Culex. Fundamental frequency 95.5 95.5 95.5 95.5 95.5 For genus, the training accuracy was 100% and the Fundamental peak power 86.9 89.5a 89.5 89.2 89.3 cross-validation accuracy was significantly lower (95%), Fundamental frequency 98.2 96.7 97.0 98.5 97.9a which indicates that the model overfits slightly and its and peak power performance could possibly be improved with more PSD 97.0 98.8a 97.9 98.8a 98.2 training samples. a Spectrogram 99.4 98.8 98.8 99.1 98.8 For Aedes sex classification, although the best models MFCC 99.4a 99.4a 98.8 98.8 98.8 gave a near perfect accuracy, the training accuracy and cross-validation accuracy are similar (99.5%), which indi- cates that the model could possibly be improved with a more complex algorithms and/or features rather than Table 3 Accuracy results for sex classification of Culex with best with more training samples. In case of Culex sex classi- results per feature indicated by a superscript letter fication the accuracy was 100%, so no error analysis was Feature Algorithm necessary. LR (%) GB (%) RF (%) SVM (%) DNN (%) Discussion Fundamental frequency 98.0 98.0 98.0 98.0 98.0 In the present study, 4335 mosquito flights were recorded Fundamental peak power 83.4 81.3 81.5 83.1 83.6a using a novel optical sensor. The sensor was attached Fundamental frequency 98.7a 98.7a 98.5 98.7a 98.7a and peak power to the entrance of a commercial mosquito suction trap PSD 99.7 99.2 99.2 100a 99.7 inside an insect rearing cage, with mosquitoes flying Spectrogram 100 a 99.7 99.7 100a 100a freely within the cage until they were sucked in by the MFCC 100 a 100 a 100 a 100 a 100a trap, through the sensor and into the catch bag within the trap. Each flight recording made by the sensor cor- responded to a different mosquito. Five features were extracted from each recording and used with five dif- set are shown in Table 3. For Culex sex classification, ferent machine learning algorithms for classification of an accuracy of 100% was achieved by all five algorithms mosquito genus and sex. trained on MFCC; by logistic regression, SVM and DNN One of the features used was wingbeat fundamental trained on spectrogram; and by SVM trained on PSD. frequency, which has been used in many studies for insect characterization and classification [15, 16, 23, 25, Summary of the best classification results 35, 38, 55]. Differences in reported values of wingbeat A summary of the classification results, which includes frequency between studies can be due to intrinsic and/ the best performing algorithms and features for each or extrinsic variables such as size, parity status, age and classification, is given in Table 4 in which training and ambient conditions [16, 34, 36, 42, 46]. In this study, the validation accuracies are also listed, with an indication of wingbeat fundamental frequency feature gave a high Table 4 Summary of machine learning classification results Classification task Using the test set Using the training dataset Error analysis indication Best test Best feature Best algorithm No. of samples Training Validation accuracy (%) accuracy (%) accuracy (%) Genus 94.2 Spectrogram DNN 2016 100 95 Slight overfitting: more training samples Sex Aedes 99.4 Spectrogram LR 1008 99.5 99.5 No overfitting MFCC LR, GB Sex Culex 100 PSD SVM 1170 100 100 No error Spectrogram LR, SVM, DNN MFCC All algorithms González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 9 of 11 accuracy in sex classification in both Aedes (95.5%) and In this study, the best performing machine learning Culex (98%), but it scored lower (67.3%) in genus clas- algorithm depended on the classification task. For genus sification. These results are consistent with the funda- classification, DNN showed the best performance, with mental frequency histograms in Additional file 1: Fig. an accuracy of 94.2%, trained on the spectrogram feature. S2, which show very little overlap between the distribu- In another work [33], DNN also gave the best perfor- tions of males and females, especially for Culex (Addi- mance for genus classification between Aedes and Culex. tional file 1: Fig. S2c) and considerable overlap between For sex classification, the best performing algorithms and genera (Additional file 1: Fig. S2a). In the fundamental features were LR with spectrogram or MFCC and GB peak power histograms of Additional file 1: Fig. S2b, c, with MFCC. Different machine learning algorithms were a higher degree of overlap is observed between the dis- also compared for mosquito classification in a previous tributions of males and females, especially for Culex, study [17], and it was concluded that the best algorithm which helps explain why the accuracy for sex using this for complex classification tasks was SVM. In our study, feature alone (89.5% for Aedes and 83.6% for Culex) was SVM had an accuracy of 93.4% for genus, although DNN, lower than that of fundamental frequency alone. which was not studied in [17], performed slightly better As other studies have indicated [16, 18, 39, 40], the (94.2%). The classification of mosquito genus achieved use of the wingbeat frequency alone as a feature to dif- a high accuracy of 94.2% while the classification of sex ferentiate between taxonomical classes or other attrib- achieved 99.4% and 100% for Aedes and Culex respec- utes of mosquito biology can be challenging because of tively. The training and validation accuracies indicate overlap in wingbeat frequency distributions. To address that genus classification could possibly be improved with this, other authors have used additional features (i.e. more training samples. depolarization ratio) [16] or metadata (i.e. localization, Other studies have successfully achieved automatic environmental variables and circadian rhythm) [18] in classification of genus [25, 33] and sex [16, 24] using combination with fundamental frequency to improve machine learning with relatively large datasets [34] and their classification methods. In the present work, we placing emphasis on class balance [17]. However, only have tested several features apart from or in combina- a small number of sensor studies have been performed tion with the fundamental frequency to better classify using a mosquito suction trap, either without an auto- mosquito genus and sex. matic classification system [39] or with only mosquito The use of both fundamental frequency and funda- and non-mosquito counting and without differentiat- mental peak power yielded better performance in sex ing mosquito genus and sex [61]. To our knowledge, we and genus classification than fundamental frequency present the first sensor system for use with a commercial alone. Although the effect of signal intensity or power mosquito suction trap, which provides automatic classi- has been investigated in mosquito mating and court- fication of genus and sex with high performance, based ship behavioural experiments [46, 47], to the best of our on a large number of training samples, with class balance. knowledge, fundamental peak power has not been used Planned further work includes the study of species clas- as a feature in mosquito classification studies. In other sification, study of age groups, training of models with sensor systems, the reported signal intensity or power more features and feature combinations, and testing of may depend on the position and orientation of the fly- the system in the field. ing mosquito with respect to the sensor [56], whilst our optical setup was designed to measure wingbeat power Conclusions relatively independently of the position and orientation In this work, we have presented the results of a novel of the mosquito within the sensing volume. sensor system for genus and sex classification of Aedes Despite the better results obtained in this work using and Culex mosquitoes captured by a commercial suc- the fundamental frequency and power features com- tion trap in laboratory conditions. The obtained results pared with fundamental frequency alone, the more are encouraging for the use of the sensor with standard complex spectrogram and MFCC features provided suction traps in the field, for the remote surveillance the best performance for genus and sex classifica- and classification of genus and sex of Aedes and Culex tion. MFCCs are normally used in applications such as mosquitoes. speech recognition [57] or music information retrieval [58], and although MFCCs are based on human percep- Abbreviations tion of pitch, they have given good results in sound rec- 2D: Two-dimensional; ADC: Analog to digital; DNN: Dense neural network; ognition studies with mosquitoes and other insects [34, GB: Gradient boosting; IoT: Internet of things; LED: Light-emitting diodes; LR: 49, 59, 60]. Logistic regression; MFCC: Mel Frequency Cepstral Coefficients; PSD: Power spectral density; RF: Random forest; SVM: Support vector machines; VBD: Vector-borne disease. González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 10 of 11 Supplementary Information Received: 25 January 2022 Accepted: 13 May 2022 The online version contains supplementary material available at https://​doi.​ org/​10.​1186/​s13071-​022-​05324-5. Additional file 1: Text S1. Mel spectrogram and MFCC generation References process. Figure S1. Diagram to illustrate MFCC generation. Figure S2. 1. World Health Organization (WHO). Vector-borne diseases fact sheet. Histogram plots showing the distributions of fundamental frequency (top) https://​www.​who.​int/​news-​room/​fact-​sheets/​detail/​vector-​borne-​disea​ and fundamental peak power (bottom) for a Genus, b Aedes sex and c ses. Accessed 20 May 2021. Culex sex. Text S2. Description of the machine learning algorithms used in 2. Rossati A. Global warming and its health impact. Int J Occup Environ this work. Figure S3. Representation of the machine learning classifica‑ Med. 2017;8:7–20. tions (in bold text), with their respective classes immediately below and 3. Khasnis AA, Nettleman MD. Global warming and infectious disease. Arch indicated by the arrow heads. Figure S4. Schematic overview of the train‑ Med Res. 2005;36:689–96. ing, validation and testing approach. 1 Dataset is randomly separated into 4. World Health Organization (WHO). A global brief on vector-borne dis‑ training and test sets, accounting for 75% and 25% of the whole dataset eases. https://​www.​who.​int. Accessed 20 May 2021. respectively. 2 Training set is separated using fourfold cross-validation into 5. Schaffner F, Versteirt V, Medlock J. Guidelines for the surveillance of native four folds with an equal number of samples in each fold. 3 Four iterations mosquitoes in Europe. https://​www.​ecdc.​europa.​eu. Accessed 20 May of training and validation take place using a different fold for validation 2021. in each iteration. 4 Model with best average validation score, obtained 6. Schaffner F, Bellini R, Petrić D, Scholte EJ, Zeller H, Marrama RL. Develop‑ by averaging the four cross-validation results, is selected. 5 Model is ment of guidelines for the surveillance of invasive mosquitoes in Europe. evaluated using test set (containing data which was previously unused) to Parasit Vectors. 2013;18:209. obtain test score. Table S1. Hyperparameters of the trained models which 7. European Centre for Disease Prevention and Control. Field sampling achieved the highest accuracies. methods for mosquitoes, sandflies, biting midges and ticks. https://​www.​ ecdc.​europa.​eu. Accessed 20 May 2021. 8. Romero-Vivas CME, Falconar AKI. Investigation of relationships between Acknowledgements Aedes aegypti egg, larvae, pupae and adult density indices where their The authors are grateful to BSL2 facilities staff of IRTA-CReSA for their technical main breeding sites were located indoors. J Am Mosq Control Assoc. contribution. 2005;21:15–21. 9. Focks DA. A review of entomological sampling methods and indicators Author contributions for dengue vectors. Special programme for research and training in tropi‑ JE, ST and NB conceived and designed the study. JE, MW and PV designed cal diseases. Florida: WHO; 2003. and built the sensor. ST, NB and CA designed the entomological part of the 10. Codeço CT, Lima AWS, Araújo SC, Lima JBP, Maciel-de-Freitas R, Honório study. JB and MG conducted the flight assays with the sensor. MG, NP and MV NA, et al. Surveillance of Aedes aegypti: comparison of house index with conducted mosquito rearing and maintenance and gave support on flight four alternative traps. PLoS Negl Trop Dis. 2015;10:e0003475. assays. BF conducted the feature extraction of flight assays and implemented 11. Krökel U, Rose A, Eiras A, Geier M. New tools for surveillance of adult the machine learning process to build the classification models. MG and BF yellow fever mosquitoes: comparison of trap catches with human analysed the results and drafted the manuscript. NB, MW, CA, ST and JE revised landing rates in an urban environment. J Am Mosq Control Assoc. the manuscript. All authors read and approved the final manuscript. 2006;22:229–38. 12. Farajollahi A, Kesavaraju B, Price DC, Williams GM, Healy SP, Gaugler R, Funding et al. Field efficacy of BG-Sentinel and industry-standard traps for Aedes This research was supported by the project VECTRACK. This project has albopictus (Diptera: Culicidae) and West Nile virus surveillance. J Med received funding from the European Union’s Horizon 2020 research and Entomol. 2009;46:919–25. innovation programme under grant agreement no. 853758. The contents of 13. Lühken R, Pfitzner WP, Börstler J, Garms R, Huber K, Schork N, et al. Field this publication are solely the responsibility of the authors, and the funding evaluation of four widely used mosquito traps in Central Europe. Parasit agency is not responsible for any use that may be made of it. Vectors. 2014;7:268. 14. Potamitis I. Classifying insects on the fly. Ecology. 2014;21:40–9. Availability of data and materials 15. Santos DAA, Rodrigues JJPC, Furtado V, Saleem K, Korotaev V. Automated The datasets generated during and/or analysed during the current study are electronic approaches for detecting disease vectors mosquitoes through not publicly available due to the protection of intellectual property defined the wing-beat frequency. J Clean Prod. 2019;217:767–75. under the H2020 agreement no. 853758, but are available from the corre‑ 16. Genoud AP, Basistyy R, Williams GM, Thomas BP. Optical remote sensing sponding author on reasonable request. for monitoring flying mosquitoes, gender identification and discussion on species identification. Appl Phys B. 2018;124:46. Declarations 17. Genoud AP, Gao Y, Williams GM, Thomas BP. A comparison of supervised machine learning algorithms for mosquito identification from backscat‑ Ethics approval and consent to participate tered optical signals. Ecology. 2020;58:e101090. Not applicable. 18. Chen Y, Why A, Batista G, Mafra-Neto A, Keogh E. Flying insect classifica‑ tion with inexpensive sensors. J Insect Behav. 2014;27:657–77. Consent for publication 19. Potamitis I, Rigakis I, Fysarakis K. Insect biometrics: optoacoustic signal Not applicable. processing and its applications to remote monitoring of McPhail type traps. PLoS ONE. 2015;10:e0140474. Competing interests 20. Kahn MC, Celestin W, Offenhauser WH. The Sounds of Disease-Carrying Irideon S.L. is currently applying for a patent relating to the content of the Mosquitoes. Recording of sounds produced by certain disease-carrying manuscript. mosquitoes. Science. 1945;101:335–6. 21. Johnson BJ, Rohde BB, Zeak N, Staunton KM, Prachar T, Ritchie SA. A Author details low-cost, battery-powered acoustic trap for surveilling male Aedes aegypti 1 IRTA, Centre de Recerca en Sanitat Animal (CReSA, IRTA-UAB), Campus de la during rear-and-release operations. PLoS ONE. 2018;13:e0201709. Universitat Autònoma de Barcelona, Bellaterra, Cerdanyola del Vallès, Spain. 22. Li Y, Zilli D, Chan H, Kiskin I, Sinka M, Roberts S, et al. Mosquito detection 2 Irideon SL, Barcelona, Spain. 3 Servei de Control de Mosquits del Consell with low-cost smartphones: data acquisition for malaria research. NIPS Comarcal del Baix Llobregat, Barcelona, Spain. Workshop on Machine Learning for the Developing World. 2017. arXiv:​ 1711.​06346​v3. González‑Pérez et al. Parasites & Vectors (2022) 15:190 Page 11 of 11 23. Mukundarajan H, Hol FJ, Castillo EA, Newby C, Prakash M. Using mobile computing and control systems. Madurai, India. 2017 Jun 15–16. IEEE. phones as acoustic sensors for high-throughput surveillance of mosquito 2017. https://​doi.​org/​10.​1109/​ICCONS.​2017.​82505​88. ecology. Elife. 2017;6:e27854. 46. Dou Z, Madan A, Carlson JS, Chung J, Spoleti T, Dimopoulos G, et al. 24. Ouyang TH, Yang EC, Jiang JA, Lin T-T. Mosquito vector monitoring Acoustotactic response of mosquitoes in untethered flight to incidental system based on optical wingbeat classification. Comput Electron Agric. sound. Sci Rep. 2021;11:1884. 2015;118:47–55. 47. Menda G, Nitzany EI, Shamble PS, Wells A, Harrington LC, Miles RN, et al. 25. Potamitis I, Rigakis I. Measuring the fundamental frequency and the The long and short of hearing in the mosquito Aedes aegypti. Curr Biol. harmonic properties of the wingbeat of a large number of mosquitoes in 2019;29:709–14. flight using 2D optoacoustic sensors. Appl Acoust. 2016;109:54–60. 48. Oppenheim AV. Speech spectrograms using the fast Fourier transform. 26. Brydegaard M. Towards quantitative optical cross sections in entomologi‑ IEEE Spect. 1970;7:57–62. cal laser radar—potential of temporal and spherical parameterizations for 49. Zhu LQ. Insect sound recognition based on MFCC and PNN. In: Interna‑ identifying atmospheric fauna. PLoS ONE. 2015;10:e0135231. tional conference on multimedia and signal processing. Uttar Pradesh, 27. Kirkeby C, Wellenreuther M, Brydegaard M. Observations of move‑ India. 2011 Dec 17–19. IEEE. 2011. https://​doi.​org/​10.​1109/​CMSP.​2011.​ ment dynamics of flying insects using high resolution lidar. Sci Rep. 100. 2016;6:29083. 50. Schmidhuber J. Deep Learning in Neural Networks: an overview. Neural 28. Mullen ER, Rutschman P, Pegram N, Patt JM, Adamczyk JJ, Johanson E. Netw. 2015;61:85–117. Laser system for identification, tracking, and control of flying insects. Opt 51. Phyton. Phyton documentation. https://​www.​python.​org. Accessed 15 Express. 2016;24:11828–38. Sept 2021. 29. Potamitis I, Rigakis I. Large aperture optoelectronic devices to record and 52. Scikit Learn. Scikit learn user guide. https://​scikit-​learn.​org. Accessed 15 time-stamp insects’ wingbeats. IEEE Sens J. 2016;16:6053–61. Sept 2021 30. Song Z, Zhang B, Feng H, Zhu S, Hu L, Brydegaard M, et al. Application of 53. Tensor Flow. Tensor flow documentation. https://​www.​tenso​r flow.​org. lidar remote sensing of insects in agricultural entomology on the Chinese Accessed 15 Sept 2021. scene. J Appl Entomol. 2020;144:161–9. 54. XGBoost. XGBoost documentation. https://​xgboo​st.​readt​hedocs.​io. 31. Richards IR. Photoelectric cell observations of insects in flight. Nature. Accessed 15 Sept 2021. 1955;175:128–9. 55. Cator LJ, Arthur BJ, Ponlawat A, Harrington LC. Behavioral observations 32. Batista GEAPA, Hao Y, Keogh E, Mafra-Neto A. Towards automatic classifi‑ and sound recordings of free-flight mating swarms of Ae. aegypti (Diptera: cation on flying insects using inexpensive sensors. In: 10th International Culicidae) in Thailand. J Med Entomol. 2011;48:941–6. conference on machine learning and applications. Honolulu, HI, USA. 56. Arthur BJ, Emr KS, Wyttenbach RA, Hoy RR. Mosquito (Aedes aegypti) flight 2011 Dec 18–21. IEEE. 2011. https://​doi.​org/​10.​1109/​ICMLA.​2011.​145. tones: frequency, harmonicity, spherical spreading, and phase relation‑ 33. Fanioudakis E, Geismar M, Potamitis I. Mosquito wingbeat analysis and ships. J Acoust Soc Am. 2014;135:933–41. classification using deep learning. In: 26th European signal processing 57. Ganchev T, Fakotakis N, Kokkinakis G. Comparative evaluation of various conference (EUSIPCO). Rome, Italy. 2018 Sept 3–7. IEEE. 2018. https://​doi.​ MFCC implementations on the speaker verification task. In: Proceedings org/​10.​23919/​EUSIP​CO.​2018.​85535​42. of the SPECOM. Patras, Greece. 2005 Oct 17–19. 2005;1:191–194. 34. Silva DF, Vinícius MAS, Ellis DPW, Keogh EJ, Batista GEAPA. Exploring 58. Müller M. Information retrieval for music and motion. https://​www.​ low cost laser sensors to identify flying insect species evaluation of mathw​orks.​com. Accessed 20 Sept 2021. machine learning and signal processing methods. J Intell Robot Syst. 59. Lukman A, Harjoko A, Yang C-K. Classification MFCC feature from Culex 2015;80:313–30. and Aedes aegypti mosquitoes noise using Support Vector Machine. 35. Genoud AP, Gao Y, Williams GM, Thomas BP. Identification of gravid In: International conference on soft computing, intelligent system and mosquitoes from changes in spectral and polarimetric backscatter cross information technology (ICSIIT). Denpasar, Indonesia. 2017 Sept 26–29. sections. J Biophotonics. 2019;12:e201900123. IEEE. 2017. https://​doi.​org/​10.​1109/​ICSIIT.​2017.​28. 36. Gibson G, Warren B, Russell IJ. Humming in tune: sex and species 60. Noda JJ, Travieso-González CM, Sánchez-Rodríguez D, Alonso-Hernández recognition by mosquitoes on the wing. J Assoc Res Otolaryngol. JB. Acoustic classification of singing insects based on MFCC/LFCC fusion. 2010;11:527–40. Appl Sci. 2019;9:4097. 37. Villarreal SM, Winokur O, Harrington L. The impact of temperature and 61. Day CA, Richards SL, Reiskind MH, Doyle MS, Byrd BD. Context-dependent body size on fundamental flight tone variation in the mosquito vector accuracy of the BG-Counter remote mosquito surveillance device in Aedes aegypti (Diptera: Culicidae): implications for acoustic lures. J Med North Carolina. J Am Mosq Control Assoc. 2020;36:74–80. Entomol. 2017;54:1116–21. 38. Staunton KM, Usher L, Prachar T, Ritchie SA, Snoad N, Johnson BJ. A novel methodology for recording wing beat frequencies of untethered male Publisher’s Note and female Aedes aegypti. J Am Mosq Control Assoc. 2019;35:169–77. Springer Nature remains neutral with regard to jurisdictional claims in pub‑ 39. Wang J, Zhu S, Lin Y, Svanberg S, Zhao G. Mosquito counting system lished maps and institutional affiliations. based on optical sensing. Appl Phys B. 2020. https://​doi.​org/​10.​1007/​ s00340-​019-​7361-2. 40. Kim D, Debriere TJ, Cherukumalli S, White GS, Burkett-Cadena ND. Infrared light sensors permit rapid recording of wingbeat frequency and bio‑ acoustic species identification of mosquitoes. Sci Rep. 2021;11:10042. 41. Geier M, Weber M, Rose A, Obermayr U, Abadam C. A smart Internet of Ready to submit your research ? Choose BMC and benefit from: Things (loT) device for monitoring mosquito trap counts in the field while drinking coffee at your desk. In: AMCA 82nd annual meeting. Savannah, • fast, convenient online submission Georgia, USA. 2016 Feb 7–11. 42. Potamitis I, Eliopoulos P, Rigakis I. Automated remote insect surveillance • thorough peer review by experienced researchers in your field at a global scale and the internet of things. Robotics. 2017. https://​doi.​ • rapid publication on acceptance org/​10.​3390/​robot​ics60​30019. • support for research data, including large and complex data types 43. Eliopoulos P, Tatlas NA, Rigakis I, Potamitis I. A “smart” trap device for detection of crawling insects and other arthropods in urban environ‑ • gold Open Access which fosters wider collaboration and increased citations ments. Electronics. 2018. https://​doi.​org/​10.​3390/​elect​ronic​s7090​161. • maximum visibility for your research: over 100M website views per year 44. Villwock S, Pacas M. Application of the welch-method for the iden‑ tification of two- and three-mass-systems. IEEE Ind Electron Mag. At BMC, research is always in progress. 2008;55:457–66. 45. Bisina KV, Azeez MA. Optimized estimation of power spectral density. Learn more biomedcentral.com/submissions In: Proceedings of the 2017 international conference on intelligent