ORIGINAL RESEARCH published: 09 December 2021 doi: 10.3389/fnins.2021.771480 Real-Time Edge Neuromorphic Tasting From Chemical Microsensor Arrays Nicholas LeBow 1† , Bodo Rueckauer 1,2† , Pengfei Sun 1 , Meritxell Rovira 3 , Cecilia Jiménez-Jorquera 3 , Shih-Chii Liu 1 and Josep Maria Margarit-Taulé 3* 1 Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland, 2 Donders Centre for Cognition, Radboud University, Nijmegen, Netherlands, 3 Institute of Microelectronics of Barcelona (IMB-CNM), CSIC, Barcelona, Spain Liquid analysis is key to track conformity with the strict process quality standards of sectors like food, beverage, and chemical manufacturing. In order to analyse product qualities online and at the very point of interest, automated monitoring systems must satisfy strong requirements in terms of miniaturization, energy autonomy, and real time operation. Toward this goal, we present the first implementation of artificial taste running on neuromorphic hardware for continuous edge monitoring applications. We used a solid-state electrochemical microsensor array to acquire multivariate, time- varying chemical measurements, employed temporal filtering to enhance sensor readout Edited by: Chetan Singh Thakur, dynamics, and deployed a rate-based, deep convolutional spiking neural network to Indian Institute of Science (IISc), India efficiently fuse the electrochemical sensor data. To evaluate performance we created Reviewed by: MicroBeTa (Microsensor Beverage Tasting), a new dataset for beverage classification Michael Schmuker, University of Hertfordshire, incorporating 7 h of temporal recordings performed over 3 days, including sensor drifts United Kingdom and sensor replacements. Our implementation of artificial taste is 15× more energy Ying Xu, efficient on inference tasks than similar convolutional architectures running on other Western Sydney University, Australia commercial, low power edge-AI inference devices, achieving over 178× lower latencies *Correspondence: Josep Maria Margarit-Taulé than the sampling period of the sensor readout, and high accuracy (97%) on a single josepmaria.margarit@imb-cnm.csic.es Intel Loihi neuromorphic research processor included in a USB stick form factor. † These authors have contributed Keywords: deep convolutional neural networks, spiking neural networks (SNNs), electrochemical sensors, sensor equally to this work fusion, neuromorphic engineering, electronic tongue (E-Tongue) Specialty section: This article was submitted to 1. INTRODUCTION Neuromorphic Engineering, a section of the journal Liquid analysis systems that assess process quality in sectors like food, beverage, and chemical Frontiers in Neuroscience manufacturing are in rising demand. Driven by increasingly strict regulations, and by the need Received: 06 September 2021 to boost productivity and to reduce costs, industry has promoted the development of automated Accepted: 11 November 2021 systems for monitoring physicochemical properties of products in their manufacturing cycle. To Published: 09 December 2021 allow process control when and wherever required, such systems must be small, energetically Citation: autonomous, and able to operate in real time. LeBow N, Rueckauer B, Sun P, In this context, the use of chemical multisensor arrays as “electronic tongues” stands out due Rovira M, Jiménez-Jorquera C, to their capability of recognizing quantitative and qualitative composition of complex solutions. Liu S-C and Margarit-Taulé JM (2021) Real-Time Edge Neuromorphic Inspired by human taste, artificial tongues use an array of chemical sensors (i.e., the artificial Tasting From Chemical Microsensor taste cells) selective—but not specific—to different solution properties. The multivariate sensor Arrays. Front. Neurosci. 15:771480. responses are then read out in the electrical domain and modeled by appropriate Machine doi: 10.3389/fnins.2021.771480 Learning methods. Frontiers in Neuroscience | www.frontiersin.org 1 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors To manufacture the arrays, microsensors fabricated in In Margarit-Taulé et al. (2019) we demonstrated a preliminary semiconductor technologies present advantages such as implementation of a portable electronic tongue analyzing miniaturization, robustness, high reproducibility, mass temporal microsensor data via PLS-DA and SVMs. This work fabrication, and ease of integration with readout electronic builds on these results to introduce SNNs as accurate and power circuitry, making them particularly suitable for on-site efficient models to perform chemometric data fusion on the edge measurements. Fusion algorithms can then be applied to for liquid analysis. To that end, we: multisensor readouts in order to automatize the analyses. By • present the first spiking, near-sensor implementation of taste exploiting the extended coverage of the sensor array, sensor running on neuromorphic hardware via deep learning models; fusion allows to increase the amount of relevant chemical • introduce MicroBeTa, a new dataset with temporal readings information inferred in the system. Embedded implementations from a chemical microsensor array acquired in commercial of these algorithms are, nonetheless, still incipient: most of beverages exemplifying industrial solutions. MicroBeTa was them are constrained by linear modeling, manual definition, or created for training and testing the classification performance predefined measurement durations. of machine learning classifiers, and can be used to assess other In the state of the art, taste inference is delayed until finishing future neuromorphic implementations for this task; voltammetric cycles or transient measurements when recording • propose a small spiking CNN that achieved high accuracy from the sensor array. To facilitate pattern recognition and on the MicroBeTa dataset, and that fits on a single Intel prevent overfitting, input data is often mapped into a lower- neuromorphic research processor; dimensional space using methods such as principal component • compare the performance of the spiking CNN against that of analysis (PCA) (Li et al., 2015; Gutiérrez-Capitán et al., 2019) a k-nearest neighbors (k-NN) classifier, a simple method that or partial least squares (PLS) (Qiu et al., 2014; Giménez- was recently deployed on the Loihi; Gómez et al., 2016; Gutiérrez-Capitán et al., 2019). Subsequent • determine the contribution of each sensor to beverage steps then use linear and non-linear algorithms including classification via the random forest algorithm, and discriminant analysis (DA) (Escriche et al., 2012), hierarchical demonstrate the relevance of sensor fusion for different cluster analysis (HCA) (Kundu and Kundu, 2013), and support combinations of the sensors processed by the algorithms; vector machines (SVMs) (Domínguez et al., 2014) for qualitative • benchmark the neuromorphic implementation against two or quantitative evaluation. commercial inference devices—a GPU and a Neural Compute Deep neural networks (DNNs) offer flexible and scalable Stick—and validate the implementation in terms of accuracy, representations to suit the complexity of representing dynamic energy efficiency, and inference time when processing the data with one single model. In particular, convolutional neural dynamic sensor recordings of the MicroBeTa dataset. networks (CNNs), are well suited to fuse data from a large number of sensor channels simultaneously—and to learn These contributions are discussed in detail below. useful classification functions on this information—while using fewer, shared connection weights than other neural network architectures. DNN implementations running on conventional 2. MATERIALS AND METHODS digital hardware exhibit, however, strong computational 2.1. Electronic Tongue Setup requirements that limit their incorporation in mobile and/or The electronic tongue described in this work aims to combine compact analytical devices. Spiking neural networks (SNNs) can two core hardware components—solid-state electrochemical offer a significant advantage in power efficiency over continuous- microsensors and a neuromorphic processor—in a novel system valued architectures when implemented on appropriate for electrochemical inference. The use of machine learning hardware (Esser et al., 2016). While they are poorly served algorithms facilitates fusion of multivariate sensor readings and by conventional von Neumann processors due to both their modeling of structure within the data while exploiting cross- highly parallel nature and the asynchronous character of sparse sensitivity between individual sensors to increase classification spiking sequences, SNNs can attain high energy efficiency on accuracy. Figure 1 illustrates the hardware configuration used to neuromorphic hardware such as IBM’s TrueNorth or Intel’s acquire the electrochemical readings and to discriminate between Loihi (Akopyan et al., 2015; Davies et al., 2018). beverages. The system employs a chemical sensor array immersed In this study, we pre-trained continuous-valued CNNs and in the beverage under test, two USB readout boards (Giménez- converted them to SNNs following the rate-based approach Gómez et al., 2015, 2016), and Intel’s Kapoho Bay (KB)—a developed by Rueckauer et al. (2017). This framework allows mobile, USB form factor of the Loihi SNN accelerator —locally direct mapping of deep neural network structures, offering attached to a laptop together with the readout boards. accuracies equivalent to non-spiking Artificial Neural Networks (ANNs) and sparse event-driven computation alike to the 2.1.1. Chemical Sensors and Readout aforementioned neuromorphic processors. Once converted, The sensor array shown in Figure 1 comprises one sensor each computational efficiency can more easily be optimized: zero for electrical conductivity, temperature, oxidation-reduction activations are natively skipped in the activity-driven operation potential (ORP), and six ion-sensitive field-effect transistor of spiking networks, and accuracy can be tailored to a (ISFET) sensors selective to specific ions (H+ , Na+ , K+ , Ca2+ , given latency and power budget in terms of number of Cl− , and NO− 3 ). The silicon-based electrochemical sensors were additive operations. monolithically integrated at the clean room of the Institute Frontiers in Neuroscience | www.frontiersin.org 2 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors of Microelectronics of Barcelona (IMB-CNM). Their technical the channel conductivity of a MOSFET with the metal gate specifications are given in Table 1. We combined electrochemical electrode omitted can be modulated by ion activity. Depositing measurement techniques of different nature (conductometry, an ion-selective polymeric membrane on top of the gate oxide ion-selective potentiometry, reducing or oxidizing (redox) and in contact with the electrolyte solution allows the sensor potentiometry) in a hybrid electronic tongue. Merging stable— to measure concentrations of particular ions in combination but general—sensors like conductivity and ORP with more with a (second) Dri-ref reference electrode. While the polymeric specific ones—but also more unreliable—like the ISFETs allows membranes are selective to one ion in particular, they do exhibit one to achieve a certain degree of operative independence with cross-responses to other ion types. Sensor fusion can then be respect to unexpected ISFET failure or drift. This approach has applied to exploit such cross-sensitivity for higher performance. already been reported in the literature (Giménez-Gómez et al., As charge-sensing devices, ISFETs are susceptible to the 2016; Gutiérrez-Capitán et al., 2019; Margarit-Taulé et al., 2019) buildup of residual charge, which can affect the measurement as successfully improving the performance of electrochemical reading. While this drift effect may be tolerable and compensated microsensor technologies in the chemometric analysis of water within certain limits, excessive trapped charge may drive and wine. sensor biasing close to or beyond supply voltage levels. Either ORP and conductivity sensors both use chemically-inert Pt this effect or the sensitivity loss to changing ion activities microelectrodes. The former measures redox potential between due to aging of the sensing membranes may necessitate the a working and a Dri-Ref reference electrode (World Precision replacement of the sensor. A temperature readout was also Instruments, Sarasota, Florida, USA); the latter employs a 4-bar included because the sensors can be themselves influenced by configuration, where an alternating current is applied between temperature changes. Such variations may be encountered after outer electrodes, and conductivity is measured between inner transferring the sensor array through the air between beverage electrodes. ISFETs use a modified metal-oxide-semiconductor samples. By providing this information, the classifier is given field-effect transistor (MOSFET) structure to obtain sensitivity to the opportunity to compensate for—or otherwise exploit—the ion concentrations in an electrolyte. When placed in a solution, temperature dependence of the individual sensors. Two readout custom boards connected the microsensors to the host system through a USB interface and handled sensor biasing, analog readout, and digitization. The six silicon microsensor channels were interfaced with a dedicated ISFET meter, while the other board provided integrated support for the conductivity, temperature, and ORP sensors (Giménez-Gómez et al., 2016). 2.1.2. Intel Loihi Neuromorphic Processor To demonstrate low-power edge tasting in real time, we trained our beverage classification models on the MicroBeTa dataset, and deployed them on one of the Intel Loihi neuromorphic chips (Davies et al., 2018) of the Kapoho Bay USB stick form factor (see Figure 1). Loihi is a digital multi-core processor optimized for running spiking neural networks. A single Loihi chip consists of FIGURE 1 | Major components of the neuromorphic electronic tongue used in 128 asynchronous neuro-cores with 1,024 current based (CUBA) this work. Detail of sensors and electrodes: 1× temperature Pt-100 (a), 2× Pt Leaky Integrate and Fire (LIF) neurons each. Neural states and microelectrodes (ORP and conductivity, b), 2× reference electrodes (c), 6× ISFET microsensors (d). Adapted from Margarit-Taulé et al. (2019). configurations are locally stored in the cores. Three synchronous ×86 processors are also included for handling input/output TABLE 1 | Technical specifications of the IMB-CNM’s electrochemical microsensors used in this work. Parameter Sensor Sensitivity Range Channel label Conductivity Pt µelectrode 900 mV·cm/mS 0.35–12 mS/cm COND Redox potential Pt µelectrode – 160–650 mV ORP pH ISFET 54 mV/dec pH 2–13 ISF1 K+ ISFET 52 mV/dec 1 · 10−5 –1 · 10−2 M ISF2 Na+ ISFET 54 mV/dec 1 · 10−5 –1 · 10−1 M ISF3 Cl− ISFET −61 mV/dec 1 · 10−5 –1 · 10−1 M ISF4 NO− 3 ISFET −59 mV/dec 6 · 10−5 –2 · 10−1 M ISF5 Ca2+ ISFET 29 mV/dec 6 · 10−7 –1 · 10−1 M ISF6 Frontiers in Neuroscience | www.frontiersin.org 3 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors TABLE 2 | Beverages represented in the MicroBeTa dataset, along with the label contact between a sensor and the reference electrodes, needing values they were assigned. the recording to be interrupted prematurely. Beverage Commercial brand Label Data from each recording session was labeled manually, with the washing and transfer periods as well as transient instabilities White wine Macabeu Celler Mas Bella 2017 0 of individual sensors assigned to a “non-beverage” class that was Red wine Merlot Duc de Foix 2015 1 later discarded. Signals from all sensor channels were recorded Still water Veri5 2 at a rate of 1 Hz, resulting in approximately 26,000 labeled Sparkling water San Pellegrino 3 measurements across all three sessions. Figure 2 shows raw data Cava Chardonnay Martí Serdà 4 from a complete measurement cycle of the first recording session used in this work. The number of measurements and data samples in relevant subsets of the data is listed in Table 3. spikes and other general tasks such as monitoring and setting up 2.3. Dataset Preparation the neural network. 2.3.1. Preprocessing Several preprocessing steps were performed on the measurement 2.2. The MicroBeTa Dataset data before it was used to train a classifier model. Incomplete 2.2.1. Beverage Types measurement cycles in which not all beverages were recorded, The beverage types selected for the MicroBeTa dataset are or measurements of specific beverage samples that were much given in Table 2. This beverage selection covers a wide range shorter than others, were removed entirely to preserve statistical of characteristics within a limited set of classes, with several balance with respect to both the total recording time per semi-overlapping sets of attributes that could be expected to beverage and also the sequence of transitions between individual provide insight into how the data from various sensors could beverages. Any measurements lasting significantly longer than 5 be used by the classifier: The red and white wine samples minutes were truncated to that length. might be expected to be chemically similar due to sharing The data from each recording session was filtered and broadly analogous production processes, with the most obvious normalized independently for several reasons. Not only were differences arising from complex organic chemicals rather than sensor offsets and sensitivities observed to change from one the simple measurements made by each separate sensor channel. acquisition session to the next, but several sensors were The subgroup of still and sparkling water, red wine and cava replaced for the second (Na+ , Cl− , Ca2+ ) and third (H+ , Cl− ) covers four general cases arising from the presence or absence recording sessions due to performance degradation. Independent of carbonation and fermentation byproducts, respectively. normalization additionally allowed to compensate for biases due The number of different beverage samples was limited by the to changes in ambient conditions between sessions. We removed time requirements and manual interaction necessary to obtain any data points with corrupted beverage readings from one or representative data from each additional beverage. more sensors, e.g., during transfer from one beverage to the next, sensor cleaning in deionized water, or accidental contact with 2.2.2. Recordings reference electrodes. MicroBeTa comprises 7 h of temporal multivariate readings The sensor readouts typically show large offsets corresponding obtained from the electrochemical microsensor array described to the various beverage types. The rate-based encoding scheme in section 2.1.1 when immersed in the various beverages used when converting the trained classifier to a spiking network (Table 2). The recordings were conducted at IMB-CNM during translates the constant offsets into dense spike trains, reducing three sessions performed over the course of 3 days. Data from the energy efficiency of the spiking model and masking the sparse all sensors was read out continuously and simultaneously during dynamic signals which are more appropriate for neuromorphic each session, while the sensor array was moved from one processing. Encoding such high-magnitude offset components beverage sample to another at fixed intervals of 5 min. During would also limit the range of signals that can be represented on each transfer, the sensor array was washed with deionized water low bit resolution systems commonly used in edge applications. before being placed in the next sample to avoid unnecessary Therefore, a configurable high-pass filter was used to attenuate cross-contamination of subsequent beverages in the series. level offsets in the input signals while emphasizing their dynamic The sequence of transitions between beverage samples was components. Its transfer function was adapted to the MicroBeTa chosen to cover all combinations from one beverage to another. dataset by setting a pole at 0.5 mHz, a zero at the origin (i.e., at 0 In this way, a dataset made up of complete measurement cycles— Hz), and unity gain. In practice, values between 0.5 and 0.8 mHz that is, cycles containing approximately 300 s of sensor readings were found to give accuracies higher than 90% in both ANNs and from each beverage sample and following a predetermined SNNs. Figure 3 illustrates the effects of the filter when applied to sequence of transitions between beverages—remains balanced the dataset. with respect to the number and types of beverages, the total Outliers were removed following the filtering operation, and individual measurement time per beverage, and any effects and the resulting signal was subsequently normalized. Both due to contamination of the sensors with traces of the previous operations were performed according to the statistics of beverage. Yet several cycles from the second and third recording training data only. Outlier values were deleted by excluding all sessions were incomplete due to charge buildup or accidental measurements in which at least one sensor channel contained Frontiers in Neuroscience | www.frontiersin.org 4 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors FIGURE 2 | Raw time series data (Z-score-normalized) from the first recording session, showing traces from the Cl− , NO− 3 , redox potential, conductivity and temperature sensors over the first full measurement cycle. The remaining ISFET channels are omitted for clarity. Individual sensor signals and the associated labels are plotted against the sample index (corresponding to seconds of valid measurements). Note the predominance of constant-offset components over transient dynamics in the signals. TABLE 3 | Measurement (sensor readings) and sample (overlapping time windows fraction of input values from being pushed close to zero. Figure 3 of length 16) counts in the MicroBeTa dataset and relevant subsets thereof. shows the effects of this normalization method on the data. Session Raw measurements Samples (window length 16) 1st 11,931 11,673 2.3.2. Sample Generation Following normalization, the corresponding time series from 2nd 8,535 5,355 each recording session were concatenated to produce a single, 3rd 5,579 4,169 piecewise-normalized series for each sensor, from which samples Total 26,045 21,197 could be drawn for training and validating a classifier model. The Sample counts indicate the number of time windows available for learning, after data samples used in this work are fixed-length time windows preprocessing. containing the signal values from all sensors over a contiguous range of timestamps. The length of the time window may be chosen arbitrarily to a value further than four standard deviations from the mean correspond to a time scale of interest. To preserve causality, the of that channel. Each sensor channel was then normalized label that corresponds to a given time window is defined as the independently using quantile normalization (Bolstad et al., label assigned to the latest measurement in the window. In this 2003), which transforms the data to a normal distribution context, a sample is an array with shape T × N, where T is before nonlinearly mapping it to a uniform distribution on the length of a time window that slides over the multivariate [0, 1]. Quantile-normalized data was found to preserve a high time series, and N is the number of sensors. Therefore, the i-th correlation between the ANN and the converted SNN, because labeled SN sample (xi , li ) in the dataset comprises the sample xi = the initial mapping to a normal distribution prevents a large k=1 k (ti−T ) . . . Ck (ti )] and label li = L(ti ) with Ck indicating [C Frontiers in Neuroscience | www.frontiersin.org 5 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors FIGURE 3 | Dataset preprocessing transforms when using a high-pass filter with a cutoff frequency of 0.5 mHz, shown for the first 1,500 measurements in the dataset for a single ISFET sensor channel (Cl− ). Note the nonlinear enhancement of near-zero signals by the quantile scaler. Signal traces are not continuous in time everywhere due to the removal of invalid measurements and edge discontinuities between labels. with multiple labels were discarded, as in these cases the beginning and end of the window contain measurements from different beverages with an implicit discontinuity between them. Therefore, two consecutive samples share all but the first and last values in each sensor time series. Figure 4 shows a diagram of the sampling scheme used in this work. Shorter time windows are preferable to longer ones for several reasons: Firstly, the number of time windows discarded due to spanning several labels increases with the window’s length, allowing a system using shorter time windows to make better use of a limited dataset. Secondly, they afford the trained system a shorter response time during online inference. Lastly, longer time windows require networks with a greater number FIGURE 4 | Scheme used for sampling a multivariate time series with of connections than would otherwise be necessary. The shortest channels C1 . . . Cn for the n = 9 sensors, and label L. Two consecutive, overlapping samples (time windows) and their corresponding labels are shown sample length that suffices to capture the input signal’s relevant in green and blue, respectively. A third time window, shown in red, would be dynamics should thus be preferred. A window length of 16 discarded because the measurements it contains span multiple labels. measurements (i.e., window length 16) was used throughout this work, corresponding to 16 s of sensor recordings. 2.3.3. Other Practical Considerations the time series of the k-th channel (sensor) and L indicating the Most of the available samples from all recording sessions were time series of labels. Note that the channel order is not relevant used for training the ANN models, with the last complete for the model architectures used in this work, and changing this measurement cycle from the final session reserved for testing. order would not be expected to affect final performance. Our motivation not to sample the dataset randomly was twofold: Because labels were available for every measurement, On the one hand, the use of overlapping time windows means overlapping time windows were used to make the most of that if the dataset were sampled randomly, a large fraction of the the available data. Time windows that contained measurements measurements from each sample in the test set would be identical Frontiers in Neuroscience | www.frontiersin.org 6 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors incorporating future features such as learning new sensor inclusions or replacements in a continual manner. All convolutional layers in each model implement one-dimensional, causal convolutions along the time dimension of the input sample. A kernel size of four was used throughout, and each layer learned 32 convolutional kernels. No intermediate pooling operations were used, and appropriate padding ensured that the dimensions of the sample did not decrease as it passed through the network. Multi-layer classifiers were also explored to map the signal from the last convolutional layer to the target classes, but they were not found to provide sufficient improvement to offset their significant added complexity. All hyperparameters were swept through realizable ranges during training. No biases were used in the network as they can lead to reduced accuracy in the converted SNN unless carefully regularized during training (Rückauer et al., 2019). While including a batch normalization layer following each convolutional step improved ANN performance and reduced FIGURE 5 | Causal CNN architectures implemented in the work. Opaque channels depict the final seven-sensor configuration deployed on all training time significantly, it also increased variance in the SNN benchmarked systems. Transparent channels the additional sensor inputs accuracy. For this reason, no normalization layers were used, originally used to classify the beverages from all nine sensors. and the training duration was increased from 200 to 1,500 epochs. Networks were trained with the AdaBound optimizer on categorical cross-entropy loss, with norm clipping and L2 regularization. A batch size of 128 samples was chosen. with measurements from several other samples in the training set; on the other hand, from the point of view of a given test sample, 2.4.2. ANN-SNN Conversion and Mapping to Loihi random sampling would allow the model to train on data samples The classification models were initially trained in Keras and then subsequent to those used for test. This is not a condition that will converted to rate-based spiking networks using the SNN Toolbox happen in practice. (Rueckauer et al., 2017). The networks were then deployed on The recordings in the MicroBeTa dataset pose several Loihi via the NxTF framework (Rueckauer et al., 2021), which potential challenges for a classification algorithm. In particular, provides a Keras-like API to instantiate neural networks on three ISFET sensors had to be replaced before the second Loihi, and includes a compiler to optimally apply Loihi’s resource recording session; and two more sensors had to be replaced sharing features while mapping convolutional topologies to the before the third session, as mentioned above. Furthermore, the multi-core hardware. first session contains significantly more data than either of the The SNN Toolbox configuration used for the experiments subsequent sessions after preprocessing, comprising 55% of the in this paper was made available together with the dataset. dataset as shown in Table 3. Because of the sequential sampling Sensor readings were mapped as bias currents of the input strategy and difference in session lengths, using all sessions means SNN layer, and membrane potentials were reset by subtraction that the network is trained primarily on data from the first via 2-compartment neurons after spiking. The weights of the session, while the test set contains measurements from the third. neural network were normalized setting the maximum ReLU Nonetheless, the models were ultimately found to generalize well. activations in each layer to the 99-th percentile of their total Such a good generalization could be favored by the small number activity distribution. Finally, the output softmax layer was of sensors replaced along the third session of recordings, and to converted to spiking by computing spike rates according to the the presence of full measurement cycles from that session in the membrane potentials of the output neurons (Rueckauer et al., training set. 2017). Keeping the CNN depth to three layers helped mitigating the accumulation of discretization errors across the hierarchy, an 2.4. Fusing Sensor Readings effect deeper rate-based converted SNNs are more prone to. Each 2.4.1. Neural Network Model of the test samples was ran for 50 algorithmic time steps on the We aimed for a small, energy-efficient model size that fits in one Loihi (Davies et al., 2018). of the two chips included in the Kapoho Bay platform, avoiding any overheads in terms of latency and power consumption that 2.4.3. Selecting Informative Sensors might be added in inter-chip communication. We determined the contribution of each sensor to beverage The results described below were obtained with networks classification via the random forest (RF) (Breiman, 2001) consisting of three convolutional layers followed by a fully- algorithm with Gini impurity. RF is an efficient dimension connected Softmax classifier stage, where every sensor used reduction method that uses a simple bagging ensemble technique. corresponds to an input channel as depicted in Figure 5. We RF models comprise many decision trees, and each decision chose a deep convolutional architecture with the intent of tree learns how to split the input into smaller subsets to predict Frontiers in Neuroscience | www.frontiersin.org 7 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors the beverage class. Sensor importance is assessed by looking at TABLE 4 | Average classification performance of ANN and converted SNN how much each sensor contributes to each tree, then taking an models on the MicroBeTa dataset. average value over all the decision trees, and finally comparing Test accuracy the contributions of each sensor. The Gini impurity is the default Dataset k-NN ANN SNN metric used in the scikit-learn toolbox to determine how the trees are split at the different nodes. For a decision forest (in this 9 Sensors 88.8% 91.3 ± 0.9% 90.7 ± 0.8% work, we used 1,000 trees), it is possible to calculate the average 7 Sensors 98.2% 98.6 ± 0.6% 97.0 ± 1.6% reduction in the impurity of each sensor and use the average reduction as the criterion for sensor selection. Statistics were obtained by training each configuration independently five times and varying only the initialization seed value. All other hyperparameters were held constant for all experiments. Error intervals are given as the standard deviation. 2.5. Benchmarking Accuracy and Energy Efficiency ANN and SNN performance statistics were obtained from increases the accuracy by 5% compared to the accuracy (88.8%) five models per ANN input configuration, independently from using all the sensors of the array. Combining conductivity, trained varying only the initialization seed value. All other ORP, and temperature sensors gives 85.6% accuracy. The high hyperparameters were held constant for all experiments. As a accuracy of using only ISFETS could be due to the fact that the baseline, we studied √ the accuracy of the k-NN algorithm over a training set already contained data from all the replaced sensors sweep from 1 to 2 N nearest neighbors, where N is equal to the and the day-to-day drifts that were present in the test data, thus 19,844 training samples of the dataset. obviating the need for more stable sensors to compensate for We estimated energy efficiency from the dynamic power these effects. consumption and the inference time measured when running the To validate the effectiveness of choosing the most informative SNN on one of the Kapoho Bay’s Loihi chips. Dynamic power sensors, we computed the importance value of the nine sensors was obtained as the additional cost of performing inference using the RF algorithm. The results in Table 6 show that two of with respect to the baseline idle power, i.e., when the device is the sensors (TEMP and ISF5) are least informative. All models powered and clocked, but idling. Its relative performance in these were retrained for the seven sensor combination, achieving the same terms was compared with the results obtained from ANNs results shown in the second row of Table 4. The k-NN (a 1- deployed on two other devices for DNN acceleration: an Nvidia NN in this case), ANN and SNNs are on average 11, 8, and 7% GeForce GTX TITAN X GPU, and an Intel Movidius Neural more accurate, respectively, than the nine-sensor configurations. Compute Stick 2 (NCS2) for low power edge-AI inference. A Selecting the seven most informative sensors helped boosting batch size of 1 was used for all power measurements. performance in these terms. Power consumption on the NCS2 was monitored by means For the selected sensor combination, the ANN still exhibits of an external UM34C USB power meter. In order to account higher accuracy than the 98.2% achieved by the 1-NN. It also for I/O power consumption, we evaluated an upper bound of the surpasses the 97.5% accuracy of a perceptron similar to the model I/O power by running a single-layer softmax perceptron with the used to estimate I/O power on the Movidius. When training the same number of inputs and outputs, and by subtracting inference CNN, the variability of SNN accuracies was found to worsen power from the power consumption measured after allocating considerably with lower model complexities. The models tended the models on the device. This upper bound of I/O power was to overfit when increasing the number of kernels beyond the then deducted from the dynamic power exhibited by the NCS2 chosen value of 32. Besides its suitability for fusing data from when running the neural network models. multiple channels with a reduced number of weights, the small convolutional architecture of Figure 5 was chosen for two more 3. RESULTS reasons: for its translational invariance, which would help to pick up relevant input features even in dynamically changing 3.1. Classification Accuracies vs. Sensor sensor readings; and for having a low but enough dynamic energy Selection consumption to allow external metering of inference power, We evaluated both k-NN and CNN accuracies initially using all distinguishable from I/O power on the Movidius. Figure 6 shows nine sensor readings from the test dataset. The first row of Table 4 confusion matrices of test accuracies averaged over the five final shows the accuracies reached for this sensor combination using CNN model results. a rolling window of 16 readings. Accuracy values refer to the per-class accuracy, averaged over all time windows of the test 3.2. Time Delay Response set. The ANN outperforms the top scoring k-NN (i.e., a 201- We computed the online accuracy classification vs. time delay NN) by a 3%. Compared to the ANN, the SNN accuracy dropped fixing window length to 16 for the two sensor combinations. The approximately 1% on average, with similar standard deviation. results are shown in Figure 7. A prediction is done at every time We also tested the performance of every sensor channel on the step. The dots indicate the average accuracy over 10 time steps. k-NN algorithm. In all cases, the accuracy of a single sensor does The accuracy varies between 80 and 100% for the seven-sensor not exceed 63% (see Table 5). When fusing the outputs of each combination and between 58.8 and 100% for the nine-sensor readout board via the k-NN method, using ISFETs alone (93.8%) combination throughout this period of 10 s. The online accuracy Frontiers in Neuroscience | www.frontiersin.org 8 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors TABLE 5 | Individual accuracies reached by a k-NN classifier when using single sensor channels. Sensor type TEMP ISF1 ISF2 ISF3 ISF4 ISF5 ISF6 COND ORP Accuracy 21.4% 53.5% 35.6% 57.2% 61.6% 38.0% 34.0% 58.5% 62.7% TABLE 6 | Sensor importance values estimated using the random forest algorithm. Sensor Type TEMP ISF1 ISF2 ISF3 ISF4 ISF5 ISF6 COND ORP Importance value 0.025 0.092 0.176 0.126 0.124 0.044 0.119 0.185 0.109 TEMP and ISF5 channels in particular contribute little information. FIGURE 6 | Confusion matrices summarizing the CNN prediction results for the nine (left) and seven (right) sensor combinations. Classification performance is averaged over five independent runs of CNNs trained with random initialization seeds. using seven sensors is higher than using nine sensors at almost TABLE 7 | Average dynamic power consumption, inference time, and dynamic every time step. energy cost per inference across hardware devices when using a combination of seven sensors. The length of the time window is important because it affects the accuracy and the response latency. We investigated how Hardware Dyn. power Inf. time Dyn. energy the online accuracy of a network trained on the seven-sensor (mW) (ms) (mJ/Inf) combination changed with the time window length. From the GPU (Titan X) 9,649 3.0 29.0 results in Figure 7, it is clear that a larger time window leads Movidius (NCS2) 736 2.0 1.5 to a higher network accuracy but also results in a longer latency Loihi (KB) 15 5.6 0.1 response. The choice of the window length is thus dependent on the desired accuracy-latency trade-off. 3.3. Dynamic Energy Consumption Table 7 compares dynamic power, inference time, and dynamic model running as non-spiking ANN on the NCS2 and the GPU, energy per inference across hardware devices for the selected respectively. Power efficiency came at the cost of increasing seven-sensor combination. Our neuromorphic implementation the inference time to allow the spiking rates to converge; our of the CNN for artificial taste consumes 15 mW of dynamic SNN runs almost 2× and 3× slower on the Loihi than the power on the Kapoho Bay, 49× and 643× lower than the same ANNs deployed on the other two devices. Notably, the SNN still Frontiers in Neuroscience | www.frontiersin.org 9 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors latency needed by the SNNs to reach accuracies comparable to the ANNs without noticeable lags. Contrary to previous implementations of electronic tongues found in literature, our system is capable of classifying solutions accurately and continuously over transient sensor responses, with no need to delay inference until steady state is reached. This characteristic may be crucial in preventing critical risks (e.g., product contamination, process malfunction) early, when incidence is still low. Domain-optimized CNN architectures for sequential data offer efficient training and representation of time- FIGURE 7 | Classification accuracy vs. time delay for the seven and nine invariant features in a conceptually and computationally sensor combinations (left) and accuracy for the seven sensor combination for straightforward manner. Furthermore, this class of architectures different time window lengths (right). The dots indicate the mean and the is well understood and widely supported, affording easier shadowed regions indicate the standard deviation over five runs of ANNs integration with existing systems, and allowing more independently trained with random initialization seeds. direct translation into the physically-constrained synaptic connections of neuromorphic hardware. While Recurrent Neural Network (RNN) models can be more efficient on such hardware by operating on instantaneous measurements— achieves remarkable gains in terms of dynamic energy cost— instead of on sequential values of the data stored with 15× lower than the NCS2 and 290× lower than the GPU— fixed-time window lengths—it still remains challenging to requiring only 0.1 mJ per inference. This value is one order of train and achieve the same levels of accuracy with spiking magnitude lower than the dynamic energy reported on the k- recurrent architectures. NN implementations of Frady et al. (2020), when processing To evaluate system performance (Davies, 2019), we 76,800-sample datasets on Loihi. created MicroBeTa: a beverage tasting, benchmark dataset to discriminate between five commercial beverage varieties using 4. DISCUSSION temporal readings acquired from a solid-state electrochemical microsensor array. The dataset covers every combination of This work introduces the first neuromorphic implementation transitions from one beverage to another, including all water, of artificial taste. Our electronic tongue uses rate-based, deep wine, and cava types in each arrangement to balance the set. spiking convolutional networks to fuse dynamic, electrochemical It extends to 26 ksamples (about 7 h) of dynamic multivariate microsensor readings. It performs with high accuracy (97%) data, including sensor drifts and replacements. As the first and high energy efficiency (0.1 mJ per inference), and it can open dataset of its kind, we believe that it will be useful for run the neural networks in real time (5.6 ms, over 178× the research community to explore and compare different lower than the sampling period of the sensor readout employed spike encoding or processing approaches applied to this new by the system) on a single Loihi chip. The system exploits sensory domain. This benchmark can be used in a real-time, sparse spiking computation and the particular latency budget non-batched regime as a preliminary workload to assess the of chemical sensor dynamics to trade off inference time for latency, throughput, accuracy, energy, or resource consumption power consumption, achieving energy efficiency gains of 15× of alternative neuromorphic implementations for chemical and 290× when compared to the same CNN architecture monitoring applications. running with continuous values on the low-power Intel Movidius Our studies show that accuracy can be improved by Neural Compute Stick 2, and on an Nvidia GeForce GTX fusing readings from all input channels except for the TITAN X GPU, respectively. According to Frady et al. (2020), temperature and NO− 3 ISFET sensors. Excluding these last the proposed CNN would be an order of magnitude more two sensors makes sense given the negligible thermal effects energy efficient than the k-NN when classifying from 76,800 on sensor readings, and the erratic response exhibited by training examples on the Loihi; the approximate k-NN algorithm FET sensor in some measurement cycles. These results would distribute the training set among multiple chips. In corroborate the complementarity of all the remaining contrast to the convolutional network, both k-NN power microsensors of the array toward generating a unique consumption and chip occupancy would scale up substantially fingerprint for each beverage of the dataset (Legin et al., with the addition of more temporal samples in the dataset. 1999). Furthermore, the CNN is better suited for extensions such as When miniaturizing an autonomous electrochemical learning online from dynamically changing data due to e.g., monitoring system, power consumption becomes one of the sensor replacements. main technical challenges limiting the inclusion of embedded Neuromorphic computing is advantageous in electrochemical intelligence. Our spiking taste implementation curtails inference monitoring, as sensors follow slow dynamics that relax inference costs to an average of 100 µW at the 1-Hz multisensor sampling speed requirements. This allows for the 2×-3× increase in frequency used in this work. This power budget is close to Frontiers in Neuroscience | www.frontiersin.org 10 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors that of state-of-the-art smart electrochemical sensing devices, AUTHOR CONTRIBUTIONS which integrate the chemical sensors and the CMOS readout circuitry with a power consumption of tens of µW per sensor NL and BR contributed to the simulation work, energy (Li et al., 2017; Miscourides and Georgiou, 2019). For the measurements on hardware platforms, and writing. PS number of sensors selected in the work, power requirements contributed to the simulation work and writing. MR and could be satisfied up to several months by a single CR2032 coin CJ-J contributed to the sensor fabrication and data collection. cell. Such an energy autonomy opens the door for deploying S-CL contributed to the experiment conception and writing. chemosensory integration directly on the edge, avoiding the JM-T contributed to the experiment conception, data collection, communication bottlenecks, delays, and privacy concerns simulation work, energy measurements on hardware platforms, inherent to cloud computing. The results also indicate good and writing. All authors contributed to the article and approved robustness to sensor non-idealities, using dynamic readings the submitted version. in an architecture manufacturable at wafer level. Once the functionality of our neuromorphic tasting approach is verified, FUNDING the SNN could be applied to continuous monitoring in remote and compact locations, providing an appropriate This work was supported by the European Union’s Horizon environment to study incremental learning of new classes or 2020 research and innovation programme under the Marie sensor channels. Sklodowska-Curie grant agreement no. 747848 and the SNSF- Sinergia WeCare project (N◦ CRSII5_177255). DATA AVAILABILITY STATEMENT ACKNOWLEDGMENTS The dataset introduced in this study can be found in the Zenodo The authors acknowledge Intel Labs and the INRC community open access repository, with link http://doi.org/10.5281/zenodo. for providing access to the Loihi research chip and for the 5457501. valuable discussions and support. REFERENCES based on microsensors for the analysis of cava wines. Sensors 16, 1796. doi: 10.3390/s16111796 Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Giménez-Gómez, P., Escudé-Pujol, R., Jiménez-Jorquera, C., and Gutiérrez- et al. (2015). Truenorth: design and tool flow of a 65 mw 1 million neuron Capitán, M. (2015). Multisensor portable meter for environmental applications. programmable neurosynaptic chip. IEEE Trans. Comput. Aided Design Integr. IEEE Sens. J. 15, 6517–6523. doi: 10.1109/JSEN.2015.2460011 Circ. Syst. 34, 1537–1557. doi: 10.1109/TCAD.2015.2474396 Gutiérrez-Capitán, M., Brull-Fontseré, M., and Jiménez-Jorquera, C. (2019). Bolstad, B., Irizarry, R., Åstrand, M., and Speed, T. (2003). A comparison Organoleptic analysis of drinking water using an electronic tongue based on of normalization methods for high density oligonucleotide array electrochemical microsensors. Sensors 19:1435. doi: 10.3390/s19061435 data based on variance and bias. Bioinformatics 19, 185–193. Kundu, P. K., and Kundu, M. (2013). The e-tongue-based classification and doi: 10.1093/bioinformatics/19.2.185 authentication of mineral water samples using cross-correlation-based pca and Breiman, L. (2001). Random forests. Mach. Learn. 45, 5–32. sammon’s nonlinear mapping. J. Chemom. 27, 379–393. doi: 10.1002/cem.2521 doi: 10.1023/A:1010933404324 Legin, A., Rudnitskaya, A., Vlasov, Y., Di Natale, C., and D’Amico, A. (1999). Davies, M. (2019). Benchmarks for progress in neuromorphic computing. Nat. The features of the electronic tongue in comparison with the characteristics Mach. Intell. 1, 386–388. doi: 10.1038/s42256-019-0097-1 of the discrete ion-selective sensors. Sens. Actuators B Chem. 58, 464–468. Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Joshi, P., Lines, A., et al. (2018). doi: 10.1016/S0925-4005(99)00127-6 Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro Li, H., Liu, X., Li, L., Mu, X., Genov, R., and Mason, A. J. (2017). Cmos 38, 82–99. doi: 10.1109/MM.2018.112130359 electrochemical instrumentation for biosensor microsystems: a review. Sensors Domínguez, R. B., Moreno-Barón, L., Muñoz, R., and Gutiérrez, J. M. 17:74. doi: 10.3390/s17010074 (2014). Voltammetric electronic tongue and support vector machines for Li, Y., Lei, J., and Liang, D. (2015). Identification of fake green tea by sensory identification of selected features in mexican coffee. Sensors 14, 17770–17785. assessment and electronic tongue. Food Sci. Technol. Res. 21, 207–212. doi: 10.3390/s140917770 doi: 10.3136/fstr.21.207 Escriche, I., Kadar, M., Domenech, E., and Gil-Sánchez, L. (2012). A Margarit-Taulé, J. M., Giménez-Gómez, P., Escudé-Pujol, R., Gutiérrez-Capitán, potentiometric electronic tongue for the discrimination of honey according M., Jiménez-Jorquera, C., and Liu, S.-C. (2019). “Live demonstration: a portable to the botanical origin. comparison with traditional methodologies: microsensor fusion system with real-time measurement for on-site beverage physicochemical parameters and volatile profile. J. Food Eng. 109, 449–456. tasting,” in IEEE International Symposium on Circuits and Systems (ISCAS) doi: 10.1016/j.jfoodeng.2011.10.036 (Sapporo: IEEE). Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Miscourides, N., and Georgiou, P. (2019). Isfet arrays in cmos: A head-to-head Andreopoulos, A., et al. (2016). Convolutional networks for fast, energy- comparison between voltage and current mode. IEEE Sens. J. 19, 1224–1238. efficient neuromorphic computing. Proc. Natl. Acad. Sci. U.S.A. 113, doi: 10.1109/JSEN.2018.2881499 11441–11446. doi: 10.1073/pnas.1604850113 Qiu, S., Wang, J., and Gao, L. (2014). Discrimination and characterization of Frady, E. P., Orchard, G., Florey, D., Imam, N., Liu, R., Mishra, J., et al. strawberry juice based on electronic nose and tongue: comparison of different (2020). “Neuromorphic nearest neighbor search using intel’s pohoiki springs,” juice processing approaches by lda, plsr, rf, and svm. J. Agric. Food Chem. 62, in Proceedings of the Neuro-inspired Computational Elements Workshop 6426–6434. doi: 10.1021/jf501468b Heidelberg, 1–10. Rückauer, B., Känzig, N., Liu, S.-C., Delbruck, T., and Sandamirskaya, Y. (2019). Giménez-Gómez, P., Escudé-Pujol, R., Capdevila, F., Puig-Pujol, A., Jiménez- Closing the accuracy gap in an event-based visual recognition task. arXiv Jorquera, C., and Gutiérrez-Capitán, M. (2016). Portable electronic tongue [preprint]. arXiv:1906.08859. Frontiers in Neuroscience | www.frontiersin.org 11 December 2021 | Volume 15 | Article 771480 LeBow et al. Neuromorphic Tasting From Chemical Microsensors Rueckauer, B., Bybee, C., Goettsche, R., Singh, Y., Mishra, J., and Wild, A. (2021). the publisher, the editors and the reviewers. Any product that may be evaluated in Nxtf: an api and compiler for deep spiking neural networks on intel loihi. arXiv this article, or claim that may be made by its manufacturer, is not guaranteed or [preprint]. arXiv:2101.04261. doi: 10.1109/IJCNN52387.2021.9533837 endorsed by the publisher. Rueckauer, B., Lungu, I.-A., Hu, Y., Pfeiffer, M., and Liu, S.-C. (2017). Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Front. Neurosci. 11:682. doi: 10.3389/fnins.2017.00682 Copyright © 2021 LeBow, Rueckauer, Sun, Rovira, Jiménez-Jorquera, Liu and Margarit-Taulé. This is an open-access article distributed under the terms of Conflict of Interest: The authors declare that the research was conducted in the the Creative Commons Attribution License (CC BY). The use, distribution absence of any commercial or financial relationships that could be construed as a or reproduction in other forums is permitted, provided the original author(s) potential conflict of interest. and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, Publisher’s Note: All claims expressed in this article are solely those of the authors distribution or reproduction is permitted which does not comply with these and do not necessarily represent those of their affiliated organizations, or those of terms. Frontiers in Neuroscience | www.frontiersin.org 12 December 2021 | Volume 15 | Article 771480