This is the accepted version of the article:
Martin Martinez, Javier; Rodriguez, Rosana; Nafria, Montserrat; [et al.]. «An un-
supervised and probabilistic approach to Pavlov’s dog experiment with OxRAM
devices». Microelectronic Engineering, Vol. 215 (July 2019), art. 111024. DOI
10.1016/j.mee.2019.111024


This version is available at https://ddd.uab.cat/record/249285

under the terms of the                license
                               An unsupervised and probabilistic approach

                            to Pavlov’s dog experiment with OxRAM devices

        M. Pedro1*, J. Martin-Martinez1, R. Rodriguez1, M.B. Gonzalez2, F. Campabadal2 and M. Nafria1
1Departament     d’Enginyeria Electrònica, Universitat Autònoma de Barcelona (UAB) Campus UAB. 08193 Bellaterra. Spain.
           2   Institut de Microelectrònica de Barcelona, IMB-CNM, CSIC. Campus UAB. 08193 Bellaterra. Spain.
                                  E-mail address of corresponding author: marta.pedro@uab.es

Abstract
In this work, a potential basis for implementing unsupervised associative learning in two or more memristors
within neuromorphic architectures is proposed. The experimental demonstration is carried out by means of
emulating the Pavlov’s dog classical conditioning experiment with two OxRAM devices, in which the
dependence of the probability of association on test parameters, such as the pulse amplitude, is studied.
Keywords: Associative learning, Memristors, Neuromorphic, OxRAM, Unsupervised Learning
1. Introduction
Memristors are being widely investigated for their application as synaptic devices in neuromorphic architectures.
Their characteristics allow merging computation and memory, which is of great interest when a bio-inspired
approach is considered. With this purpose, learning models coming from psychobiology are being demonstrated
concerning the programmable conductance that these devices present, which is identified as the synaptic weight.
The ability of fine-tuning their conductivity state according to the features of the past electrical activity
demonstrates their plasticity property. Unsupervised learning rules, which rely on the dependences of plasticity
on time or frequency correlations between the pre and post-synaptic activities, have been already proven in
single electronic devices in a large variety of memristive technologies [1-4]. However, there is still a lack of
understanding on how bio-inspired local learning rules should be designed when the interaction between two or
more memristors is considered. In the biological brain, complex cognitive processes arise from associative
learning mechanisms, involving the causal interaction between multiple neuronal layers, and thus, of a large
amount of synapses. The study of associative learning between electronic synapses would suppose a step
towards the implementation of neuromorphic associative memories and hierarchical computing networks, able to
store and recall symbolic knowledge, as occurs in the mammalian neural system.

A simple case of associative learning can be found in the classical conditioning experiments initially conducted
by I. Pavlov [5]. In classical conditioning, two stimuli are considered: one is the so-called unconditioned
stimulus (US), and the second is the neutral stimulus (NS). The US triggers a response of the system, labelled as
the unconditioned response (UR). On the other hand, the NS does not provoke the UR. It is through a stage of
conditioning in which the NS is associated to US, so that the NS becomes a conditioned stimulus (CS),
generating a system response similar to the UR. The experiment that Pavlov carried out was actually with dogs:
the US was the sight of food, which made the dogs salivate (UR). The NS he employed was the ringing of a bell,
which was neutral to the dogs in the sense that, only with its presence no salivation was induced. Pavlov
discovered that, by consecutively ringing the bell when feeding the dogs, they would start responding similarly
to both US (food) and NS (ringing of the bell), the latter becoming a CS. That is, they would salivate when
hearing the bell without the presence of food, because the dogs associated it with their feeding time. This finding
had great implications, being today the basis of the associative learning theory.

Electronic emulations of classical conditioning and Pavlov’s dog test have actually been carried out with
memristive devices, either using a single partially-volatile device [6], a single non-volatile device [7, 8], or a pair
of volatile organic transistors with discrete CMOS neurons, being the latter focused on the implementation of a
short-term associative memory [9]. In our work, we introduce an unsupervised electronic setup for studying a
basic process of associative learning, involving the use of a pair of non-volatile OxRAM devices and being the
test scheme analogous to the original Pavlov’s dog experiment. Moreover, we propose to analyze the
conditioning efficiency by establishing a probabilistic association criterion. This is motivated by, on the one
hand, taking into account the device-level variability found in memristive technologies such as OxRAM, which
is often considered as an issue. On the other hand, significant variability is also found in biological neural and
synaptic populations, as well as in their plasticity property, yet this variability is thought to be a typical (and
maybe necessary) feature for a healthy nervous system [10-12]. This is the main reason for which statistical
methods and models have been successfully incorporated to the computational neuroscience research during the
last decades [13]. Therefore, it is plausible to design probabilistic learning rules for their application in
neuromorphic architectures considering the variability observed in the devices to be employed.
2.     Materials and methods
     2.1. Tested samples and setup description
The tested RRAM devices consist of 5x5μm2 TiN-Ti-HfO2-W Metal-Insulator-Metal (MIM) structures. Details
of their fabrication can be found in [14]. An example of their I-V characteristics is shown in Figure 1, in which
bipolar resistive switching phenomena can be observed.




     Figure 1. Examples of I-V characteristics of the tested
     samples, S1 (blue) and S2 (red). Resistive switching is
     described by two main processes: the SET process, which is
     the transition from a high resistance state to a lower one, and   Figure 2. Scheme of the experimental setup for the electronic
     the RESET, which is the reverse behavior. Both samples            Pavlov’s dog experiment. V1 and V2 are the input signals of the
     present similar SET and RESET voltages.                           system, connected to the post-synaptic neuron via two OxRAM
                                                                       devices, S1 and S2. Initial conditions for the two samples are also
                                                                       displayed, being S1 at a high conductivity state, whereas S2 has a low
                                                                       conductivity state.

The scheme of the employed characterization setup for implementing associative learning is shown in Figure 2,
involving a four-point manual probe station and a semiconductor parameter analyzer controlled via GPIB. In
Figure 2, two RRAM devices represent the synaptic connection (S1, S2) between two pre-synaptic sensors and a
post-synaptic neuron (POST), which S1 and S2 have in common. The signals coming from the sensors are
identified as V1 and V2, which are the inputs of the system, and share the same amplitude value, Vd. On the other
hand, V3 represents the feedback signal triggered by the POST, and provides also a way to measure the response
of the system. All of the employed signals are applied with a semiconductor parameter analyzer (SPA) Agilent
4156C. The smart control of the SPA is employed for emulating the POST as an integrate-and-fire neuron. The
current flowing through S1 and S2 is collected by the SPA, and the amount of charge that the neuron is receiving
is computed and accumulated over time. When the accumulated charge reaches a defined threshold Qthr, the
virtually stored charge is cleared, and the POST neuron fires a voltage pulse towards S1 and S2, with amplitude
V3=−Vd and a determined pulse-width PW. Since this post-synaptic pulse represents a response to the pre-
synaptic activity, two different situations can occur for each of the synapses: a voltage drop of 2Vd if there is
some pre-synaptic activity coming from the sensor, or a voltage drop of −Vd if there is no pre-synaptic signal.

     2.2. Test scheme
    The electrical classical conditioning scheme that we propose consists in emulating the Pavlov’s dog test. The
goal is to associate the V1 and V2 inputs through experience, so that the activation of the POST (salivation) is
achieved not only with the presence of V1 (food), but also with the single application of V2 (bell ringing). This is
because during the trials V2 appears paired with V1, so an associative process between the inputs takes place.
Initially, S1 has a high conductivity level, since the dog salivation is related to the sight of food, whereas S2 has
a low conductivity, because the ringing of the bell is assumed to not cause salivation. The whole learning
sequence is represented in Figure 3. The voltage drops at S1 (VS1 in Figure 3.a) and S2 (VS2 in Figure 3.b), which
are defined as VS1 = V1 – V3 and VS2 = V2 – V3, are plotted over time. Next, the V1 (US) and V2 (NS → CS)
waveforms (Figure 3.c and Figure 3.e, respectively) are plotted over time. The neuron response |V3| is also
depicted in Figure 3.g, symbolizing the dogs’ salivation. The currents through S1 (I1 in Figure 3.d) and S2 (I2 in
Figure 3.f) which were measured during one of the tests are also shown as an example, which were recorded
only when the POST was firing a pulse, in order to speed the performance of the tests.

First of all, a current-controlled forming process is performed on the S1 and S2 devices, which are located within
the same dice. Next, five compliance-free RESET and SET processes are induced to each of the samples (not
shown in Figure 3), in order to set their conductivity states within a certain range, following the procedure of the
previous works [A-B]. The maximum voltage applied to the sample with the S1 role was of Vset=2Vd, whereas in
the case of S2, it was of Vset=Vd. In this way, the conductivity state of S1 was set to a high value (G>30Go),
whereas S2 was set to a low value (G≤5Go). The learning sequence starts with the single application of V1=Vd
  Figure 3. Particular case of voltage and intensity evolutions over a learning sequence where the association process took place. The
  bottom part of the figure shows an analogy between the POST activity and the Pavlov dog’s behavior. (a) Voltage drop at S1, defined
  as Vs1 = V1-V3. (b) Voltage drop at S2, defined as Vs2 = V2-V3. (c) Voltage waveform V1 corresponding to the US, given by sensor 1.
  Its high level is of Vd. (d) Intensity flowing through S1, measured only when the POST triggered a spike. The conductivity state of S1
  is also indicated in Go units, being of 35.36Go, corresponding to a high conductivity state. (e) Voltage waveform V2 with a high level
  of Vd. V2 corresponds to the NS, given by sensor 2, which after the conditioning stage becomes CS if the association process has
  occurred. (f) Intensity flowing through S2, measured only when the POST triggered a spike. The conductivity state of S2 is also
  indicated in Go units, being of 0.063Go before the conditioning stage, and of 31.67Go after the association process, corresponding to
  low and high conductivity states, respectively. (g) Voltage waveform |V3| corresponding to the POST activity, given as spikes with
  amplitude -Vd. In this case, the POST responds first only to the presence of V1, whereas after the association process, it also responds
  to V2.


(−20s<t<−15s), and then of V2 =Vd (−10s<t<−5s). The POST responds to V1 because S1 presents a high
conductivity state from the beginning of the test. On the other hand, no firing is expected to occur when only
V2=Vd is applied, since it represents the NS, and S2 has a very low conductivity. Next, both V1 and V2 are active
together during a training time of ttr = 50s. Because of the current through S1 is high, eventually, the
accumulated charge in POST reaches Qthr, so that it fires a voltage pulse as a response. The voltage drop at S2,
which has a magnitude of 2Vd, causes S2 to increase its conductivity. This event may occur multiple times
during this phase, as shown in Figure 3. The value of Vd has to be carefully chosen as to induce a positive
change in S2 conductivity only when V2 and the neuron pulses are overlapping. That is, the voltage drop at S2 of
2Vd has to be large enough to induce a SET process, but a voltage drop of magnitude Vd in device S2 should not.
On the other hand, the conductivity state of S1 remains constant during the whole test, since a voltage drop of
2Vd is not large enough as to increase its conductivity state above its initial value, which is already large. This
stage with paired V1 and V2 is the core of the classical conditioning experiment, because it is when associative
learning, which implies a change in S2 conductivity when V1 and V2 are active simultaneously, occurs.
However, this does not mean that necessarily V2 and the POST will be fully connected in a single learning phase,
with S2 presenting a conductivity level similar to the one of S1. Different levels of association might be achieved
over time by playing with the test conditions, such as Vd, since S2 can display many conductivity levels.

Finally, V2=Vd is applied as a single input (55s<t<60s), and the current through S2 is high enough to trigger the
POST response, which is similar to the response evoked by single V1 application, so V2 has been effectively
associated to V1. In this work, we tested different Vd, ranging from 0.2V up to 0.4V, while keeping the rest of
test parameters fixed: Qthr=0.1mC and PW=0.1ms.

3. Results and Discussion
The relative conductivity change Δg2(t) of S2 during the conditioning stage is defined as the difference between
the measured conductivity g2(t), and the conductivity before the conditioning stage, gNC. In Figure 4, a few
examples of the evolution over time of the relative conductivity change normalized to gNC (Δg2(t)/gNC) measured
on the same device are depicted. Each color represents a different value of the employed Vd. Despite of using the
same test parameters, different Δg2(t) wav eforms are observed during the trials due to device-level variability.
In order to study the effects of the test parameters on the association process, the cumulative distribution
functions (cdf) of the conductivity values observed after conditioning, gCC=Δg2(t=55s), are shown in Figure 5,
where each cdf corresponds to a different Vd. Results in Figure 4 and 5 show that the associative process has a
dependence with the employed Vd. As to provide a probabilistic interpretation of the conditioning efficiency, an
association criterion in terms of a conductivity ratio threshold (CCRth) is defined, assuming that if the
conditioning conductivity ratio CCR=gCC/gNC is larger or equal than the CCRth (CCR≥CCRth), the conditioning
stage has been effective.




                                Figure 4. Δg2(t)/gNC examples for different Vd levels during the
                                conditioning stage. Each curve is related to the I2 values
                                measured when POST was firing a pulse for different Vd values,
                                performed on the same device.


In Figure 5, three criteria are marked: CCRth=0.1, CCRth=1 and CCRth=10. The probability of association is
computed as the number of trials where a particular association criterion CCRth is met, corresponding to the
number of trials in which the CCR is above CCRth, and corresponds to:

                                Probability of association = 1−P(CCR>=CCRth)

The probabilities of association as a function of the tested Vd values are depicted in Figure 6. It can be observed
that the probability of association increases with increasing Vd, up to the maxima Vd=0.325V. Above this value,
any association process can take place: the voltage drop of Vd, originated when V2 is active before the training
stage (−10s<t<−5s) is large enough to evoke a SET process to S2 on its own, being its conductivity state
increased in the wrong scenario.
                                                                             Figure 6. The association probability for three different
    Figure 5. Cdfs of the relative conductivity change achieved after        criteria of association (CCRth = 0.1,1 and 10) is represented as
    the conditioning stage, gCC/gNC, for differents Vd. The associative      a function of Vd. Probability increases with increasing Vd up to
    process requires a proper choice of Vd. Three association criteria       a limit were no association process takes place, since S2
    examples are depicted: CCRth1=0.1, CCRth2=1 and CCRth3=10.               conductivity changes before any pairing between V1 and V2 is
    Interpretation of these data is given in Figure 6.                       made. Maxima are found at Vd=0.325V.

4. Conclusions
An associative memory presents the ability to learn and recall the relationship between unrelated items, being the
interaction between neurons and the synaptic plasticity their basis. Animal-beings are able to develop non-
instinctive behaviors and establish symbolic relationships between facts or events through learning processes
such as classical conditioning, observed by I. Pavlov and his dogs. Taking advantage of the synaptic plasticity
property of memristive devices, the study of fundamental associative learning rules would suppose a step
towards the implementation of neuromorphic associative memories and hierarchical computing networks. In this
work, an electronic version of the Pavlov’s dog experiment to be employed in two or more memristive devices is
proposed, as a possible basis for implementing unsupervised associative learning within neuromorphic
architectures. The experiments show that the probability of having the post-synaptic neuron responding to the
neutral stimulus when it is the only active input is highly increased after the conditioning stage, in analogy to
Pavlov’s dog behavior. Because of device-level variability, an association criterion has to be defined. We tested
the association probability dependence on the amplitude of the employed voltage waveforms, and found that the
association level reached during conditioning increases with this parameter, up to a certain value, for which any
association learning process cannot occur.
References
[1] S.H Jo et al. Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 4 (2010) 1297-1301.
https://doi.org/10.1021/nl904092h
[2] T.Serrano-Gotarredona et al. STDP and STDP variations with memristors for spiking neuromorphic learning systems. Front Neurosci, 7
(2013): 2. https://doi.org/10.3389/fnins.2013.00002
[3] M.Prezioso et al. Self-Adaptive Spike-Time-Dependent Plasticity of Metal-Oxide Memristors. Sci Rep 6 (2016): 21331.
https://doi.org/10.1038/srep21331
[4] Z.Yide et al. Implementation of memristive neural networks with spike-rate-dependent plasticity synapses. IEEE IJCNN (2014): 2226-
2233. https://doi.org/10.1109/IJCNN.2014.6889740
[5] A.H. Klopf. A neuronal model of classical conditioning. Psychobiology, 16.2 (1988): 85-125. https://doi.org/10.3758/BF03333113
[6] Z-H. Tan et al. Pavlovian conditioning demonstrated with neuromorphic memristive devices. Sci Rep 7.1 (2017): 713.
https://doi.org/10.1038/s41598-017-00849-7
[7] M. Ziegler et al. An Electronic               Version    of   Pavlov's     Dog.    Adv    Funct    Mater    22.13   (2012):    2744-2749.
https://doi.org/10.1002/adfm.201200244
[8] C. Wu et al. Mimicking classical conditioning based on a single flexible memristor. Adv Mater 29(10) (2017): 1602890.
https://doi.org/10.1002/adma.201602890
[9] O. Bichler et al. Pavlov's dog associative learning demonstrated on synaptic-like organic transistors. Neural Comput 25.2 (2013): 549-
566. https://doi.org/10.1162/NECO_a_00377.
[10] E. Marder. Variability, compensation, and modulation in neurons and circuits. PNAS 108. Supplement 3 (2011): 15542-15548.
https://doi.org/10.1073/pnas.1010674108
[11] D. Parker. Synaptic Variability Introduces State-Dependent Modulation of Excitatory Spinal Cord Synapses. Neural Plast. 2015; 2015:
512156. https://doi.org/10.1155/2015/512156
[12] A. Neishabouri and A.A. Faisal. Axonal Noise as a Source of Synaptic Variability. PLoS Comput Bio 10.5 (2014): e1003615.
https://doi.org/10.1371/journal.pcbi.1003615
[13] MD. McDonnell et al. Neuronal stochastic variability: influences on spiking dynamics and network activity. Front Comput Neurosci
10 (2016): 38. https://doi.org/10.3389/fncom.2016.00038
[14] S. Poblador et al. Investigation of the multilevel capability of TiN/Ti/HfO2/W resistive switching devices by sweep and pulse
programming. Microelectron Eng 187-188 (2018): 148-153. https://doi.org/10.1016/j.mee.2017.11.007

[15] M. Pedro et al. A Flexible Characterization Methodology of RRAM: Application to the Modeling of the Conductivity Changes as
Synaptic Weight Updates. Solid-State Electron Special Issue of EUROSOI-ULIS 2018 Accepted (2019).
 [16] M. Pedro et al. Tuning the conductivity of resistive switching devices for electronic synapses. Microelectron Eng Special Issue of
INFOS 2017, 178, 89-92. https://doi.org/10.1016/j.mee.2017.04.040

Ackowledgements
This work has been partially supported by the Spanish MINECO and ERDF (TEC2016-75151-C3-1-R,
TEC2013-45638-C3-1-R and TEC2017-84321-C4-1-R), and it has made use of the Spanish ICTS Network
MICRONANOFABS.