164 
MATHEMATICAL ANALYSIS OF LEARNING BEHAVIOR 
OF NEURONAL MODELS 
BY 
.JOHN Y. CHEUNG 
MASSOUD OMIDVAR 
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE 
UNIVERSITY OF OKLAHOMA 
NORMAN, OK 73019 
Presented to the IEEE Conference on Neural Information Processing Systems- 
Natural and Syntheticf Denver, November 8-12, 1987, and to be published in 
the Collection of Papers from the IEEE Conference on NIPS. 
Please address all further correspondence to: 
John Y. Cheung 
School of EECS 
202 W. Boyd, CEC 219 
Norman, OK 73019 
(405)325-4721 
November, 1987 
 American Institute of Physics 1988 
165 
MATHEMATICAL ANALYSIS OF LEARNING BEHAVIOR 
OF NEURONAL MODELS 
John Y. Cheung and Massoud Omidvar 
School of Electrical Engineering 
and Computer Science 
ABSTRACT 
In this paper, we wish to analyze the convergence behavior of a number 
of neuronal plasticity models. Recent neurophysiological research suggests that 
the neuronal behavior is adaptive. In particular, memory stored within a neuron 
is associated with the synaptic weights which are varied or adjusted to achieve 
learning. A number of adaptive neuronal models have been proposed in the 
literature. Three specific models will be analyzed in this paper, specifically the 
Hebb model, the Sutton-Barto model, and the mot recent trace model. In this 
paper we will examine the conditions for convergence, the position of conver- 
gence and the rate at convergence, of these models as they applied to classical 
conditioning. Simulation results are also presented to verify the analysis. 
INTRODUCTION 
A number of static models to describe the behavior of a neuron have been 
in use in the past decades. More recently, research in neurophysiology suggests 
that a static view may be insufficient. Rather, the parameters within a neuron 
tend to vry with past history to achieve learning. It was suggested that by 
altering the internal parameters, neurons may adapt themselves to repetitive 
input stimuli and become conditioned. Learning thus occurs when the neurons 
are conditioned. To describe this behavior of neuronal plasticity, a number 
of models have been proposed. The earliest one may have been postulated 
by Hebb and more recently by Sutton and Barto . We will also introduce a 
new model, the most recent trce (or MRT) model in this paper. The primary 
objective of this paper, however, is to analyze the convergence behavior of these 
models during adaptation. 
The general neuronal model used in this paper is shown in Figure 1. There 
are a number of neuronal inputs xi(t),i = 1,... ,N. Ech input is scaled by 
the corresponding synaptic weights wi(t),i - 1,...,N. The weighted inputs 
are arithmetically summed. 
y(t) =  xi(t)wi(t) - e(t) (1) 
i----I 
where O(t) is taken to be zero. 
166 
Neuronal inputs are assumed to take on numerical values ranging from zero 
to one inclusively. Synaptic weights are allowed to take on any reasonable values 
for the purpose of this paper though in reality, the weights may very well be 
bounded. Since the relative magnitude of the weights and the neuronal inputs 
are not well defined at this point, we will not put a bound on the magnitude 
of the weights also. The neuronal output is normally the result of a sigmoidal 
transformation. For simplicity, we will approximate this operation by a linear 
transformation. 
x 1 
x 2 
$yuaptc 
vl ca Silodial 
Transformation 
v2   Y 
neurorl 
output 
Figure 1. A eneral euronal m!el. 
For convergence analysis, we will assume that there are only two neuronal 
inputs in the traditional classical conditioning environment for simplicity. Of 
course, the analysis techniques can be extended to any number of inputs. In 
classical conditioning, the two inputs are the conditioned stimulus x(t) and 
the unconditioned stimulus x,(t). 
THE SUTTON-BARTO MODEL 
More recently, Sutton and Barto I have proposed an adaptive model based 
on both the signal trace i(t) and the output trace (t) as given below: 
wi(t + 1) =wi(t) + ci(t)(y(t)) - (t) 
(t + 1) =/3(t) + (1 - t3)y(t) 
 ,(t + 1)=.,(t)+ 
(2a) 
(2b) 
(2c) 
where both a and f/are positive constants. 
167 
Condition of Convergence 
In order to simplify the analysis, we will choose  = 0 and/3 = 0, i.e.: 
(t) - (t- ) 
nd 
y(t)=y(t-1) 
In other words, (2a) becomes: 
wi(t + 1) = wi(t) + xi(t)(y(t) - y(t - 1)) 
(3) 
The above assumption only serves to simplify the analysis and will not affect the 
convergence conditions because the boundedhess of i(t) and (t) only depends 
on that for xi(t) and y(t- 1) respectively. 
As in the previous section, we recognize that (3) is a recurrence relation so 
convergence can be checked by the ratio test. It is also possible to rewrite (3) 
in matrix format. Due to the recursion of the neuronal output in the equation, 
we will include the neuronal output y(t) in the parameter vector also: 
(t + ) ) 
w(t 9- 1) 
y(t) 
= cx(t)x(t) l+cx](t) -cx(t) w(t) 
x(t) x(t) o y(t- 1) 
(4) 
or 
w(S-")(t + ) = A(s-.) WS-)(t ) 
To show convergence, we need to set the magnitude of the determinant of 
A (s-B) to be less than unity. 
It(s-")l- c(;(t) + (t)) (s) 
Hence, the condition for convergence is: 
1 
c < (t) + (t) 
From (6), we can see that the adaptation constant must be chosen to be less 
than the reciprocal of the Euclidean sum of energies of all the inputs. The 
same techniques can be extended to any number of inputs. This can be proved 
merely by following the same procedures outlined above. 
Position At Convergence 
168 
Having proved convergence of the Sutton-Barto model equations of neu- 
tonal plasticity, we want to find out next at what location the system remains 
when converged. We have seen earlier that at convergence, the weights cease to 
change and so does the neuronal output. We will denote this converged position 
as (w(S-S)) * -= W(S-S)(oo). In other words: 
(w(S-S)) * = A(S-S)(w(S-S)) * 
(7) 
Since any arbitrary parameter vector can always be decomposed into a weighted 
svm of the eigenvectors, i.e. 
w(S-S)(O) = cq + c2V2 + a3V3 
The constants a, a2, and a3 can easily be found by inverting A (s-s). The 
eigenvalues of A ($-s) can be shown to be 1, 1, and c(x q- x22). When c is 
within the region of convergence, the magnitude of the third eigenvalue is less 
than unity. That means that at convergence, there will be no contribution from 
the third eigenvector. Hence, 
lim w(S_S)(t ) _ a -F a2V2 (O) 
(w(S-S))' = t --. oo 
From (9), we can predict precisely what the converged position would be given 
only with the initial conditions. 
Rate of Convergence 
We have seen that when  is carefully chosen, the Sutton-Barto model will 
converge and we have also derived an expression for the converged position. 
Next we want to find out how fast convergence can be attained. The rate 
of convergence is a measure of how fast the initial parameter approaches the 
optimal position. The asymptotic rate of convergence is2: 
Roo( A (s-a)): -logS( A (s-a)) (10) 
where S(A (s-s) is the spectral radius and is equalled to c(x + x) in this 
case. This completes the convergence analysis on the Sutton-Barto model of 
neuronal plasticity. 
THE MRT MODEL OF NEURONAL PLASTICITY 
The most recent trace (MRT) model of neuronal plasticity 3 developed by 
the authors can be considered as a cross between the Sutton-Barto model and 
the Klopf's model 4. The adaptation of the synaptic weights can be expressed 
as follows: 
wi(t -f- 1) --- wi(t) -f- cwi(t)xi(t)(y(t) - y(t - 1)) (11) 
169 
A comparison of (11) nd the Sutton-Barto model in (3) show that the econd 
term on the right hand side contnz n extra factor, w(t), which is used to 
speed up the convergence z shown later. The output trace hz been replaced 
by It(t- 1), the most recent output, hence the name, the most recent trace 
model. The input trace is also replaced by the most recent input. 
Condition of Convergence 
We can now proceed to analyse the condition of convergence for the MRT 
model. Due to the presence of the ,vi(t) factor in the second term in (31), the 
ratio test cannot be applied here. To nalyze the convergence behavior further, 
let us rewrite (11) in matrix format: 
x,(t)) (xz(t) 0 
+c(w,(,) w,(,) 1)) x,(,) o ,,(,) 
-1 0 0 
(12) 
or 
W(M!tT)(t -t- 1) = A(M!tT)w(M!tT)(t) 't- c (W(MRT)(t))T.Bcv(MRT)(t) 
The superscript T denotes the matrix transpose operation. The above equation 
is quadratic in W(tRr)(t). Complete convergence analysis of this equation is 
extremely difficult. 
In order to understand the convergence behavior of (12), we note that 
the dominant term that determines convergence mainly relates to the second 
quadratic term. Hence for convergence analysis only, we will ignore the first 
term: 
W(MItr)(t -t- 1)  (W(MItr)(t))r BCW(MIrr)(t) (13) 
We can readily see from above that the primary convergence factor is BrC '. 
Since C is only dependent on xi(t), convergence can be obtained if the duration 
of the synaptic inputs being active is bounded. It can be shown that the 
condition of convergence is bounded by: 
1 
< + 
170 
We can readily see that the adaptation constant c can be chosen according 
to (14) to ensure convergence for t < T. 
SIMULATIONS 
To verify the theoretical analysis of these three adaptive neuronal models 
based on classical conditioning, these models have been simulated on the IBM 
3081 mainframe using the FORTRAN language in single precision. Several test 
scenarios have been designed to compare the analytical predictions with actual 
simulation results. 
To verify the conditions for convergence, we will vary the value of the 
adaptation constant c. The conditioned and unconditioned stimuli were set 
to unity and the value of c varies between 0.1 to 1.0. For the Sutton-Barto 
model the simulation given in Fig. 2 shows that convergence is obtained for 
c < 0.$ as expected from theoretical analysis. For the MRT model, simulation 
results given in Fig. 3 shows that convergence is obtained for c < 0.7, also as 
expected from theoretical analysis. The theoretical location at convergence for 
the Sutton and Barto model is also shown in Figure 2. It is readily seen that 
the simulation results confirm the theoretical expectations. 
2: c - 0.2 
3: c - 0.3 
&: c - O.& 
5: c - 0.5 
6: c - 0.6 
7: c - 0.7 
Bmber of Iterations 
Figure 2. Plots of euro''l outputs versus the mber of iterations for the 
$utton-arto model rtb different values of ndaptatou constant c. 
171 
urol 
Otput 
2: 
3: 
&: 
6: 
c-O.1 
c-0.2 
c-0.3 
c-O.& 
c - 0.. 
c-0.6 
Bumbet of Aterations 
Figure 3. Plots of neurons! outputs versus the utmber of iterations 
for the JfJ7 model with different values of mdatation 
col tnnt c. 
To illustrate the rate of convergence, we will plot the trajectory of the 
deviation in synaptic weights from the optimal values in the logarithmic scale 
since this error is logarithmic as found earlier. The slope of the line yields the 
rate of convergence. The trajectory for the Sutton-Barto Model is given in 
Figure 4 while that for the MRT model is given in Figure 5. It is clear from 
Figure 4 that the trajectory in the logarithmic form is a straight line. The 
slope ,(A (s-B)) can readily be calculated. The curve for the MRT model 
given in Figure 5 is also a straight line but with a much larger slope showing 
faster convergence. 
SUMMARY 
In this paper, we have sought to discover analytically the convergence 
behavior of three adaptive neuronal models. From the analysis, we see that 
the Hebb model does not converge at all. With constant active inputs, the 
output will grow exponentially. In spite of this lack of convergence the Hebb 
model is still a workable model realizing that the divergent behavior would 
be curtailed by the sigmoidal transformation to yield realistic outputs. The 
172 
Figure &. 
Trajectories of neuroual output deviations frou static values 
for the Suttore-Barto oael rlth ifferent values of misptation 
constant c. 
!/usber of iterations 
Figure 5. 
Trajectories of neurons] output deviations froe static 
values for the MT model vitb different values of 
adaptation constant c, 
173 
analysis on the Sutton and Barto model shows that this model will converge 
when the adaptation constant c is carefully chosen. The bounds for c is also 
found for this model. Due to the structure of this model, both the location at 
convergence and the rate of convergence are also found. We have also introduced 
a new model of neuronal plasticity called the most recent trace (MRT) model. 
Certain similarities exist between the MRT model and the Sutton-Barto model 
and also between the MRT model and the Klopf model. Analysis shows that the 
update equations for the synaptic weights are quadratic resulting in polynomial 
rate of convergence. Simulation results also show that much faster convergence 
rate can be obtained with the MRT model. 
REFERENCES 
1. Sutton, R.S. and A.G. Barto, Psychological Review, vol. 88, p. 135, (1981). 
2. Hageman, L. A. and D.M. Young. Applied Interactive Methods. (Aca- 
demic Press, Inc. 1981). 
3. Omidvar, Massoud. Analysis of Neuronal Plasticity. Doctoral disserta- 
tion, School of Electrical Engineering and Computer Science, University of 
Oklahoma, 1987. 
4. Klopf, A.H. Proceedings of the American Institute of Physics Conference 
#151 on Neural Networks for Computing, p. 265-270, (1986). 
