A Rigorous Analysis Of 
Linsker-type Hebbian Learning 
J. Feng 
Mathematical Department 
University of Rome "La Sapienza" 
P. le A. Moro, 00185 Rome, Italy 
f engmat. uniromal. it 
H. Pan V.P. Roychowdhury 
School of Electrical Engineering 
Purdue University 
West Lafayette, IN 47907 
hpanecn. purdue. edu 
vwanidrum. ecn. purdue. edu 
Abstract 
We propose a novel rigorous approach for the analysis of Linsker's 
unsupervised Hebbian learning network. The behavior of this 
model is determined by the underlying nonlinear dynamics which 
are parameterized by a set of parameters originating from the Heb- 
bian rule and the arbor density of the synapses. These parameters 
determine the presence or absence of a specific receptive field (also 
referred to as a 'connection pattern') as a saturated fixed point 
attractor of the model. In this paper, we perform a qualitative 
analysis of the underlying nonlinear dynamics over the parameter 
space, determine the effects of the system parameters on the emer- 
gence of various receptive fields, and predict precisely within which 
parameter regime the network will have the potential to develop 
a specially designated connection pattern. In particular, this ap- 
proach exposes, for the first time, the crucial role played by the 
synaptic density functions, and provides a complete precise picture 
of the parameter space that defines the relationships among the 
different receptive fields. Our theoretical predictions are confirmed 
by numerical simulations. 
320 Jianfeng Feng, H. Pan, V. P. Roychowdhury 
I Introduction 
For the purpose of understanding the self-organization mechanism of primary vi- 
sual system, Linsker has proposed a multilayered unsupervised Hebbian learning 
network with random uncorrelated inputs and localized arborization of synapses 
between adjacent layers (Linsker, 1986 & 1988). His simulations have shown that 
for appropriate parameter regimes, several structured connection patterns (e.g., 
centre-surround and oriented afferent receptive fields (aRFs)) occur progressively 
as the Hebbian evolution of the weights is carried out layer by layer. The behavior 
of Linsker's model is determined by the underlying nonlinear dynamics which are 
parameterized by a set of parameters originating from the Hebbian rule and the 
arbor density of the synapses. For a nonlinear system, usually, there coexist several 
attractors for the same set of system parameters. That is, for a given set of the 
parameters, the state space comprises several attractive basins, each corresponding 
to a steady state respectively. The initial condition determines which attractor will 
be eventually reached. At the same time, a nonlinear system could have a different 
group of coexisting attractors for a different set of system parameters. That is, 
one could make the presence or absence of a specific state as a fixed point attrac- 
tor by varying the set of the parameters. For a development model like Linsker's 
network, what is expected to be observed is that the different aRFs could emerge 
under different sets of parameters but should be relatively not sensitive to the initial 
conditions. In other words, the dynamics should avoid the coexistence of several 
attractors in an appropriate way. The purpose of this paper is to gain more insights 
into the dynamical mechanism of this self-organization model by performing a rig- 
orous analysis on its parameter space without any approximation. That is, our goal 
is to reveal the effects of the system parameters on the stability of aRFs, and to 
predict precisely within which parameter regime the network will have the poten- 
tial to develop a specially designated aRF. The novel rigorous approach presented 
here applies not only to the Linsker-type Hebbian learning but also to other related 
self-organization models about neural development. 
In Linsker's network, each cell in the present layer J4 receives synaptic inputs from 
a number of cells in the preceding layer/2. The density of these synaptic connections 
decreases monotonically with distance rr from the point underlying the Jd-cell's 
position. Since the synaptic weights change on a long time scale compared to the 
variation of random inputs, by averaging the Hebb rule over the ensemble of inputs 
in layer /2, the dynamical equation for the development of the synaptic strength 
Wr (i) between a J4-cell and i-th/2-cell at time r is 
N 
3w_[_1(i ) -' f {3-( i) --[- k 1 --[- E[Qi  .-[-/c2]r(j)dw(j) } 
j--1 
(1) 
where kl, k2 are system parameters which are particular combinations of the con- 
stants of the Hebb rule, r(-) is a non-negative normalized synaptic density function 
(SDF) 1, and 'ier r(i) = 1, and f(.) is a limiter function defined by f(x) = 
if x > w,,; = x, if lxl _ w,,; and = -w,,, if x < -w,,. The covariance 
1The SDF is explicitly incorporated into the dynamics (1) which is equivalent to 
Linsker's formulation. A rigorous explanation for this equivalence is given in MacKay 
Miller, 1990. 
A Rigorous Analysis of Linsker-type Hebbian Learning 321 
matrix {Qij } of the layer  describes the correlation of activities of the i-th and the 
j-th -cells. Actually, the covariance matrix of each layer is determined by SDFs 
r(.) of all layers preceding the layer under consideration. 
The idea of this paper is the following. It is well known that in general it is in- 
tractable to characterize the behavior of a nonlinear dynamics, since the nonlinearity 
is the cause of the coexistence of many attractors. And one has the difficulty in 
obtaining the complete characteristics of attractive basins in the state space. But 
usually for some cases, it is relatively easy to derive a necessary and sufficient con- 
dition to check whether a given state is a fixed point of the dynamics. In terms of 
this condition, the whole parameter regime for the emergence of a fixed point of the 
dynamics may be obtained in the parameter space. If we are further able to prove 
the stability of the fixed point, which implies that this fixed point is a steady state 
if the initial condition is in a nonempty vicinity in the state space, we can assert 
the occurrence of this fixed point attractor in that parameter regime. For Linsker's 
network, fortunately, the above idea can be carried out because of the specific form 
of the nonlinear function f(.). Due to space limitations, the rigorous proofs are in 
(Feng, Pan, & Roychowdhury, 1995). 
2 
The Set Of Saturated Fixed Point Attractors And The 
Criterion For The Division Of Parameter Regimes 
In fact, Linsker's model is a system of first-order nonlinear difference equations, 
taking the form 
W+l(i) = f[w(i) + hi(w.,kl, k2)], w = {w(j),j = 1, ..., Nr}, (2) 
Na 
where hi(w,kl,k2) = k + y.j=[Q/ + k2]r(j)w(j). And the aRFs observed in 
Linsker's simulation are the saturated fixed point attractors of this nonlinear system 
(2). Since the limiter function f(.)is defined on 
in weight state space within which the dynamics is dominated by the linear system 
w+l (i) = w(i) + hi(w, k l, k2), the short-time behaviors of evolution dynamics of 
connection patterns can be fully characterized in terms of the properties of eigen- 
vectors and their eigenvalues. But this method of stability analysis will not be 
suitable for the long-time evolution of equation (1) or (2), provided the hypercube 
constraint is reached as the first largest component of w reaches saturation. How- 
ever, it is well-known that a fixed point or an equilibrium state of dynamics (2) 
satisfies 
Wr(i) = f[w(i) + hi(wr,kl,k2) ]. (3) 
Because of the special form of the nonlinear function f(.), the fixed point equation 
(3) implies that 3T, such that for r > T, 
l w(i) + h(w,kl,kS) l _> 
So a saturated fixed point wr(i) must have the same sign as 
if hi(w, kl, ks)  O. 
hi(w-, kl, ks), i.e. 
w.(i)hi(w.,kl,ks) > O. 
By using the above idea, our Theorems 1 & 2 (proven in Feng, Pan, & Roychowd- 
hury, 1995) state that the set of saturated fixed point attractors of the dynamics in 
322 Jianfeng Feng, tt. Pan, V. P. Roychowdhury 
equation (1) is given by 
FP ---- {W [ co(i)hi(co-, kl, k2) > O, 1 _< i _< N}, 
and co  f,, is stable, where the weight vector co belongs to the set of all extreme 
points of the hypercube f (we assume coma = 1 without loss of generality). 
We next derive an explicit necessary and sufficient condition for the emergence of 
structured aRFs, i.e., we derive conditions to determine whether a given co belongs 
to fire. Define J+(co) = {i I co(i) = 1} as the index set of cells at the preceding layer 
 with excitatory weight for a connection pattern co, and J-(co) = {ilco(i ) = -1} 
as the index set of -cells with inhibitory weight for w. Note from the property of 
fixed point attractors that a connection pattern co is an attractor of the dynamics 
(1) if and only if for i  J+(co), we have 
co(i){]gl q- E[Q/5 q- ]g2I/'(J)co(j)) '-- 
co(i){kl q- EjeJ+(w)[Qj q- k2]r(j)co(j) + Ejes-()[Qi5 + k2]r(j)co(j)} > O. 
By the definition of J+(co) and J-(co), we deduce from the above inequality that 
+ [o5 + [o5 + > 0 
jea+() jes-() 
namely 
kl+ k2[ E 
jes+() 
j e s-() j e s-() j e s+() 
Inequality above is satisfied for M1 i in J+(w), and the left hand is independent of 
i. Hence, 
k+k,[  r(j)-  r(j)]> max [ P Qr(j)-  Qr(j)]. 
On the other hand, for i  J-(w), we can similarly deduce that 
k+k2[ E r(j) E r(j)]< min [ E  ' 
- %r(3)  ' 
- QirO)]- 
ies-() 3 
 +()  -() '  -() +() 
We introduce the slope function: 
C(co) de-"":'f E r(j)-- E 
jes+(,) 
which is the difference of sums of the SDF r(.) over J+(co) and J-(co), and two 
k-intercept functions: 
der{ 
a(co) = 
and 
def{ 
a=(co) = 
max/e.+(w)(5'jeg_(w) QiSr(j) - .+() QiSr(j)), 
min/es-()(5'jes-() QiS r(j) - 5's+() Qjr(j)), 
if S+(co)  0 
if J+(co) = 0 
if J-(co) k 0 
if J-(co) = 0 
A Rigorous Analysis of Linsker-type Hebbian Learning 323 
C k, C E E k F F 
F F H E E 
D D 
(a) (b) 
Figure 1: The parameter subspace of (/1,]c2). (a) Parameter regime of (]Cl,/2) 
to ensure the emergence of all-excitatory (regime A) and all-inhibitory (regime B) 
connection patterns. The dark grey regime C is the coexistence regime for both 
all-excitatory and all-inhibitory connection patterns. And the regime D without 
texture are the regime that Linsker's simulation results are based on, in which both 
all-excitatory and all-inhibitory connection patterns are no longer an attractor. (b) 
The principal parameter regimes. 
Now from our Theorem 3 in Feng, Pan, & Roychowdhury, 1995, for every layer of 
Linsker's network, the new rigorous criterion for the division of stable parameter 
regimes to ensure the development of various structured connection patterns is 
d2() > k + c()k2 > dl(). 
That is, for a given SDF r(.), the parameter regime of (k, k2) to ensure that o is a 
stable attractor of dynamics (1) is a band between two parallel lines kl + c(co)k2 > 
dl(O) and k +c(o)ku < du(w) (See regimes E and F in Fig.l(b)). It is noticed that 
as dl(W) > d2(o), there is no regime of (kl, k2) for the occurrence of that aRE as 
an attractor of equation (1). Therefore, the existence of such a structured aRE w as 
an attractor of equation (1)is determined by k-intercept functions dl(.) and d2(.), 
and therefore by the covariance matrix Qr or SDFs r(.) of all preceding layers. 
3 Parameter Regimes For aRFs Between Layers /3 And C 
Based on our general theorems applicable to all layers, we mainly focus on describing 
the stabilization process of synaptic development from the 2nd (B) to the 3rd layer 
(C) by considering the effect of the system parameters on the weight development. 
For the sake of convenience, we assume that the input at 1st layer (,4) is independent 
normal distribution with mean 0 and variance 1, and the connection strengths from 
layer ,4 to B are all-excitatory same as in Linsker's simulations. The emergence of 
various aREs between layer B and C have been previously studied in the literature, 
and in this paper we mention only the following new results made possible by our 
approach: 
(1) For the cell in layer C, the all-excitatory and the all-inhibitory connection 
patterns still have the largest stable regimes. Denote both SDFs from layer A to B 
and from B to C as r'4s( ., .) and rtc(.) respectively. The parameter plane of (kl, k2) 
324 Jianfeng Feng, H. Pan, V. P. Roychowdhury 
Table 1: The Principal Parameter Regimes 
TYPE 
Regime 
A 
Regime B 
Regime C 
=Af3B 
Regime D 
=(Au B)  
Regime E 
PARAMETER REGIME 
kl+k2 >dl(+) 
(approx. -k/k2 < 1) 
kl - k < d(-) 
(approx. -k/k2 > -1) 
kl q-k > dl(+) and 
k - k2 < d(-) 
 < d(+)=-d(-) .d 
approx. -1 < -k/k < I 
Regime F 
d,() > 1 + c(,o) > dl(,,,) 
where c(w) > 0 
d() > kl + c() > dl() 
where c(w) < 0 
ATTRACTOR 
All-excitatory aRF 
All-inhibitory aRF 
All-excitatory and aB-inhibitory 
aRFs coexist 
The structured aRFs may have 
separate parameter regimes 
Any connection pattern in which 
the excitatory connections 
constitute the majority 
Any connection pattern in which 
the inhibitory connections 
constitute the majority 
Regime G 
=EnFnAvB 
d2(0./1) ' kl 4- C(. 11)k2 ' d! 
d(.) > 1 + c(.)k > 
A small coexistence regime of 
many connection patterns around 
the origin point of the parameter 
plane of (ki, k2) 
is divided into four regimes by 
kz + ks > dr(+) = - 
Ns NA 
min 1  " (i' 0" (j' )'0) 
l (i(Ns 
- - '= I=1 
for all-excitatory pattern and 
Ns NA 
min . E r'413(i' l)r'a13(j' l)rt3c(J) = -dl(2r-) 
i<i<N/s j__ I--1 
for all-inhibitory pattern (See Fig. l(a)). 
(2) The parameter with large and negative ks and approximately -1 < -k/ks < 1 
is favorable for the emergence of various structured connection patterns (e.g., ON- 
center cells, OFF-center cells, bi-lobed cells, and oriented cells). This is because 
this regime (See regime D in Fig.l) is removed from the parameter regime where 
both all-excitatory and all-inhibitory aRFs are dominant, including the coexistence 
regime of many kind of attractors around the origin point (See regime G in Fig. l(b)). 
The above results provide a precise picture about the principal parameter regimes 
summarized in Table 1. 
(3) The relative size of the radiuses of two SDFs r'as( ., -) and rsc(.) plays a key role 
in the evolution of various structured aRFs from B to . A given SDF r(i, j), i  
jr4, j   will be said to have a range r if r(i, j)is 'sufncient small' for IIi-Jll > 
r. For a Gaussian SDF r(j,k) ~ exp(-llj-kll/r), j  ,  M, the range 
r is its standard deviation. We give the analytic prediction about the influence of 
the SDF's ranges rs, rc on the dynamics by changing rs from the smallest extreme 
to the largest one with respect to re. For the smallest extreme of rs (i.e. the 
A Rigorous Analysis of Linsker-type Hebbian Learning 325 
synaptic connections from 4 to B are concentrated enough, and those from layer B 
to C are fully feedforward connected), we proved that any kind of connection pattern 
has a stable parameter regime and emerge under certain parameters, because each 
synaptic connection within an aRF is developed independently. As rt is changed 
from the smallest to the largest extreme, the development of synaptic connections 
between layer B and C will depend on each other stronger and stronger in the 
sense that most of connections have the same sign as their neighbors in an aRF. 
So for the largest extreme of rt (i.e. the weights from layer A to B are fully 
feedforward but there is no constraint on the SDF rtc(.)), any structured aRFs 
except for the all-excitatory and the all-inhibitory connection patterns will never 
arise at all, although there exist correlation in input activities (for a proof see Feng, 
Pan, & Roychowdhury, 1995). Therefore, without localized SDF, there would be 
no structured covariance matrix Q = {[Qij + k2]r(j)} which embodies localized 
correlation in afferent activities. And without structured covariance matrix ), no 
structured aRFs would emerge. 
(4) As another application of our analyses, we present several numerical results 
on the parameter regimes of (kl, k2, rs, Pc) for the formation of various structured 
aRFs (Feng & Pan, 1993; Feng, Pan, & Roychowdhury, 1995) (where we assume that 
r 2 
r'4t(i,j) ~ exp(-Ili-jll/r), i  B,j  .4, and risc(i) ~ exp(-IIil]/c), i  B as in 
(Linsker, 1986 & 1988)). For example, we show that various aRFs as attractors have 
different relative stability. For a fixed pc, the SDF's range rt of the preceding layer 
as the third system parameter has various critical values for different attractors. 
That is, an attractor will no longer be stable if rs exceeds its corresponding critical 
value (See Fig. 2). For circularly symmetric ON-center cells, those aRFs with large 
ON-center core (which have positive or small negative slope value c(w)  -kl/k2) 
always have a stable parameter regime. But for those ON-center cells with large 
negative slope value c(w), their stable parameter regimes decrease in size with 
Similarly, circularly symmetric OFF-center cells with large OFF-center core (which 
have negative or small positive slope value c(w)) will be more stable than those with 
large positive average of weights. But for non-circularly-symmetric patterns (e.g., 
bi-lobed cells and oriented cells), only those attractors with zero average synaptic 
strength might always have a stable parameter regime (See regime H in Fig.l(b)). 
If the third parameter rs is large enough to exceed its critical values for other aRFs 
and k2 is large and negative, then ON-center aRFs with positive c(o) and OFF- 
center aRFs with negative c(o) will be almost only attractors in regime DOlE and 
regime D91F respectively. This conclusion makes it clear why we usually obtain 
ON-center aRFs in regime DOlE and OFF-center aRFs in regime DFIF much more 
easily than other patterns. 
4 Concluding Remarks 
One advantage of our rigorous approach to this kind of unsupervised Hebbian learn- 
ing network is that, without approximation, it unifies the treatment of many diverse 
problems about dynamical mechanisms. It is important to notice that there is no 
assumption on the second item hi(o) on the right hand side of equation (1), and 
there is no restriction on the matrix Q. Our Theorems 1 and 2 provide the gen- 
eral framework for the description of the fixed point attractors for any difference 
equation of the type stated in (2) that uses a limiter function. Depending on the 
326 Jianfeng Feng, H. Pan, V. P. Roychowdhury 
ON-center aRFs (;= o) ;, Oriented aRFs ;== o) 
-I/k -k/k 
-095 -0.5 -0.68 -0.4! -0.1! 0.09 0.37 068 I -1 -0.8 -0.43 -0.22 0 0.37 0.52 0.7 O I 
Figure 2: The critical values of the SDF's range rs for different connection patterns. 
structure of the second item, hi(w), it is not difficult to adapt our Theorem 3 to 
obtain the precise relationship among system parameters in other kind of models 
as long as f(.) is a limiter function. Since the functions in the necessary and suf- 
ficient condition are computable (like our slope and kl-intercept functions), one is 
always able to check whether a designated fixed point is stable for a specific set of 
parameters. 
Acknowledgements 
The work of V. P. Roychowdhury and H. Pan was supported in part by the General 
Motors Faculty Fellowship and by the NSF Grant No. ECS-9308814. J. Feng 
was partially supported by Chinese National Key Project of Fundamental Research 
"Climbing Program" and CNR of Italy. 
References 
. Linsker. (1986) From basic network principle to neural architecture (series). 
Proc. Natl. Acad. $ci. USA 83: 7508-7512, 8390-8394, 8779-8783. 
R. Linsker. (1988) Self-organization in a perceptual network. Computer 21(3): 
105-117. 
D. MacKay,  K. Miller. (1990) Analysis of Linsker's application of Hebbian rules 
to linear networks. Network 1: 257-297. 
J. Feng,  H. Pan. (1993) Analysis of Linsker-type Hebbian learning: Rigorous 
results. Proc. 1993 IEEE Int. Conf. on Neural Networks - San Francisco Vol. III, 
1516-1521. Piscataway, NJ: IEEE. 
J. Feng, H. Pan,  V. P. Roychowdhury. (1995) Linsker-type Hebbian learning: A 
qualitative analysis on the parameter space. (submitted). 
