590 Atiya and Abu-Mostafa 
A Method for the Associative Storage 
of Analog Vectors 
Amir Atiya (*) and Yaser Abu-Mostafa (**) 
(*) Department of Electrical Engineering 
(**) Departments of Electrical Engineering and Computer Science 
California Institute Technology 
Pasadena, Ca 91125 
ABSTRACT 
A method for storing analog vectors in Hopfield's continuous feed- 
back model is proposed. By analog vectors we mean vectors whose 
components are real-valued. The vectors to be stored are set as 
equilibria of the network. The network model consists of one layer 
of visible neurons and one layer of hidden neurons. We propose 
a learning algorithm, which results in adjusting the positions of 
the equilibria, as well as guaranteeing their stability. Simulation 
results confirm the effectiveness of the method. 
1 INTRODUCTION 
The associative storage of binary vectors using discrete feedback neural nets has 
been demonstrated by Hopfield (1982). This has attracted a lot of attention, and 
a number of alternative techniques using also the discrete feedback model have 
appeared. However, the problem of the distributed associative storage of analog 
vectors has received little attention in literature. By analog vectors we mean vec- 
tors whose components are real-valued. This problem is important because in a 
variety of applications of associative memories like pattern recognition and vector 
quantization the patterns are originally in analog form and therefore one can save 
having the costly quantization step and therefore also save increasing the dimension 
of the vectors. In dealing with analog vectors, we consider feedback networks of the 
continuous-time graded-output variety, e.g. Hopfield's model (1984): 
du 
= -u + Wf(u) + a, x = f(u), (1) 
dt 
where u - (ux, ..., us) T is the vector of neuron potentials, x - (xx, ..., XN) T is the 
vector of firing rates, W is the weight matrix, a is the threshold vector, and f(u) 
means the vector (f(ul),..., f(uN)) T where f is a sigmoid-shaped function 
, 
The vectors to be stored are set as equilibria of the network. Given a noisy version 
of any of the stored vectors as the initial state of the network, the network state has 
A Method for the Associative Storage of Analog Vectors 591 
to reach eventually the equilibrium state corresponding to the correct vector. An 
important requirement is that these equilibria be asymtotically stable, otherwise 
the attraction to the equilibria will not be guaranteed. Indeed, without enforcing 
this requirement, our numerical simulations show mostly unstable equilibria. 
2 THE MODEL 
It can be shown that there are strong limitations on the set of memory vectors 
which can be stored using Hopfield's continuous model (Atiya and Abu-Mostafa 
1990). To relieve these limitations, we use an architecture consisting of both visible 
and hidden units. The outputs of the visible units correspond to the components of 
the stored vector. Our proposed architecture will be close to the continuous version 
of the BAM (Kosko 1988). The model consists of one layer of visible units and 
another layer of hidden units (see Figure 1). The output of each layer is fed as an 
input to the other layer. No connections exist within each of the layers. Let y and 
x be the output vectors of the hidden layer and the visible layer respectively. Then, 
in our model, 
du 
= -u + Wf(z) + a = e, y = f(u) (2a) 
dt 
dz 
= -z + Vf(u) + b _= h, x = f(z) (2b) 
dt 
where W = [wij] and V = [vii] are the weight matrices, a and b are the threshold 
vectors, and f is a sigmoid function (monotonically increasing) in the range from 
-1 to 1, for example 
f(u) = tanhCu). 
hidden visible 
Figure 1: The model 
592 Atiya and Abu-Mostafa 
As we mentioned before, for a basin of attraction to exist around a given mem- 
ory vector, the corresponding equilibrium has to be asymtoticMly stable. For the 
proposed architecture a condition for stability is given by the following theorem. 
Theorem: An equilibrium point (u*, z*) satisfying 
(3a) 
for all i is asymptotically stable. 
(3b) 
Proof: We linearize (2a), (2b) around the equilibrium. We get 
dq 
du = Jq' 
where 
ui - u, if i = 1, ..., N 
qi = z* if i = Nx + 1, Nx + N2, 
Zi-N -- i-N , "', 
N and N2 are the number of units in the hidden layer and the visible layer respec- 
tively, and J is the Jacobian matrix, given by 
J __ 
the partial derivatives evaluated at the equilibrium point. Let Ax and A2 be re- 
spectively the N x Nx and N2 x N2 diagonal matrices with the ith diagonal element 
being respectively f'(u?) and f'(z). Furthermore, let 
a:(o o) 
A2 ' 
The J acobian is evaluated as 
j= (--IN WA) 
VAx --IN: 
where I means the L x L identity matrix. Let 
A Method for the Associative Storage of Analog Vectors 593 
Then, 
J=AA. 
Eigenvalues of AA are identical to the eigenvalues of AX/2AA 1/2 because if A is an 
eigenvalue of AA corresponding to eigenvector v, then 
AAv = Av, 
and hence 
(AX/2AAX/)(A/v) = A(AX/v). 
Now, we have 
/ --IN  . 1/2 .... X/) 
AIAA/ = ( al/2xra,/ A1 A 2 . 
By Gershgorin's Theorem (Franklin 1968), an eigenvalue of J h to satisfy at let 
one of the inequalities: 
i = 1, ..., Nx 
It follows that under conditions (3a), (3b) that the eigenvalues of J will have neg- 
ative real parts, and hence the equilibrium of the original system (2a), (2b) will be 
asymptotically stable. 
Thus, if the hidden unit values are driven far enough into the saturation region 
(i.e. with values close to 1 or -1), then the corresponding equilibrium will be stable 
because then, /'(u?) will be very small, causing Inequalities (3) to be satisfied. 
Although there is nothing to rule out the existence of spurious equilibria and limit 
cycles, if they occur then they would be far away from the memory vectors because 
each memory vector has a basin of attraction around it. In our simulations we have 
never encountered limit cycles. 
3 TRAINING ALGORITHM 
Let x'" m = 1, ..., M be the vectors to be stored. Each x'"should correspond to the 
visible layer component of one of the asymptotically stable equilibria. We design 
the network such that the hidden layer component of the equilibrium corresponding 
to x'"is far into the saturation region. The target hidden layer component y'"can 
be taken as a vector of l's and -l's, chosen arbitrarily for example by generating 
the components randomly. Then, the weights have to satisfy 
y? = f(Zwj,xn q- aj), 
1 
x?: f[Zvijf(Zwjtxr q- aj) q- bi]. 
594 Atiya and Abu-Mostafa 
Training is performed in two steps. In the first step we train the weights of the 
hidden layer. We use steepest descent on the error function 
m,j I 
In the second step we train the weights of the visible layer, using steepest descent 
on the error function 
E - Z[[x? - f[Zvijf(Zwjx n + aj)+ bi][[ 2. 
m,i j 
We remark that in the first step convergence might be slow since the targets are I or 
- 1. A way to have fast convergence is to stop if the outputs are within some constant 
(say 0.2) from the targets. Then we multiply the weights and the thresholds of the 
hidden layer by a big positive constant, so as to force the outputs of the hidden 
layer to be close to I or -1. 
4 IMPLEMENTATION 
We consider a network with 10 visible and 10 hidden units. The memory vectors are 
randomly generated (the components are from -0.8 to 0.8 rather than the full range 
to have a faster convergence). Five memory vectors are considered. After learning, 
the memory is tested by giving memory vectors plus noise (100 vectors for a given 
variance). Figure 2 shows the percentage correct recall in terms of the signal to 
noise ratio. Although we found that we could store up to 10 vectors, working close 
to the full capacity is not recommended, as the recall accuracy dteriorates. 
correct 
lOO 
8o 
60 
40 
8o 
-2 2 6 lO 
snr (db) 
Figure 2: Recall accuracy versus signal to noise ratio 
A Method for the Associative Storage of Analog Vectors 595 
Acknowledgement 
This work is supported by the Air Force Office of Scientific Research under grant 
AFOSR-88-0231. 
Peferences 
J. Hopfield (1982), "Neural networks and physical systems with emergent collective 
computational abilities", Proc. Nat. Acad. $ci. USA, vol. 79, pp. 2554-2558. 
J. Hopfield (1984), "Neurons with graded response have collective computational 
properties like those of two state neurons", Proc. Nat. Acad. Sci. USA, vol. 81, p. 
3088-3092. 
A. Atiya and Y. Abu-Mostafa (1990), "An analog feedback associative memory", 
to be submitted. 
B. Kosko (1988), "Bidirectional associative memories", IEEE Trans. Syst. Man 
Cybern., vol SMC-18, no. 1, pp. 49-60. 
J. Franklin (1968) Matrix Theory, Prentice-Hall, Englewood Cliffs, New Jersey. 
PART VII: 
EMPIRICAL ANALYSES 
