Asynchronous Dynamics of Continuous 
Time Neural Networks 
Xin Wang 
Computer Science Department 
University of California at Los Angeles 
Los Angeles, CA 90024 
Qingnan Li 
Department of Mathematics 
University of Southern California 
Los Angeles, CA 90089-1113 
Edward K. Blum 
Department of Mathematics 
University of Southern California 
Los Angeles, CA 90089-1113 
ABSTRACT 
Motivated by mathematical modeling, analog implementation and 
distributed simulation of neural networks, we present a definition of 
asynchronous dynamics of general CT dynamical systems defined 
by ordinary differential equations, based on notions of local times 
and communication times. We provide some preliminary results 
on globally asymptotical convergence of asynchronous dynamics 
for contractive and monotone CT dynamical systems. When ap- 
plying the results to neural networks, we obtain some conditions 
that ensure additive-type neural networks to be asynchronizable. 
1 INTRODUCTION 
Neural networks are massively distributed computing systems. A major issue in par- 
allel and distributed computation is synchronization versus asynchronization (Bert- 
sekas and Tsitsiklis, 1989). To fix our idea, we consider a much studied additive-type 
model (Cohen and Grossberg, 1983; Hopfield, 1984; Hirsch, 1989) of a continuous- 
time (CT) neural network of n neurons, whose dynamics is governed by 
ki(t) = -aixi(t) + E wiJ'J(ljxj  + Ii, i = 1,2, ...,n, (1) 
j=l 
493 
494 Wang, Li, and Blum 
with neuron states zi(t) at time t, constant decay rates ai, external inputs h, gains 
/d, neuron activation functions a i and synaptic connection weights wij. Simu- 
lation and implementation of idealized models of neural networks such as (1) on 
centralized computers not only limit the size of networks, but more importantly 
preclude exploiting the inherent massive parallelism in network computations. A 
truly faithful analog implementation or simulation of neural networks defined by 
(1) over a distributed network requires that neurons follow a global clock t, com- 
municate timed states xj(t ) to all others instantaneously and synchronize global 
dynamics precisely all the time (e.g., the same x I (t) should be used in evolution of 
all xi(t) at time t). Clearly, hardware and software realities make it very hard and 
sometimes impossible to fulfill these requirements; any mechanism used to enforce 
such synchronization may have an important effect on performance of the net- 
work. Moreover, absolutely insisting on synchronization contradicts the biological 
manifestation of inherent asynchrony caused by delays in nerve signal propagation, 
variability of neuron parameters such as refractory periods and adaptive neuron 
gains. On the other hand, introduction of asynchrony may change network dynam- 
ics, for example, from convergent to oscillatory. Therefore, validity of asynchronous 
dynamics of neural networks must be assessed in order to ensure desirable dynamics 
in a distributed environment. 
Motivated by the above issues, we study asynchronous dynamics of general CT dy- 
namical systems with neural networks in particular. Asynchronous dynamics has 
been thoroughly studied in the context of iterative maps or discrete-time (DT) dy- 
namical systems; see, e.g., (Bertsekas and Tsitsiklis, 1989) and references therein. 
Among other results are that P-contractive maps on It n (Baudet, 1978) and contin- 
uous maps on partially ordered sets (Wang and Parker, 1992) are asynchronizable, 
i.e., any asynchronous iterations of these maps will converge to the fixed points 
under synchronous (or parallel) iterations. The synchronization issue has also been 
addressed in the context of neural networks. In fact, the celebrated DT Hopfield 
model (Hopfield, 1982) adopts a special kind of asynchronous dynamics: only one 
randomly chosen neuron is allowed to update its state at each iterative step. The 
issue is also studied in (Barhen and Gulati, 1989) for CT neural networks. The 
approach there is, however, to convert the additive model (1) into a DT version 
through the Euler discretization and then to apply the existing result for contrac- 
tive mappings in (Baudet, 1978) to ensure the discretized system to be asynchro- 
nizable. Overall, studies for asynchronous dynamics of CT dynamical systems are 
still lacking; there are even no reasonable definitions for what it means, at least to 
our knowledge. 
In this paper, we continue our studies on relationships between CT and DT dy- 
namical systems and neural networks (Wang and Blum, 1992; Wang, Blum and Li, 
1993) and concentrate on their asynchronous dynamics. We first extend a concept 
of asynchronous dynamics of DT systems to CT systems, by identifying the distinc- 
tion between synchronous and asynchronous dynamics as (i) presence or absence of 
a common global clock used to synchronize the dynamics of the different neurons 
and (ii) exclusion or inclusion of delay times in communication between neurons, 
and present some preliminary results for asynchronous dynamics of contractive and 
monotone CT systems. 
Asynchronous Dynamics of Continuous Time Neural Networks 495 
2 MATHEMATICAL FORMULATION 
To be general, we consider a CT dynamical system defined by an n-dimensional 
system of ordinary differential equations, 
ki(t) = fi(ah(t),...,xn(t)), i= 1,2,...,n, 
(2) 
where fi: R n '--* R are continuously differentiable and x(t)  R n for all t in R+ (the 
set of all nonnegative real numbers). In contrast to the asynchronous dynamics 
given below, dynamics of this system will be called synchronous. An asynchronous 
scheme consists of two families of functions ci : R+  R+ and rj : R+  R+, 
i, j = 1, ..., n, satisfying the following constraints: for any t _> 0, 
(i) Initiation: ci(t) >_ 0 and rj(t) _> 0; 
(ii) Non-starvation: ci's are differentiable and ki(t) > 0; 
(iii) Liveness: limt_.. ci(t) = cx> and limt_... rj(t) 
(iv) Accessibility: rj(t) _< cj(t). 
Given an asynchronous scheme ({ci}, {rj}), the associated asynchronous dynamics 
of the system (2) is the solution of the following parametrized system: 
i(Ci()) --' 
We shall call this system an asynchronized system of the original one (2). 
The functions ci(t) should be viewed as respective "local" times (or clocks) of com- 
ponents i, as compared to the "global" time (or clock) t. As each component i 
evolves its state according to its local time ci(t), no shared global time t is needed 
explicitly; t only occurs implicitly. The functions rj(t) should be considered as time 
instants at which corresponding values xj of components j are used by component 
i; hence the differences (cj(t)- rj(t)) _> 0 can be interprated as delay times in 
communication between the components j and i. Constraint (i) reflects the fact 
that we are interested in the system dynamics after some global time instance, say 
0; constraint (ii) states that the functions ci are monotone increasing and hence the 
local times evolve only forward; constraint (iii) characterizes the liveness property 
of the components and communication channels between components; and, finally, 
constraint (iv) precludes the possibility that component i accesses states xj ahead 
of the local times cj(t) of components j which have not yet been generated. 
Notice that, under the assumption on monotonicity of ci (t), the inverses c- x (t) exist 
and the asynchronized system (3) can be transformed into 
9,(t) = a,(t) 9}(t), ..., 
(4) 
by letting yi(t) = xi(ci(t)) and .0j (t) = xj(rj(t)) = yj(cj-X(rj(t)) for i,j = 1,2,..., n. 
The vector form of (4) can be given by 
t = C'F[] (5) 
496 Wang, Li, and Blum 
w_here y(t) = [yx (t), ..., yn(t)] T, C' = diag(dc (t)/dt, ..., dcn(t)/dt), F = [f, ..., fn] T 
Y = [] and 
., 
r[?] = f2((t),(t),...,}(t)) . 
Notice that the complication in the way F applies to  simply means that every 
component i will use possibly different "global" states [.O(t),.O}(t), ...,-i 
y.(t)]. This 
peculiarity makes the equation (5) fit into none of the categories of general functional 
differential equations (Hale, 1977). However, if rj(t) for i : 1,...,n are equal, 
all the components will use a same global state .0 = [.O(t),.O(t),...,(t)l and 
the asynchronized system (5) assumes a form of retarded functional differential 
equations, 
= (6) 
We shall call this case uniformly-delayed, which will be a main consideration in the 
next section where we discuss asynchronizable systems. 
The system (5) includes some special cases. In a no communication delay situation, 
rj(t) = cj(t) for alii and the system (5) reduces to : C'F(y). This includes the 
simplest case where the local times ci(t) are taken as constant-time scalings cit of 
the global time t; specially, when all ci(t) = t the system goes back to the original 
one (2). If, on the other hand, all the local times are identical to the global time t 
and the communication times take the form of vj(t) = t - 0(t) one obtains a most 
general delayed system 
i(t) -- fi(Yl(t --O(t)),y2(t -- 0}(/)), ...,yn(t- O/n(/))), 
(7) 
where the state yj(t) of component j may have different delay times O(t) for dif- 
ferent other components i. 
Finally, we should point out that the above definitions of asynchronous schemes and 
dynamics are analogues of their counterparts for DT dynamical systems (Bertsekas 
and Tsitsiklis, 1989; Blum, 1990). Usually, an asynchronous scheme for a DT 
system defined by a map f  X -4 X, where X = Xx x X2 x  .. x X,, consists of a 
family {T i C_Nli: 1, ...,n} of subsets of discrete times (N) at which components 
i update their states and a family {vj  N -4 Nli = 1, 2, ..., n} of communication 
times. Asynchronous dynamics (or chaotic iteration, relaxation) is then given by 
;ri(l q- 1) -- { fi(Xl(T())'''"X"(Tni())) if t  T  
xi(t) otherwise. 
Notice that the sets T i can be interpreted as local times of components i. In fact, 
one can define local time functions ci ' N -4 N as ci(O) = 0 and ci(t + 1) = ci(t) + 1 
if t  T/ and ci(t) otherwise. The asynchronous dynamics can then be defined by 
xi(t + 1)- xi(t) = (ci(t + 1)-ci(t))(f(x(r(t)),...,,(r(t)))- x(t)), 
which is analogous to the definition given in (4). 
Asynchronous Dynamics of Continuous Time Neural Networks 497 
3 ASYNCHRONIZABLE SYSTEMS 
In general, we consider a CT dynamical system as asynchronizable if its synchronous 
dynamics (limit sets and their asymptotic stability) persists for some set of asyn- 
chronous schemes. In many cases, asynchronous dynamics of an arbitrary CT sys- 
tem will be different from its synchronous dynamics, especially when delay times 
in communication are present. An example can be given for the network (1) with 
symmetric matrix W. It is well-known that (synchronous) dynamics of such net- 
works is quasi-convergent, namely, all trajectories approach a set of fixed points 
(Hirsch, 1989). But when delay times are taken into consideration, the networks 
may have sustained oscillation when the delays exceed some threshold (Marcus and 
Westervelt, 1989). A more careful analysis on oscillation induced by delays is given 
in (Wu, 1993) for the networks with symmetric circulant weight matrices. 
Here, we focus on asynchronizable systems. We consider CT dynamical systems on 
R n of the following general form 
A(t) = -x(t) + F(t)) (8) 
where x(t)  R , A = diag(ax,a2,...,a, 0 with ai > 0 and F = [fi]  Cx(R). It 
is easy to see that a point x  R'* is a fixed point of (8) if and only if x is a fixed 
point of the map F. Without loss of generality, we assume that 0 is a fixed point 
of the map F. According to (5), the asynchronized version of (8) for an arbitrary 
asynchronous scheme ({ci), {rj}) is 
= + F[?]), (9) 
where 0 [9}(t),-2 t ..., 
= y2(), 92(t)]. 
3.1 Contractlye Systems 
Our first effort attempts to obtain a result similar to the one for P-contractive 
maps in (Baudet, 1978). We call the system (8) strongly P-contractive if there is 
symmetric and invertible matrix $ such that I$ -xF($x)[ <_ Ix I for all x G R n and 
[$-XF($x)[ = Ix[ only for x = 0; here Ix[ denotes the vector with components 
and <_ is component-wise. 
Theorem 1 If the system (8) is strongly P-contractive, then it is asynchronizable 
for any asynchronous schemes without self time delays (i.e., rj(t) - ci(t) for all 
i= 1,2,...,n). 
Proof. It is not hard to see that synchronous dynamics of a strongly P-contractive 
system is globally convergent to the fixed point 0. Now, consider the transformation 
z = A-y and the system for z 
A = C'(-z + $- F[S2]) = C'(-z + a[2]), 
where G[,] = S-XFS[,]. This system has the same type of dynamics as (9). 
Define a function E: R+ x R'*  R+ by E(t) = z ' (t)Az(t)/2, whose derivative 
with respect to t is 
k = z T C' (-z q- S()) < I1'11 (-z T z q-Izl T IS()l) < 11'11( -zT z + Izl T Izl) < 0. 
498 Wang, Li, and Blum 
Hence E is an energy function and the asynchronous dynamics converges to the 
fixed point 0. cl 
Our second result is for asynchronous dynamics of contractive systems with no 
communication delay. The system (8) is called contractive if there is a real constant 
0 < c < 1 such that 
II?(x)- r(y)11 < allz- yll 
for all x, y 6 Rn; here II. II denotes the usual Euclidean norm on R n. 
Theorem 2 If the system (8) is contractive, then it is asynchronizable for asyn- 
chronous schemes with no communication delay. 
Proof. The synchronous dynamics of contractive systems is known to be globally 
convergent to a unique fixed point (Kelly, 1990). For an asynchronous scheme with 
no communication delay, the system (8) is simplified to Ay: C (-y + F(y)). W 
consider again the function E = yq'Ay/2, which is an energy function as shown 
below. 
:k = y'C' (-y + r(y)) 5 IIC'11(-IlYll 2 q- [lYll [Ir(y)[I) < 0. 
Therefore, the asynchronous dynamics converges to the fixed point 0. r 
For the additive-type neural networks (1), we have 
Corollary 1 
type with 0 < a(z) _< supzcR a(z) = 1. If it satisfies the condition 
IM-xWMII < 1, 
where M -- diag(yx,...,yn), then it is asynchronizable for any 
schemes with no communication delay. 
Let the network (1) have neuron activation functions ri of sigmoidal 
(10) 
asynchronous 
Proof. The condition (10) ensures the map F(x) = A-1Wo'(Mx)q- A-I to be 
contractive. r 
Notice that the condition (10) is equivalent to many existing ones on globally asymp- 
totteal stability based on various norms of matrix W, especially the contraction con- 
dition given in (Kelly, 1990) and some very recent ones in (Matsuoka, 1992). The 
condition (10) is also related very closely to the condition in (Barhen and Gulati, 
1989) for asynchronous dynamics of a discretized version of (1) and the condition 
in (Marcus and Westervelt, 1989) for the networks with delay. 
We should emphasize that the results in Theorem 2 and Corollary 1 do not directly 
follow from the result in (Kelly, 1990); this is because local times ci(t) are allowed 
to be much more general functions than linear ones ci t. 
3.2 Monotone Systems 
A binary relation -g on R n is called a partial order if it satisfies that, for all x, y, z 6 
[i n , (i) x -g x; (ii) x -g y and y -g x imply x = y; and (iii) x -g y and y -g z 
imply x -g z. For a partial order -g on [i n , define << on [i n by x << y iffx < y 
and xi : yi for all i - 1,..-, n. A map F  [i n --+ [i n is monotone if x _ y implies 
Asynchronous Dynamics of Continuous Time Neural Networks 499 
F(x)  F(y). ACT dynamical system ofthe form (2) is monotone if xx -4_ x2 implies 
the trajectories x(t),x2(t) with z(0) = z and x2(0) = x2 satisfy x(t) _ x2(t) 
for all t _ 0 (Hirsch, 1988). 
Theorem 3 If the map F in (8) is monotone, then the system (8) is asynchroniz- 
able for uniformly-delayed asynchronous schemes, provided that all orbits x(t) have 
compact orbit closure and there is a to > 0 with x(to) >> x(O) or x(to) << x(O). 
Proof. This is an application of a Henry's theorem (see Hirsch, 1988) that im- 
plies that the asynchronized system (9) in the no communication delay situation 
is monotone and Hirsch's theorem (Hirsch, 1988) that guarantees the asymptotic 
convergence of monotone systems to fixed points. r 
Corollary 2 If the additive-type neural network (I) with sigmoidal activation func- 
tions is cooperative (/.e., wij _> 0 for i k j (Hirsch, 1988 and I989)), then it is 
asynchronizable for uniformly-delayed asynchronous schemes, provided that there is 
a to > 0 with x(to) >> x(O) or x(to) << x(O). 
Proof. According to (Hirsch, 1988), cooperative systems are monotone. As the 
network has only bounded dynamics, the result follows from the above theorem. [] 
4 CONCLUSION 
By incorporating the concepts of local times and communication times, we have 
provided a mathematical formulation of asynchronous dynamics of continuous-time 
dynamical systems. Asynchronized systems in the most general form haven't been 
studied in theories of dynamical systems and functional differential equations. For 
contractive and monotone systems, we have shown that for some asynchronous 
schemes, the systems are asynchronizable, namely, their asynchronizations preserve 
convergent dynamics of the original (synchronous) systems. When applying these 
results to the additive-type neural networks, we have obtained some special condi- 
tions for the networks to be asynchronizable. 
We are currently investigating more general results for asynchronizable dynamical 
systems, with a main interest in oscillatory dynamics. 
References 
G. M. Baudet (1978). Asynchronous iterative methods for multiprocessors. Journal 
of the Association for Computing Machinery, 25:226-244. 
J. Barhen and S. Gulati (1989). "Chaotic relaxation" in concurrently asynchronous 
neurodynamics. In Proceedings of International Conference on Neural Networks, 
volume I, pages 619-626, San Diego, California. 
Bertsekas and Tsitsiklis (1989). Parallel and Distributed Computation: Numerical 
Methods. Englewood Cliffs, N J: Prentice Hall. 
E. K. Blum (1990). Mathematical aspects of outer-product asynchronous content- 
addressable memories. Biological Cybernetics, 62:337-348, 1990. 
500 Wang, Li, and Blum 
E. K. Blum and X. Wang (1992). Stability of fixed-points and periodic orbits, and 
bifurcations in analog neural networks. Neural Networks, 5:577-587. 
J. Hale (1977). Theory of Functional Differential Equations. New York: Springer- 
Verlag. 
M. W. Hirsch (1988). Stability and convergence in strongly monotone dynamical 
systems. J. reine angew. Math., 383:1-53. 
M. W. Hirsch (1989). Convergent activation dynamics in continuous time networks. 
Neural Networks, 2:331-349. 
J. Hopfield (1982). Neural networks and physical systems with emergent computa- 
tional abilities. Proc. Nat. Acad. Sci. USA, 79:2554-2558. 
J. Hopfield (1984). Neurons with graded response have collective computational 
properties like those of two-state neurons. Proc. Nat. Acad. Sci. USA, 81:3088- 
3092. 
D. G. Kelly (1990). Stability in contractive nonlinear neural networks. IEEE Trans. 
Biomedi. Eng., 37:231-242. 
Q. Li (1993). Mathematical and Numerical Analysis of Biological Neural Networks. 
Unpublished Ph.D. Thesis, Mathematics Department, University of Southern Cali- 
fornia. 
C. M. Marcus and R. M. Westervelt (1989). Stability of analog neural networks 
with delay. Physical Review A, 39(1):347-359. 
K. Matsuoka (1992). Stability conditions for nonlinear continuous neural networks 
with asymmetric connection weights. Neural Networks, 5:495-500. 
J. M. Ortega and W. C. Rheinboldt (1970). Iterative solution of nonlinear equations 
in several variables. New York: Academic Press. 
X. Wang, E. K. Blum, and Q. Li (1993). Consistency on Local Dynamics and 
Bifurcation of Continuous-Time Dynamical Systems and Their Discretizations. To 
appear in the AMS proceedings of Symposia in Applied Mathematics, Mathematics 
of Computation 19d3 - 1993, Vancouver, BC, August, 1993, edited by W. Gautschi. 
X. Wang and E. K. Blum (1992). Discrete-time versus continuous-time neural 
networks. Computer and System Sciences, 49:1-17. 
X. Wang and D. S. Parker (1992). Computing least fixed points by asynchronous 
iterations and random iterations. Technical Report CSD-920025, Computer Science 
Department, UCLA. 
J.-H. Wu (1993). Delay-Induced Discrete Waves of Large Amplitudes in Neural Net- 
works with Circulant Connection Matrices. Preprint, Department of Mathematics 
and Statistics, York University. 
