Noisy Spiking Neurons with Temporal 
Coding have more Computational Power 
than $igmoidal Neurons 
Wolfgang Maass 
Institute for Theoreticell Computer Science 
Technische Universite[et Gre[z, Klosterwiesge[sse 32/2 
A-8010 Graz, Austrie[, e-me[il: maass@igi.tu-gre[z.e[c.e[t 
Abstract 
We exhibit e[ novel we[y of simule[ting sigmoide[l neure[l nets by net- 
works of noisy spiking neurons in tempotell coding. Furthermore 
it is shown theit networks of noisy spiking neurons with tempotell 
coding helve e[ strictly le[rger compute[tione[l power thein sigmoide[l 
neure[1 nets with the se[me number of units. 
1 Introduction and Definitions 
We consider e[ formal model SNN for e[ spiking n_euron network that is basically 
e[ reformule[tion of the spike response model (e[nd of the leaky integrate e[nd fire 
model) without using &functions (see [Maass, 1996e[] or [Me[ass, 1996b] for further 
be[ckground). 
An SNN consists of e[ finite set V of spiking neurons, e[ set E _C V x V of synapses, e[ 
weight Wu, _> 0 and e[ response function eu,  R + - R for each syne[pse (u, v)  E 
(where R + := {x e R: x _> 0}) , and a threshold function Ov' R + - R + for ee[ch 
neuron v  V . 
If Fu C_ R + is the set of firing times of a neuron u, then the potential at the trigger 
zone of neuron ve[t time t is given by 
In e[ noise-free model e[ neuron v fires e[t time t as soon as Pt) reaches Ot - t'), 
where t  is the time of the most recent firing of v . One says then that neuron v 
sends out e[n "action potential" or "spike" e[t time t. 
212 W. Maass 
For some specified subset n C_ V of input neurons one assumes that the firing times 
("spike trains") Fu for neurons u  n are not defined by the preceding convention, 
but are given from the outside. The firing times Fv for all other neurons v  V are 
determined by the previously described rule, and the output of the network is given 
in the form of the spike trains Fv for a specified set of output neurons Vout C_ V 
$ t 
v 
0 s  t 
t' t 
Figure 1: Typical shapes o,f response functions eu,v(EPSP and IPSP) and threshold 
functions 6).for biological neurons. 
We will assume in our subsequent constructions that all response functions eu,v 
and threshold fimctions Ovin an SNN are "stereotyped", i.e. that the response 
functions differ apart from their "sign" (EPSP or IPSP) only in their delay d,, 
(where du,v := inf {t _> 0: eu,v(t) y 0}) , and that the threshold functions 
only differ by an additive constant (i.e. for all u and v there exists a constant 
such that O,,(t) = O.t) + c,,,.for all t > O) . We refer to a term of the form 
Wu,v' eu,.(t- s) as an _excitatory respectively inhibitory __posts_s_snaptic potential 
(abbreviated: EPSP respectively IPSP). 
Since biological netirons do not always fire in a reliable manner one also considers 
the related model of noisy spiking neurons, where P.(t) is replaced by P isy(t) := 
Pv(t) + Cry(t) and Ov(t t') is replaced by 0 noisy Ov(t t') +/3v(t t'). 
-  (t-t') := - - 
av(t) and /3.t - t') are allowed to be arbitrary functions with bounded absolute 
value (hence they can also represent "systematic noise"). 
Furthermore one allows that the current value of the difference D(t) := Pisy(t) - 
noisy 
O, (t - t') does not determine directly the firing time of neuron v, but only its 
current firing probability. We assume that the firing probability approaches 1 if 
D --,  , and 0 if D --, - . We refer to spiking neurons with these two types of 
noise as "noisy spiking neurons". 
We will explore in this article the power of analog computations with noisy spiking 
neurons, and we refer to [Magss, 1996a] for restilts about digital computations in 
this model. Details to the results in this article appear in [Maass, 1996b] and 
[Maass, 1997]. 
2 
Fast Simulation of Sigmoidal Neural Nets with Noisy 
Spiking Neurons in Temporal Coding 
So far one has only considered simulations of sigmoidal neural nets by spiking neu- 
rons where each analog variable in the sigmoidal neural net is represented by the 
firing rate of a spiking neuron. However this "firing rate interpretation" is incon- 
sistent with a number of empirical results about computations in biological neural 
Spiking Neurons have more Computational Power than Sigmoidal Neurons 213 
systems. For example [Thorpe & Imbert, 1989] have demonstrated that visual pat- 
tern analysis and pattern classification can be carried out by humans in just 150 ms, 
in spite of the fact that it involves a minimum of l0 synaptic stages from the retina 
to the temporal lobe. [de Ruyter van Steveninck & Bialek, 1988] have found that 
a blowfly can produce flight torques within 30 ms of a visual stimulus by a neural 
system with several synaptic stages. However the firing rates of neurons involved 
in all these computations are usually below 100 Hz, and interspike intervals tend 
to be quite irregular. Hence one cannot interpret these analog computations with 
spiking neurons on the basis of an encoding of analog variables by firing rates. 
On the other hand experimental evidence has accumulated during the last few years 
which indicates that many biological neural systems use the timing of action poten- 
tials to encode information (see e.g. [Bialek & Rieke, 1992], [Bait & Koch, 1996]). 
We will now describe a new way of simulating sigmoidal neural nets by networks of 
spiking neurons that is based on temporal coding. The key mechanism for this alter- 
native simulation is based on the well known fact that EPSP's and IPSP's are able 
to shift the firing time of a spiking neuron. This mechanism can be demonstrated 
very clearly in our formal model if one assumes that EPSP's rise (and IPSP's fall) 
linearly during a certain initial time period. Hence we assume in the following that 
there exists some constant A > 0 such that each response function e,,v (x) is of the 
form au,v '(x-du,v) with au,v e {-1, 1} for xe [du,v, du,v + A] ,and eu,v(X)-0 
for x e [0, du,v] . 
Consider a spiking netiron v that receives postsynaptic potentials from n presynaptic 
neurons a,..., an  For simplicity we asslime that interspike intervals are so large 
that the firing time tv of neuron v depends just on a single firing time ta of each 
neuron ai, and Ov has returned to its "resting value" O(0) before v fires again. 
Then if the next firing of v occurs at a time when the postsynaptic potentials 
described by wa,v . ea,v (t - ta) are all in their initial linear phase, its firing time 
tv is determined in the noise-free model for wi := wa,v  ca,v by the equation 
'-i"--- wi. (t - ta - d,,) = O(0) , or equivalently 
o(o) O' + 
n q- n (1) 
tv - Ei=i wi Ei=i wi 
This equation reveals the somewhat surprising fact that (for a certain range of 
their parameters) spiking netirons can compute a weighted sum in terms of firing 
times, i.e. temporal coding. One should also note that in the cause where all delays 
da,v have the same value, the "weights" wi of this weighted sum are encoded in 
the "strengths" wa,,v of the synapses and their "sign" ca,v , as in the "firing rate 
interpretation". Finally according to (1) the coefficients of the presynaptic firing 
times ta, are automatically normalized, which appears to be of biological interest. 
In the simplest scheme for temporal coding (which is closely related to that in 
[Hopfield, 1995]) an analog variable x  [0, 1] is encoded by the firing time T - -/. x 
of a neuron, where T is assumed to be independent of x (in a biological context T 
might be time-locked to the onset of a stimulus, or to some oscillation) and -), is 
some constant that is determined in the proof of Theorem 2.1 (e.g. -), = A/2 in the 
noise-free case). In contrast to [Hopfield, 1995] we asslime that both the inputs and 
the outputs of computations are encoded in this fashion. This has the advantage 
that one can compose computational modules. 
214 W. Maass 
We will first focus in Theorem 2.1 on the simulation of sigmoidal neural nets that 
employ the piecewise linear "linear saturated" activation function -: R -- [0, 1] 
defined by r(y) = 0 ify < 0, r(y) = y if0 <_ y _< 1, andr(y) -- 1 ify > 1 
The Theorem 3.1 in the next section will imply that one can simulate with spik- 
ing neurons also sigmoidal neural nets that employ arbitrary continuous activation 
functions. Apart from the previously mentioned assumptions we will assume for 
the proofs of Theorem 2.1 and 3.1 that any EPSP satisfies eu,vx) = 0 for all suffi- 
ciently large x, and eu, (x) _> eu, (du, + A) for all x e [du, + A, du, + A + ?] . 
We assume that each IPSP is continuous, and has value 0 except for some interval 
of R . Furthermore we assume for each EPSP and IPSP that ]eu,vx)l grows at 
most linearly during the interval [du,v+ A, du,v+ A + ?] . In addition we assume 
that 13vx) = 130) for sufficiently large x , and that 13v(x) is sufficiently large for 
0<x<_. 
Theorem 2.1 For any given e, 5 > 0 one can simulate any given feedforward sig- 
moidal neural net N with activation function r by a network AfN,e, of noisy spiking 
neurons in temporal coding. More precisely, for any network input x,...,xm  
[0, 1] the output of AfN, e, differs with probability _> 1 - 5 by at most e from that 
of N . Furthermore the computation time of AfN,e,, depends neither on the number 
of gates in N nor on the parameters e, 5, but only on the number of layers of the 
sigmoidal neural network N . 
We refer to [Maass, 1997] for details of the somewhat complicated proof. One 
employs the mechanism described by (1) to simulate through the firing time of a 
spiking neuron v a sigmoidal gate with activation function 7r for those gate-inputs 
where - operates in its linearly rising range. With the help of an auxiliary spiking 
neuron that fires at time T one can avoid the automatic "normalization" of the 
weights wi that is provided by (1), and thereby compute a weighted sum with 
arbitrary given weights. In order to simulate in temporal coding the behaviour of 
the gate in the input range where - is "saturated" (i.e. constant), it suffices to 
employ some auxiliary spiking neurons which make sure that v fires exactly once 
during the relevant time window (and not shortly before that). 
Since inputs and outputs of the resulting modules for each single gate of N are all 
given in temporal coding, one can compose these modules to simulate the multi- 
layer sigmoidal neural net N. With a bit of additional work one can ensure that 
this construction also works with noisy spiking neurons.  
3 
Universal Approximation Property of Networks of Noisy 
Spiking Neurons with Temporal Coding 
It is known [Leshno et al., 1993] that feedforward sigmoidal neural nets whose gates 
employ the activation function ' can approximate with a single hidden lyer for 
any n, k  N any given continuous function F: [0, 1] n -- [0, 1] k within any e ) 0 
with regard to the Leo-norm (i.e. uniform convergence). Hence we can derive the 
following result from Theorem 2.1: 
Theorem 3.1 Any given continuous function F'[0, 1] n - [0, 1] k can be approxi- 
mated within any given e > 0 with arbitrarily high reliability in temporal coding by 
Spiking Neurons have more Computational Power than Sigmoidal Neurons 215 
a network of noisy spiking neurons (SNN) with a single hidden layer (and hence 
within 15 ms for biologically realistic values of their time-constants).  
Because of its generality this Theorem implies the same result also for more general 
schemes of coding analog variables by the firing times of neurons, besides the par- 
ticular one that we have considered so far. In fact it implies that the same result 
holds for any other coding scheme C that is "continuously related" to the previ- 
ously considered one in the sense that the transformation between firing times that 
encode an analog variable x in the here considered coding scheme and in the coding 
scheme C can be described by uniformly continuous functions in both directions. 
4 
Spiking Neurons have more Computational Power than 
Sigmoidal Neurons 
We consider the "element distinctness function" EDn ' (R+) n '-} {0, 1} defined 
by 
EDn(Sl, . . . ,Sn) 
1, 
arbitrary, 
ifsi=sj for someij 
if Isi - sj] _> 1 for all i,j with i  j 
else . 
If one encodes the value of input variable si by a firing of input neuron ai at time 
Tin - c' si , then for sufficiently large values of the constant c > 0 a single noisy 
spiking neuron v can compute EDn with arbitrarily high reliability. This holds for 
any reasonable type of response fimctions, e.g. the ones shown in Fig. 1. The binary 
output of this computation is assumed to be encoded by the firing/non-firing of v. 
Hair-trigger situations are avoided since no assumptions have to be made about the 
firing or non-firing of v if EPSP's arrive with a temporal distance between 0 and c. 
On the other hand the following result shows that a fairly large sigmoidal neural 
net is needed to compute the same function. Its proof provides the first application 
for Sontag's recent results about a new type of "dimension" d of a neural network 
N, where d is chosen maximal so that every subset of d inputs is shattered by N. 
Furthermore it expands a method due to [Koiran, 1995] for using the VC-dimension 
to prove lower bounds on network size. 
Theorem 4.1 Any sigmoidal neural net Af that computes EDn has at least n-4 1 
hidden units. 
Prooff Let Af be an arbitrary sigmoidal neural net with k gates that computes 
EDn. Consider any set $ c_ R + of size n - 1. Let , > 0 be sufficiently large so that 
the numbers in A. $ have pairwise distance _ 2 . Let A be a set of n - 1 numbers 
> max (A  $) + 2 with pairwise distance _ 2 . 
By assumption Af can decide for n arbitrary inputs from ,. $ U A whether they 
are all different. Let Afx be a variation of Af where all weights on edges from the 
first input variable are multiplied with ,. Then Afx can compute any function from 
216 W. Maass 
$ into (0, 1) after one has assigned a suitable fixed set of n - 1 pairwise different 
numbers from A. $ tJ A to the last n - 1 input variables. 
Thus if one considers as programmable parameters of.hf the factor A in the weights 
on edges from the first input variable and the _ k thresholds of gates that are 
connected to some of the other n - 1 input variables, then .hf shatters $ with these 
k + 1 programmable parameters. 
Since S _c R + of size n- 1 was chosen arbitrarily, we can now apply the result from 
[Sontag, 1996], which yields an lipper bound of 2w + 1 for the maximal number d 
such that every set of d different inputs can be shattered by a sigmoidal neural net 
with w programmable parameters (note that this parameter d is in general much 
smaller than the VC-dimension of the neural net). For w := k + 1 this implies in 
our case that n-1 _ 2(k+l)+l , hencek_ (n-4)/2. ThusAf has at least 
(n - 4)/2 computation nodes, and therefore at least (n - 4)/2 - 1 hidden units. One 
should point out that due to the generality of Sontag's result this lower bound is 
valid for all common activation functions of sigmoidal gates, and even if A/' employs 
heaviside gates besides sigmoidal gates.  
Theorem 4.1 yields a lower bound of 4997 for the number of hidden units in any 
sigmoidal neural net that computes EDn for n = l0 000, where l0 000 is a common 
estimate for the number of inputs (i.e. synapses) of a biological neuron. 
Finally we would like to point out that to the best of our knowledge Theorem 4.1 
provides the largest known lower bound for any concrete function with n inputs on 
a sigmoidal neural net. The largest previously known lower bound for sigmoidal 
neural nets was f(n 1/4) , due to [Koiran, 1995]. 
5 Conclusions 
Theorems 2.1 and 3.1 provide a model for analog computations in network of spiking 
neurons that is consistent with experimental results on the maximal computation 
speed of biological neural systems. As explained after Theorem 3.1, this result holds 
for a large variety of possible schemes for encoding analog variables by firing times. 
These theoretical results hold rigorously only for a rather small time window of 
length ? for temporal coding. However a closer inspection of the construction 
shows that the actual shape of EPSP's and IPSP's in biological neurons provides 
an automatic adjustment of extreme values of the inputs ta towards their average, 
which allows them to carry out rather similar computations for a substantially larger 
window size. It also appears to be of interest from the biological point of view that 
the synaptic weights play for temporal coding in our construction basically the same 
role as for rate coding, and hence the same network is in principle able to compute 
closely related analog functions in both coding schemes. 
We have focused in our constructions on feedforward nets, but our method can for 
example also be used to simulate a Hopfield net with graded response by a network 
of noisy spiking neurons in temporal coding. A stable state of the Hopfield net 
corresponds then to a firing pattern of the simulating SNN where all neurons fire 
at the same frequency, with the "pattern" of the stable state encoded in their phase 
differences. 
Spiking Neurons have more Computational Power than Sigmoidal Neurons 217 
The theoretical results in this article may also provide additional goals and direc- 
tions for a new computer technology based on artificial spiking neurons. 
Acknowledgement 
I would like to thank David Haussler, Pascal Koiran, and Eduardo Sontag for helpful 
communications. 
References 
[Bair & Koch, 1996] W. Bair, C. Koch, "Temporal precision of spike trains in 
extrastriate cortex of the behaving macaque monkey", Neural Computation, 
vol. 8, pp 1185-1202, 1996. 
[Bialek & Rieke, 1992] W. Bialek, and F. Rieke, "Reliability and information trans- 
mission in spiking neurons", Trends in Neuroscience, vol. 15, pp 428-434, 1992. 
[Hopfield, 1995] J. J. Hopfield, "Pattern recognition computation using action po- 
tential timing for stimulus representations", Nature, vol. 376, pp 33-36, 1995. 
[Koiran, 1995] P. Koiran, "VC-dimension in circuit complexity", Proc. of the 11th 
IEEE Conference on Computational Complexity, pp 81-85, 1996. 
[Leshno et al., 1993] M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, "Multilayer 
feedforward networks with a nonpolynomial activation function can approxi- 
mate any function", Neural Networks, vol. 6, pp 861-867, 1993. 
[Maass, 1996a] W. Maass, "On the computational power of noisy spiking neurons", 
Advances in Neural Information Processing Systems, vol. 8, pp 211-217, MIT 
Press, Cambridge, 1996. 
[Maass, 1996b] W. Maass, "Networks of spiking neurons: the third generation of 
neural network models", FTP-host: archive.cis.ohio-state.edu, FTP-filename: 
/pu b/neu roprose/maass.t hird-generation.ps. Z, Neural Networks, to appear. 
[Mass, 1997] W. Maass, "Fast sigmoidal networks via spiking neurons", to appear 
in Neural Computation. FTP-host: archive.cis.ohio-state.edu FTP-filename: 
/pub/neuroprose/maass.sigmoidal-spiking.ps.Z, Neural Computation, to ap- 
pear in vol. 9, 1997. 
[de Ruyter van Steveninck & Bialek, 1988] R. de Ruyter van Steveninck, and 
W. Bialek, "Real-time performance of a movement sensitive neuron in the 
blowfly visual system", Proc. Roy. $oc. B, vol. 234, pp 379-414, 1988. 
[Sontag, 1996] E. D. Sontag, "Shttering all sets of k points in 'general position' re- 
quires (k- 1)/2 parameters", http://www.math.rutgers.edu/sontag/, follow 
links to FTP archive. 
[Thorpe & Imbert, 1989] S. T. Thorpe, and M. Imbert, "Biological constraints 
on connectionist modelling", In: Connectionism in Perspective, R. Pfeifer, 
Z. Schreter, F. Fogelman-Souli., and L. Steels, eds., Elsevier, North-Holland, 
1989. 
