WATTLE: A Trainable Gain Analogue 
VLSI Neural Network 
Richard Coggins and Marwan Jabri 
Systems Engineering and Design Automation Laboratory 
Department of Electrical Engineering J03, 
University of Sydney, 2006. 
Australia. 
Email: richardc@sedal.su.oz.au 
marwan@sedal.su.oz.au 
Abstract 
This paper describes a low power analogue VLSI neural network 
called Wattle. Wattle is a 10:6:4 three layer perceptron with multi- 
plying DAC synapses and on chip switched capacitor neurons fabri- 
cated in 1.2um CMOS. The on chip neurons facillitate variable gain 
per neuron and lower energy/connection than for previous designs. 
The intended application of this chip is Intra Cardiac Electrogram 
classification as part of an implantable pacemaker/defibrillator sys- 
tem. Measurements of the chip indicate that 10pJ per connection 
is achievable as part of an integrated system. Wattle has been suc- 
cessfully trained in loop on parity 4 and ICEG morphology classi- 
fication problems. 
I INTRODUCTION 
A three layer analogue VLSI perceptron has been previously developed by 
[Leong and Jabri, 1993]. This chip named Kakadu uses 6 bit digital weight storage, 
multiplying DACs in the synapses and fixed value off chip resistive neurons. The 
chip described in this paper called Wattle has the same synapse arrays as Kakadu, 
however, has the neurons implemented as switched capacitors on chip. For both 
Kakadu and Wattle, analogue techniques have been favoured as they offer greater 
opportunity to achieve a low energy and small area design over standard digital 
874 
WATTLE: A Trainable Gain Analogue VLSI Neural Network 875 
Iout+ md font- opt <   
rn oth sps  ...... , 
oonnt h 
w,aoi-rr sTooE v+ tl- 
' WR 
Figure 1' Wattle Synapse Circuit Diagram 
SYNAPSE R CUlT 
techniques since the transistor count for the synapse can be much lower and the 
circuits may be biased in subthreshold. Some work has been done in the low en- 
ergy digital area using subthreshold and optimised threshold techniques, however 
no large scale circuits have been reported so far. [Burr and Peterson, 1991] The cost 
of using analogue techniques is however, increased design complexity, sensitivity to 
noise, offsets and component tolerances. In this paper we demonstrate that difficult 
nonlinear problems and real world problems can be trained despite these effects. 
At present, commercially available pacemakers and defibrillators use timing deci- 
sion trees implemented on CMOS microprocessors for cardiac arrythmia detection 
via peak detection on a single ventricular lead. Even when atrial leads are used, In- 
tra Cardiac Electrogram (ICEG) morphology classification is required to separate 
some potentially fatal rhythms from harmless ones. [Leong and Jabri, 1992] The 
requirements of such a morphology classifier are: 
 Adaptable to differing morphology within and across patients. 
 Very low power consumption. ie. minimum energy used per classification. 
 Small size and high reliability. 
This paper demonstrates how this morphology classification may be done using a 
neural network architecture and thereby meet the constraints of the implantable 
arrythmia classification system. In addition, in loop training results will also be 
given for parity 4, another difficult nonlinear training problem. 
876 Coggins and Jabri 
reset 
clock 
Vdd 
ctk 
charging 
clock 
synapse 
row 
connects 
cop 
COM 
fan out to 
F=i next laye[ 
Figure 2: Wattle Neuron Circuit Diagram 
[ Row Address ] 
Column 
Address 
ur 
10x6 Synapse Array 
neurons ] 
[ hi mutipbxor I [nclk demux ] 
IJ  
$X4 Synapse   
Array [--"] [ 
,.uron. I--I I--1 
Data Register I Buffers 
Figure 3: Wattle Floor Plan 
WATI'LE: A Trainable Gain Analogue VLSI Neural Network 877 
l, 
'L 
Figure 4: Photomicrograph of Wattle 
2 ARCHITECTURE 
Switched capacitors were chosen for the neurons on Wattle after a test chip was fab- 
ricated to evaluate three neuron designs. [Coggins and Jabri, 1993] The switched 
capacitor design was chosen as it allowed flexible gain control of each neuron, in- 
vestigation of gain optimisation during limited precision in loop training and the 
realisation of very high effective resistances. The wide gain range of the switched 
capacitor neurons and the fact that they are implemented on chip has allowed Wat- 
tle to operate over a very wide range of bias currents from lpA LSB DAC current 
to 10nA LSB DAC current. 
Signalling on Wattle is fully differential to reduce the effect of common mode noise. 
The synapse is a multiplying digital to analogue convertor with six bit weights. The 
synapse is shown in figure 1. This is identical to the synapse used on the Kakadu 
chip [Leong and Jabri, 1993]. The MDAC synapses use a weighted current source 
to generate the current references for the weights. The neuron circuit is shown in 
figure 2. The neuron requires reset and charging clocks. The period of the charging 
clock determines the gain. Buffers are used to drive the neuron outputs off chip to 
avoid the effects of stray pad capacitances. 
Figure 3 shows a floor plan of the wattle chip. The address and data for the weights 
access is serial and is implemented by the shift registers on the boundary of the chip. 
The hidden layer multiplexor allows access to the hidden layer neuron outputs. The 
neuron demultiplexor switches the neuron clocks between the hidden and output 
layers. Figure 4 shows a photomicrograph of the wattle die. 
3 ELECTRICAL CHARACTERISTICS 
Tests have been performed to verify the operation of the weighted current source 
for the MDAC synapse arrays, the synapses, the neurons and the buffers driving the 
neuron voltages off chip. The influences of noise, offsets, crosstalk and bandwidth 
of these different elements have been measured. In particular, the system level noise 
measurement showed that the signal to noise ratio was 40dB. A summary of the 
electrical characteristics appears in table 1. 
878 Coggins and Jabri 
Table 1: Electrical Characteristics and Specifications 
Parameter 
Area 
Technology 
Resolution 
Energy per connection 
LSB DAC current 
Feedforward delay 
Synapse Offset 
Gain cross talk delta 
Value 
2.2 x 2.2ram 2 
1.2urn Nwell CMOS 2M2P 
weights 6bit, gains 7bit 
43pJ 
200pA 
1.5ms 
5mV 
20% 
Comment 
standard process 
weights on chip, gains off 
all weights maximum 
typical 
@200pA, 3V supply 
typical maximum 
maximum 
A gain cross talk effect between the neurons was discovered during the electrical 
testing. The mechanism for this cross talk was found to be transients induced on the 
current source reference lines going to all the synapses as individual neuron gains 
timed out. The worst case cross talk coupled to a hidden layer neuron was found 
to be a 20% deviation from the singularly activated value. However, the training 
results of the chip do not appear to suffer significantly from this effect. 
A related effect is the length of time for the precharging of the current summation 
lines feeding each neuron due to the same transients being coupled onto the current 
source when each neuron is active. The implication of this is an increase in energy 
per classification for the network due to the transient decay time. However, one of 
the current reference lines was available on an outside pin, so the operation of the 
network free of these transients could also be measured. For this design, including 
the transient conditions, an energy per connection of 43pJ can be achieved. This 
may be reduced to 10pJ by modifying the current source to reduce transients and 
neglecting the energy of the buffers. This is to be compared with typical digital 
10nJ per connection and analogue of 60pJ per connection appearing in the literature. 
[Delcorso et. al., 1993], Table 1. 
4 TRAINING BOTH GAINS AND WEIGHTS 
A diagram of the system used to train the chip is shown in figure 5. The training 
software is part of a package called MUME [Jabri et. al., 1992], which is a multi 
module neural network simulation environment. Wattle is interfaced to the work 
station by Jiggle, a general purpose analogue and digital chip tester developed by 
SEDAL. Wattle, along with gain counter circuitry, is mounted on a separate daugh- 
ter board which plugs into Jiggle. This provides a software configurable testing 
environment for Wattle. In loop training then proceeds via a hardware specific 
module in MUME which writes the weights and reads back the analogue output of 
the chip. Wattle can then be trained by a wide variety of algorithms available in 
MUME. 
Wattle has been trained in loop using a variation on the Combined Search Algorithm 
(CSA) for limited precision training. [Xie and Jabri, 1992] (Combination of weight 
perturbation and axial random search). The variation consists of training the gains 
WATI'LE: A Trainable Gain Analogue VLSI Neural Network 879 
SUN IPC 
Running 
MUME 
(CSA) 
SBUS to VME 
Convertor 
Figure 5: The in loop training of Wattle 
in exactly the same way that the weights are trained rather than scaling the gains 
based on the average value of the weights connected to a neuron. In this case, the 
gains are 7 bits and are implemented off chip. The in loop training results of the 
chip are compared with that of a numerical model. The numerical model used does 
not model noise, cross talk, offsets or common mode saturation and in this sense is 
an idealised model of the chip. A comparison is done between the training of each 
neuron gain versus leaving the neuron gain fixed. 
5 
TRAINING AND GENERALISATION 
PERFORMANCE 
The parity 4 problem converged 8 out of 10 times training the gains within a max- 
imum of 300 iterations of CSA. Leaving the gains fixed, convergence was obtained 
3 out 6 times within 500 iterations. On the numerical model, parity 4 did not 
converge within 300 iterations. This suggests that the influence of noise may have 
assisted the training as in annealing based algorithms. In all cases a 5:5:1 network 
was used where the 5th input is a bias. 
The ICEG training problem consisted of separating the ICEG morphology of pa- 
tients with Retrograde Ventricular Tachycardia. The morphologies to be separated 
were Normal Sinus Rhythm and Ventricular Tachycardia on an individual patient 
basis. In this case, 10 morphology samples are used as inputs to 10:H:2 network 
where H was varied from 1 to 6. The chip trained a patient using trainable gains. 
The 8 training patterns were selected and the network trained twice on each archi- 
tecture. 
Table 2 summarises the classification performance on data not previously seen by 
880 Coggins and Jabri 
4.4O 
40 
4.00 
:].so 
3.60 
3,40 
3.20 
3.00 
2.80 
2.60. 
2.4o 
2.20 
2.oo 
1.8o 
1.6o 
1.4o 
1.2o 
r,t o 
0.40 
o..20 
oo 
Tralntng_51d_units 
t.I 
Figure 6: An example training run of Wattle 
the network for simulated and in loop chip training. Figure 6 shows an example 
of the CSA algorithm training the chip in loop. The CSA algorithm should always 
reduce the error, however the effect of the noise in the hardware is evident once the 
training reaches smaller error values. 
6 CONCLUSION 
The results from the Wattle chip have shown the combination of switched capacitor 
and analogue design techniques have produced a very low power trainable neural 
network for the ICEG classifica[ion problem. When used with the CSA training 
algorithm the parity 4 problem is trained routinely. Work is continuing to assess 
trainable gain requirements and advantages for different problems. With some 
design modification 10pJ per connection energy useage is achievable. 
Acknowledgement s 
The work presented in this paper was jointly funded by: 
 The Department of Industry, Technology and Regional Development. 
 Telectronics Pacing Systems Ltd., Australia. 
The authors also acknowledge the contribution of Philip Leong, Barry Flower and 
Steve Pickard for their help in laying out and testing wattle and Jeevan Gagarin for 
his assistance with the test jig hardware. 
WATTLE: A Trainable Gain Analogue VLSI Neural Network 881 
Table 2: ICEG Generalisation Performance 
No. Of Hidden Testing Patterns Correct % 
Units 
Sim. Chip. 
NSR VT NSR VT 
1 24 100 91 99 
24 100 90 98 
2 94 99 97 93 
94 99 96 93 
3 97 87 100 84 
97 87 100 82 
4 96 97 90 97 
96 97 91 99 
5 100 99 100 97 
100 99 97 99 
6 94 93 94 95 
94 93 96 97 
References 
[Leong and Jabri, 1993] Leong, P.H.W. and Jabri, M.A., A Low Power Analogue 
Neural Network Classifier Chip, Proceedings of the IEEE Custom Integrated 
Circuits Conference, p4.5.1-4.5.4, San Diego, USA, May 1993. 
[Leorig and Jabri, 1992] Leong, P.It.W. and Jabri, M.A., MATIC - An Intracardiac 
Tachycardia Classification System, PACE, September, 1992. 
[Burr and Peterson, 1991] Burr, J. and Peterson A., Ultra Low Power CMOS Tech- 
nology, 3rd NASA Symposium on VLSI Design, 1991. 
[Coggins and Jabri, 1993] R.J. Coggins, M.A. Jabri, S.J. Pickard, A Comparison 
of Three On Chip Neuron Designs For a Low Power Analogue VLSI MLP, Mi- 
cronemro '93, Edinburgh, Scotland, UK., April, 1993. 
[Delcorso et. al., 1993] D. Del Corso, F. Gregoretti, L.M. Reyneri, An Artificial 
Neural System Using Coherent Pulse Width and Edge Modulations, Micronemro 
'93, Edinburgh, Scotland, UK., April, 1993. 
[Xie and Jabri, 1992] Xie, Yun and Jabri, Marwan, Training Limited Precision 
Feedforward Neural Networks, ACNN92, p68-71, Canberra, Australia, February 
1992. 
[Tsividis ,1989] Yannis P. Tsividis, "Operation and Modelling of the MOS Transis- 
tor", McGraw-Hill, 1988. 
[Jabri et. al., 1992] Jabri M., Tinker E., Leerink L. (1992)"MUME- A Multi-NET- 
Multi- Architecture Neural Simulation Environment". Neural Network Simula- 
tion Environments, Kluwer Academic Publications. 
