Multiple Threshold Neural Logic 
Vasken Bohossian 
Jehoshua Bruck 
E-maih 
California Institute of Technology 
Mail Code 136-93 
Pasadena, CA 91125 
{vincent, bruck}paradise. caltech.edu 
Abstract 
We introduce a new Boolean computing element related to the Lin- 
ear Threshold element, which is the Boolean version of the neuron. 
Instead of the sign function, it computes an arbitrary (with poly- 
nomialy many transitions) Boolean function of the weighted sum of 
its inputs. We call the new computing element an LTM element, 
which stands for Linear Threshold with Multiple transitions. 
The paper consists of the following main contributions related to 
our study of LTM circuits: (i) the creation of efficient designs of 
LTM circuits for the addition of a multiple number of integers and 
the product of two integers. In particular, we show how to compute 
the addition of m integers with a single layer of LTM elements. 
(ii) a proof that the area of the VLSI layout is reduced from O(n 2) 
in LT circuits to O(n) in LTM circuits, for n inputs symmetric 
Boolean functions, and (iii) the characterization of the computing 
power of LTM relative to LT circuits. 
I Introduction 
Human brains are by far superior to computers in solving hard problems like combi- 
natorial optimization and image and speech recognition, although their basic build- 
ing blocks are several orders of magnitude slower. This observation has boosted 
interest in the field of artificial neural networks [Hop field 82], [Rumelhart 82]. The 
latter are built by interconnecting artificial neurons whose behavior is inspired by 
that of biological neurons. In this paper we consider the Boolean version of an artifi- 
cial neuron, namely, a Linear Threshold (LT) element, which computes a neural-like 
Multiple Threshold Neural Logic 253 
-w o t: t: 
1 t wD t 
LT gate SYM gate LTM gate 
Figure 1: Schematic representation of LT, $YM and LTM computing elements. 
Boolean function of n binary inputs [Muroga 71]. An LT element outputs the sign 
of a weighted sum of its Boolean inputs. The main issues in the study of networks 
(circuits) consisting of LT elements, called LT circuits, include the estimation of 
their computational capabilities and limitations and the comparison of their prop- 
erties with those of traditional Boolean logic circuits based on AND, OR and NOT 
gates (called AON circuits). For example, there is a strong evidence that LT cir- 
cuits are more efficient than AON circuits in implementing a number of important 
functions including the addition, product and division of integers [Sin 94], [Siu 93]. 
Motivated by our recent work on the VLSI implementation of LT elements 
[Bohossian 95b], we introduce in this paper a more powerful computing element, 
a multiple threshold neuron, which we call LTM, which stands for Linear Thresh- 
old with Multiple transitions, see [Haring 66] and [Olafsson 88]. Instead of the 
sign function in the LT element it computes an arbitrary (with polynomialy many 
transitions) Boolean function of the weighted sum of its inputs. 
The main issues in the study of LTM circuits (circuits consisting of LTM elements) 
include the estimation of their computational capabilities and limitations and the 
comparison of their properties to those of AON circuits. A natural approach in this 
study is first to understand the relation between LT circuits and LTM circuits. Our 
main contributions in this paper are: 
 We demonstrate the power of LTM by deriving efficient designs of LTM 
circuits for the addition of m integers and the product of two integers. 
We show that LTM circuits are more amenable in implementation than LT 
circuits. In particular, the area of the VLSI layout is reduced from O(n 2) 
in LT circuits to O(n) in LTM circuits, for n input symmetric Boolean 
functions. 
 We characterize the computing power of LTM relative to LT circuits. 
Next we describe the formal definitions of LT and LTM elements. 
1.1 Definitions and Examples 
Definition 1 (Linear Threshold Gate -LT) 
A linear threshold gate computes a Boolean function of its binary inputs: 
l(x) = san(wo + 
254 V. Bohossian and J. Bruck 
where the wi are integers and sgn(.) outputs I if its argument is greater or equal to 
O, and 0 otherwise. 
Figure I shows a n-input LT element; if  WiX i  --W 0 the element outputs 1, 
otherwise it outputs 0. A single LT gate is unable to compute parity. The latter 
belongs to the general class of symmetric functions - $YM. 
Definition 2 (Symmetric Functions - SYM) 
A Boolean function f is symmetric if its value depends only on the number of ones 
in the input denoted by IX I. 
Figure i shows an example of a symmetric function; it has three transitions, it 
outputs i for IXI < t and for t2 _< IXI < ta, and 0 otherwise. AND, OR and 
parity are examples of symmetric functions. A single LT element can implement 
only a limited subset of symmetric functions. We define LTM as a generalization 
of $YM. That is, we allow the weights to be arbitrary as in the case of LT, rather 
than fixed to 1 (see Figure 1 ). 
Definition 3 (Linear Threshold Gate with Multiple Transitions - LTM) 
A function f is in LTM if there exists a set of weights wi  Z, I <_ i _< n and a 
function h: Z  (0, 1) such that 
f(x) = for all X (0, 
il 
The only constraint on h is that it undergoes polynomialy many transitions as its 
input scans [- E=I I wi I, E. I I will' 
Notice that without the constraint on the number of transitions, an LTM gate is 
capable of computing any Boolean function. Indeed, given an arbitrary function f, 
let wi = 2 i-1 and h( 1 2i-lxi) = f(xl, ...,xn). 
Example I (XOR 6 LTM) 
XOR(X) outputs I if IXI, the number of 1's in X, is odd. Otherwise it outputs 
0. To implement it choose wi = i and h(k) = (1 - (-1) k) for 0 g k _< n. Note 
that h(k) needs not be defined for k < 0 and k > n, and has polynomialy many 
transitions. 
Another useful function that LTM can compute is ADD(X, Y), the sum of two 
n-bit integers X and Y. 
Example 2 (ADD  LTM) 
To implement addition we set ft(X,Y) = ht(i= 2i(xi + yi)) where h(k) = I for 
k  [2 , 2 x 2  - 1] U I3 x 2 t, +ee). Defined thus, f computes the m-th bit of X + Y. 
1.2 Organization 
The paper is organized as follows. In Section 2, we study a number of applications 
as well as the VLSI implementations of LTM circuits. In particular, we show how 
to compute the addition of m integers with a single layer of LTM elements. In 
Section 3, we prove t.he characterization results of LTM - inclusion relations, in 
particular LTM C_ LT2. In addition, we indicate which inclusions are proper and 
exhibit functions to demonstrate the separations. 
Multiple Threshold Neural Logic 
2 LTM Constructions 
255 
The theoretical results about LTM can be applied to the VLSI implementation of 
Boolean functions. The idea of a gate with multiple thresholds came to us as we 
were looking for an efficient VLSI implementation of symmetric Boolean functions. 
Even though a single LT gate is not powerful enough to implement any symmetric 
function, a 2-layer LT circuit is. Furthermore, it is well known that such a circuit 
performs much better than the traditional logic circuit based on AND, OR and 
NOT gates. The latter has exponential size (or unbounded depth) [Wegener 91]. 
Proposition 4 ( LT2 versus LTM for symmetric function implementation ) 
The LT2 layout of a symmetric function requires area of O(n2), while using LTM 
one needs only area of O(n). 
PROOF: 
Implementing a generalized symmetric function in LT2 requires up to n LT gates in 
the first layer. Those have the same weights wi except for the threshold w0. Instead 
of laying out n times the same linear sum  wixi we do it once and compare the 
result to n different thresholds. The resulting circuit corresponds to a single LTM 
gate. 12 
The LT2 layout is redundant, it has n copies of each weight, requiring area of at 
least O(n2). On the other hand, LTM performs a single weighted sum, its area 
requirement is O(n). 
A single LTM gate can compute the addition of m n-bit integers MADD. The 
only constraint is that m be polynomial in n. 
Theorem 5 (MADD  LTM) 
A single layer of LTM gates can compute the sum of m n-bit integers, provided 
that m is at most polynomial in n. 
PROOF: 
MADD returns an integer of at most n + log m bits. We need one LTM gate per 
bit. The least significant bit is computed by a simple m-bit XOR. For all other 
bits we use fl(X(), x(m)) h  2 i m X?)) 
", -- /(Ei=I Ej=I to compute the/-th bit of the 
sum. 12 
Corollary 6 (PRODUCT  PTM) A single layer of PTM ( which is defined 
below) gates, can compute the product of m n-bit integers, provided that m is at 
most polynomial in n. 
PROOF: 
By analogy with PT, defined in [Bruck 90], in PTM (or simply PTM) we allow a 
polynomial rather than a linear sum: f (X) -- h(w x + ... + wnxn + wo,2)x x2 + ...) 
However we restrict the sum to have polynomialy many terms (else, any Boolean 
function could be realized with a single gate). The product of two n-bit integers 
- i=xiY. We use the con- 
X and Y can be written as PRODUCT(X,Y) n 
struction of MADD in order to implement PRODUCT. PRODUCT(X, Y) -- 
MADD(xY, x2Y,...,xnY). f(X,Y) h n  
: /(Ej=I Ei=I 2ixjy i) f outputs the l-th 
bit of the product. [] 
256 V. Bohossian and J. Bruck 
ADD 
Figure 2: Relationship between Classes 
3 Classification of LTM 
We use a hat to indicate small (polynomialy growing) weights, e.g. ', L'M 
Bohossian 95a], [Siu 91], and a subscript to indicate the depth (number of layers) 
of the circuit of more than a single layer. All the circuits we consider in this paper 
are of polynomial size (number of elements) in n (number of inputs). For example, 
the class LT2 consists of those Boo.lean functions that can be implemented by a 
depth-2 polynomial size circuit of LT elements. 
Figure 2 depicts t.he membership relatio. between five classes of Boolean functions, 
including, LT, LT, LTM, LTM and LT2, along with the functions used to establish 
the separations. 
In this section we will prove the relations illustrated by Figure 2. 
Theorem 7 ( Classification of LTM ) 
The inclusions and separations shown in Figure 2 hold. That is, 
1. LT C LT C LTM 
2. LT c LTM c LTM 
3. LTM _c LT2 
J. XOR  LTM but XOR  LT 
5. COMP  LT but COMP  LTM 
6. ADD  LTM but ADD  LT 3 LT' 
7. IP  LT2 but IP  LTM 
PROOF: 
We show only the outline of the proof. The complete version can be found in 
[Bohossian 96]. Claims 1 and 2 follow from the definition. The first part of Claim 4 
was shown in Example i and the second is well known. In Claim 5, COMP stands 
for the Comparison function. the proof uses the pigeonhole principle and is related 
to the proof of COMP  LT which can be found in [Siu 91]. In Claim 6 to show 
that ADD  LTM we use the same idea as for COMP. Claim 3 is proved using a 
result from [Goldman 93]: a single LT gate with arbitrary weights can be realized 
by an LT2 circuit. Claim 7 introduces the function IPk(X,Y) - i iff E1 xiyi _ k, 
Multiple Threshold Neural Logic 257 
0 otherwise. If IPk  LTM, using the result from [Goldman 93], we can construct 
a LT2 circuit that computes IP2 (Inner Product mod 2) which is known to be false 
[Hajnal 94]. [] 
What remains to be shown in order to complete the classification picture is LT - 
LT  LTM. We conjecture that this is true. 
4 Conclusions 
Our original goal was to use theoretical results in order to efficiently lay out a 
generalized symmetric function. During that process we came to the conclusion 
that the LT2 implementation is partially redundant, which lead to the definition 
of LTM, a new, more powerful computing element. We characterized the power 
of LTM relative to LT. We showed how it can be used to reduce the area of 
VLSI layouts from O(n 2) to O(n) and derive efficient designs for multiple addition 
and product. Interesting directions for future investigation are (i) to prove the 
conjecture: LT - LT  LTM, (ii) to apply spectral techniques (ruck 90]) to the 
analysis of LTM, in particular show how PTM fits into the classification picture 
(Figure 2 ). 
Another direction for future research consists in introducing the ideas described 
above in the domain of VLSI. We have fabricated a programmable generalized 
symmetric function on a 2/, analog chip using the model described above. Floating 
gate technology is used to program the weights. We store a weight on a single 
transistor by injecting and tunneling electrons on the floating gate [Hasler 95]. 
Acknowledgments 
This work was supported in part by the NSF Young Investigator Award CCR- 
9457811 and by the Sloan Research Fellowship. 
References 
[Bohossian 95a] V. Bohossian and J. Bruck. On Neural Networks with Minimal 
Weights. In Advances in Neural Information Processing Systems 8, MIT Press, 
Cambridge, MA, 1996, pp.246-252. 
[Bohossian 95b] V. Bohossian, P. Hasler and J. Bruck. Programmable Neural Logic. 
Proceedings of the second annual IEEE International Conference on Innovative 
Systems in Silicon, pp. 13-21, October 1997. 
[Bohossian96] V. Bohossian and J. Bruck. 
ral Logic. Technical Report, ETR010, 
http://paradise.caltech.edu/ETR.html) 
Multiple Threshold Neu- 
June 1996. (available at 
[Bruck 90] J. Bruck. Harmonic Analysis of Polynomial Threshold Functions. SIAM 
J. Disc. Math, Vol. 3(No. 2)pp. 168-177, May 1990. 
[Goldman 93] M. Goldmann and M. Karpinski. Simulating threshold circuits by 
majority circuits. In Proc. 5th ACM STOC, pages pp. 551-560, 1993. 
258 V. Bohossian and J. Bruck 
IHajnal 94] A. Hajnal, W. Maass, P. Pudlak, M. Szegedy, G. Turan. Threshold 
Circuits of Bounded Depth. Journal of Computer and System Sciences, Vol. 
46(No. 2):pp. 129-154, April 1993. 
[Haring 66] D.R. Hating. Multi-Threshold Threshold Elements. IEEE Transactions 
on Electronic Computers, Vol. EC-15, No. 1, February 1966. 
[Hasler 95] P. Hasler, C. Diorio, B.A. Minch and C.A. Mead. Single Transistor 
Learning Synapses. Advances in Neural Information Processing Systems 7, 
MIT Press, Cambridge, MA, 1995, pp.817-824. 
[Hastad 94] J. Hastad. On the size of weights for threshold gates. SIAM. J. Disc. 
Math., 7:484-492, 1994. 
[Hofmeister 96] T. Hofmeister. A Note on the Simulation of Exponential Threshold 
Weights. 1996 CONCOON conference. 
[Hopfield 82] J. Hopfield. Neural networks and physical systems with emergent col- 
lective computational abilities. Proc. of the USA National Academy of Sciences, 
79:2554-2558, 1982. 
[Muroga 71] M. Muroga. Threshold Logic and its Applications. Wiley-Interscience, 
1971. 
[Olafsson 88] S. Olafsson and Y.S. Abu-Mostafa. The Capacity of Multilevel 
Threshold Functions. IEEE Transactions on Pattern Analysis and Machine 
Intelligence, Vol. 10, No. 2, March 1988. 
[Rumelhart 82] D. Rumelhart and J. McClelland. Parallel distributed processing: 
Explorations in the microstructure of cognition. MIT Press, 1982. 
[Siu 91] K. Siu and J. Bruck. On the power of threshold circuits with small weights. 
SIAM J. Disc. Math., Vol. 4(No. 3):pp. 423-435, August 1991. 
[Siu 
93] K. Siu, J. Bruck, T. Kailath, and T. Hofmeister. Depth Efficient Neural 
Networks for Division and Related Problems. IEEE Trans. on Information 
Theory, Vol. 39(No. 3), May 1993. 
[Siu 
94] K. Siu and V.P. Roychowdhury. On Optimal Depth Threshold Circuits for 
Multiplication and Related Problems. SIAM J. Disc. Math., Vol. 7(No. 2):pp. 
284-292, May 94. 
[Szegedy 89] M. Szegedy. Algebraic Methods in Lower Bounds for Computational 
Models with Limited Communication. PhD Thesis, Dep. Computer Science, 
Chicago Univ., December 1989. 
vVegener 91] I. Wegener. The complexity of the parity function in unbounded fan- 
in unbounded depth circuits. In Theoretical Computer Science, Vol. 85, pp. 
155-170, 1991. 
