Analog Circuits for Constrained Optimization 777 
Analog Circuits for Constrained Optimization 
John C. Platt x 
Computer Science Department, 256-80 
California Institute of Technology 
Pasadena, CA 91125 
ABSTRACT 
This paper explores whether analog circuitry can adequately per- 
form constrained optimization. Constrained optimization circuits 
are designed using the differential multiplier method. These cir- 
cuits fulfill time-varying constraints correctly. Example circuits in- 
clude a quadratic programming circuit and a constrained flip-flop. 
1 INTRODUCTION 
Converting perceptual and cognitive tasks into constrained optimization problems 
is a useful way of generating neural networks to solve those tasks. Researchers have 
used constrained optimization networks to solve the traveling salesman problem 
[Durbin, 1987] [Hopfield, 1985], to perform object recognition [Gindi, 1988], and to 
decode error-correcting codes [Platt, 1986]. 
Implementing constrained optimization in analog VLSI is advantageous, because an 
analog VLSI chip can solve a large number of differential equations in parallel [Mead, 
1989]. However, analog circuits only approximate the desired differential equations. 
Therefore, we have built test circuits to determine whether analog circuits can fulfill 
user-specified constraints. 
2 THE DIFFERENTIAL MULTIPLIER METHOD 
The differential multiplier method (DMM) is a method for creating differential equa- 
tions that perform constrained optimization. The DMM was originally proposed 
by [Arrow, 1958] as an economic model. It was used as a neural network by [Platt, 
1987]. 
1 Current address: Synaptics, 2860 Zanker Road, Suite 105, San Jose, CA 95134 
778 Platt 
Figure 1. The architecture of the DMM. The z capacitor in the figure repre- 
sents the zi neurons in the network. The -f box computes the current needed for 
the neurons to minimize f. The rest of the circuitry causes the network to fulfill 
the constraint g(M) = 0. 
G3 
Figure 2. A circuit that implements quadratic programming. z, y, and A are 
voltages. "TC" refers to a transconductance amplifier. 
Analog Circuits for Constrained Optimization 779 
A constrained optimization problem is find a  such that f() is minimized subject 
to a constraint g(') - 0. In order to find a constrained minimum, the DMM finds 
the critical points (', ) of the Lagrangian 
 = f(')+ Ag(), 
(1) 
by performing gradient descent on the variables g' and gradient ascent on the La- 
grange multiplier : 
d x i _ _ O_..E _ Of A Og 
dt Oxi Oxi Oxi 
d, 0 
= + = g(). 
dt Oh 
(2) 
The DMM can be thought of as a neural network which performs gradient descent 
on a function f('), plus feedback circuitry to find the  that causes the neural 
network output to fulfill the constraint g(g') = 0 (see figure 1). 
The gradient ascent on the  is necessary for stability. The stability can be exam- 
ined by combining the two equations (2) to yield a set of second-order differential 
equations 
d2xi ( O2 f O: g ) dxj Og 
+ + + = o 
 . OxOx '- gox ' 
which is analogous to the equations that govern a spring-mass-damping system. 
The differential equations (3) converge to the constrained minima if the damping 
matrix 
O2f O2g 
M= + (4) 
OxiOx OxiOx 
is positive definite. 
The DMM can be extended to satisfy multiple simultaneous constraints. The sta- 
bility of the DMM can also be improved. See [Platt, 1987] for more details. 
3 QUADRATIC PROGRAMMING CIRCUIT 
This section describes a circuit that solves a specific quadratic programming prob- 
lem for two variables. A quadratic programming circuit is interesting, because the 
basic differential multiplier method is guaranteed to find the constrained minimum. 
Also, quadratic programming is useful: it is frequently a sub-problem in a more 
complex task. A method of solving general nonlinear constrained optimization is 
sequential quadratic programming [Gill, 1981]. 
We build a circuit to solve a time-dependent quadratic programming problem for 
two variables: 
min A(x - x0? + B(y- Yo?, (5) 
subject to the constraint 
+ Oy + o. (6) 
780 Platt 
Constraint Fulfillment for Quadratic Programming 
0.2 
observed, target (V)0.0 
-0.2 
0.0 
I 
0.8 1.2 1.6 2.0 
Time (10 -2 Sec) 
Figure 3. Plot of two input voltages of transconductance amplifier. The 
dashed line is the externally applied voltage E(t). The solid line is the circuit's 
solution of-Cx- Dy. The constraint depends on time: the voltage E(t) is a 
square wave. The linear constraint is fulfilled when the two voltages are the same. 
When E(t) changes suddenly, the circuit changes -Ca: - Dy to compensate. The 
unusually shaped noise is caused by digitization by the oscilloscope. 
observed, target (V) 
Constraint Fulfillment with Ringing 
-0.1 
-0.3 
0.0 1.0 2.0 3.0 4.0 
Time (10 -2 Sec) 
Figure 4. Plot of two input voltages of transconductance amplifier: the con- 
straint forces are increased, which causes the system to undergo damped oscillations 
around the constraint manifold. 
Analog Circuits for Constrained Optimization 781 
The basic differential multiplier method converts the quadratic programming prob- 
lem into a system of differential equations: 
dx 
k dt 
= -2Az + 2Azo - CA, 
k: dy = -2By + 2Byo - D, (7) 
dt 
= Ca: + Dy + E(t). 
d 
The first two equations are implemented with a resistor and capacitor (with a fol- 
lower for zero output impedance). The third is implemented with resistor summing 
into the negative input of a transconductance amplifier. The positive input of the 
amplifier is connected to E(t). 
The circuit in figure 2 implements the system of differential equations 
dx 
dy 
C d 
-- Gi(/- ;g) q- G2(Vx - lg), 
= 63( - y) + G4(vp - y), 
= zc + CYh 
61q-"-33 ] 
(8) 
where K is the transconductance of the transconductance amplifier. The two sys- 
tems of differential equations (7) and (8) can match with suitably chosen constants. 
The circuit in figure 2 actually performs quadratic programming. The constraint is 
fulfilled when the voltages on the inputs of the transconductance amplifier are the 
same. The g function is a difference between these voltages. Figure 3 is a plot of 
-Ca: - Dy and E(t) as a function of time: they match reasonably well. The circuit 
in figure 2 therefore successfully fulfills the specified constraint. 
Decreasing the capacitance C3 changes the spring constant of the second-order dif- 
ferential equation. The forces that push the system towards the constraint manifold 
are increased without changing the damping. Therefore, the system becomes un- 
derdamped and the constraint is fulfilled with ringing (see figure 4). 
The circuit in figure 2 can be easily expanded to solve general quadratic program- 
ming for N variables: simply add more xi neurons, and interconnect them with 
resistors. 
4 CONSTRAINED FLIP-FLOP 
A flip-flop is two inverters hooked together in a ring. It is a bistable circuit: one 
inverter is on while the other inverter is off. A flip-flop can also be considered the 
simplest neural network: two neurons which inhibit each other. 
If the inverters have infinite gain, then the flip-flop in figure 5 minimizes the function 
flip-flop -- 64 V1U2 q- 62 V2U1- 6111U1- 6312112 + 61 q- 62 U1  q- 63 q- 641122 ' (9) 
2 2 
782 Platt 
U: G4 
G3 
G2 
Vl 
Figure 5. A flip-flop. Ux and U2 are voltages. 
-. G2 G 
Ux 
G3 
C1 
C2 
U2 
G4 
I 
Figure 6. A circuit for constraining a flip-flop. Ux, U2, and A are voltages. 
Analog Circuits for Constrained Optimization 783 
observed, target (V) 
Constraint Satisfaction for Non-Quadratic f 
0.8[ 
0.4 
0.0 
-0.4, 
0.0 
0.4 0.8 1.2 1.6 
Time (10 -2 Sec) 
Figure 7. Constraint fulfillment for a non-quadratic optimization function. 
The plot consists of the two input voltages of the transconductance amplifier. Again, 
E(t) is the dashed line and -Cx - Dy is the solid line. The constraint is fulfilled 
when the two voltages are the same. As the constraint changes with time, the flip- 
flop changes state and the location of the constrained minimum changes abruptly. 
After the abrupt change, the constraint is temporarily not fulfilled. However, the 
circuit quickly fulfills the constraint. The temporary violation of the constraint 
causes the transient spikes in the -Cx - Dy voltage. 
784 Platt 
Now, we can construct a circuit that minimizes the function in equation (9), subject 
to some linear constraint Cx q-Dy q-E(t) = 0, where x and y are the inputs to the 
inverters. The circuit diagram is shown in figure 6. Notice that this circuit is very 
similar to the quadratic programming circuit. Now, the x and y circuits are linked 
with a flip-flop, which adds non-quadratic terms to the optimization function. 
The voltages -Cx -Dy and E(t) for this circuit are plotted in figure 7. For most 
of the time, -Cx - Dy is close to the externally applied voltage E(t). However, 
because G1  (74 and G2  Gs, the flip-flop moves from one minime[ to the other 
and the constraint is temporarily violated. But, the circuitry gradually enforces the 
constraint again. The temporary constraint violation can be seen in figure 7. 
5 CONCLUSIONS 
This paper examines real circuits that have been constrained with the differential 
multiplier method. The differential multiplier method seems to work, even when the 
underlying circuit is non-linear, as in the case of the constrained flip-flop. Other pa- 
pers examine applications of the differential multiplier method [Platt, 1987] [Gindi, 
1988]. These applications could be built with the same parallel analog hardware 
discussed in this paper. 
Acknowledgement 
This paper was made possible by funding from AT&T Bell Labs. 
provided by Carver Mead, and Synaptics, Inc. 
Hardware was 
References 
Arrow, K., Hurwicz, L., Uzawa, H., [1958], Studies in Linear Nonlinear Program- 
ming, Stanford University Press, Stanford, CA. 
Durbin, R., Willshaw, D., [1987], "An Analogue Approach to the Travelling Sales- 
man Problem," Nature, 326, 689-691. 
Gill, P. E., Murray, W., Wright, M. H., [1981], Practical Optimization, Academic 
Press, London. 
Gindi, G, Mjolsness, E., Anandan, P., [1988], "Neural Networks for Model Matching 
and Perceptual Organization," Advances in Neural Information Processing Systems 
I, 618-625. 
Hopfield, J. J., Tank, D. W., [1985], "'Neural' Computation of Decisions in Opti- 
mization Problems," Biol. Cyber., 52, 141-152. 
Mead, C. A., [1989], Analog VLSI and Neural Systems, Addison-Wesley, Reading, 
MA. 
Platt, J. C., Hopfield, J. J., [1986], "Analog Decoding with Neural Networks," 
Neural Networks for Computing, Snowbird, UT, 364-369. 
Platt, J. C., Barr, A., [1987], "Constrained Differential Optimization," Neural In- 
formation and Processing Systems, 612-621. 
