Grouping Contours by Iterated Pairing Network 
Arenon $hashua 
M.I.T. Artificial Intelligence Lab., NE43-737 
and Department of Brain and Cognitive Science 
Cambridge, MA 02139 
Abstract 
Shimon Ullman 
We describe in this paper a network that performs grouping of image con- 
tours. The input to the net are fragments of image contours, and the 
output is the partitioning of the fragments into groups, together with a 
saliency measure for each group. The grouping is based on a measure of 
overall length and curvature. The network decomposes the overall opti- 
mization problem into independent optimal pairing problems performed 
at each node. The resulting computation maps into a uniform locally 
connected network of simple computing elements. 
I The Problem: Contour Grouping 
A problem that often arises in visual information processing is the linking of con- 
tour fragments into optimal groups. For example, certain subsets of contours spon- 
taneously form perceptual groups, as illustrated in Fig. 1, and are often detected 
immediately without scanning the image in a systematic manner. Grouping process 
of this type are likely to play an important role in object recognition by segmenting 
the image and selecting image structures that are likely to correspond to objects of 
interest in the scene. 
We propose that some form of autonomous grouping is performed at an early stage 
based on geometrical characteristics, that are independent of the identity of objects 
to be selected. The grouping process is governed by the notion of saliency in a way 
that priority is given to forming salient groups at the expense of potentially less 
salient ones. This general notion can again be illustrated by Fig. 1; it appears that 
certain groups spontaneously emerge, while grouping decisions concerning the less 
salient parts of the image may remain unresolved. As we shall see, the computation 
below exhibits a similar behavior. 
We define a grouping of the image contours as the formation of a set of disjoint 
335 
336 Shashua and Ullman 
Figure 1: Contours that spontaneously form perceptual groups with various degrees of 
saliency. On the left is an edge image of a plane surrounded by a car, a house, trees 
and texture. The image on the right contains three circles, having decreasing degrees of 
saJiency, in a background of randomly placed and oriented segments. 
groups, each corresponding to a curve that may have any number of gaps, and 
whose union covers all the contour fragments in the image. Given a function F(A) 
that measures some desired property of a group A, we would like to find a disjoint set 
of groups {A1, ...,A,} that maximizes y' F(A) over all possible groupings. Our 
definition of the problem is related to, but not identical with, problems studied in 
the past under headings of "perceptual organization", "segmentation", "cueing" and 
"figure-ground separation". In our definition of grouping, local grouping decisions 
based on collinearity of neighboring edge segments may be overridden in favor of 
more global decisions that are governed by the overall saliency of the groups. 
The paper introduces a novel grouping method having the following properties: (i) 
the grouping optimizes all overall saliency measure, (ii) the optimization problem 
is mapped onto a uniform locally connected network of simple computing elements, 
and (iii) the network's architecture and its computation are different in several 
respects from traditional neural network models. 
2 Optimal Grouping 
For the purpose of grouping it is convenient to consider the image as a graph of 
edge elements. The vertices of the graph correspond to image pixels, and the arcs to 
elementary edge fragments. The input to the grouping problem is a contour image, 
represented by a subset E" of the elements in the graph. A path in the graph 
corresponds to a contour in the image having any number of gaps. This implies 
that the grouping process implicitly bridges across gaps. This filling-in process is 
critical to any grouping scheme as demonstrated by the circles in Fig. 1. 
The emphasis in this paper is on 1-D chains of elements such as objects' bounding 
contours. Grouping is therefore a collections of chains of A1, ..., Am such that Ai  
Aj - 0 i  j and LJiAi D E r. To define an optimal grouping we will define a 
function F(A) that measures the quality of a group A. An optimal grouping is then 
a grouping that maximizes x F(A) over all possible groupings of the elements. 
Grouping Contours by Iterated Pairing Network 337 
2.1 The Quality Measure of a Group, F(A) 
The definition of the measure F(A) is motivated by both perceptual and compu- 
tational considerations. In agreement with perceptual observations, it is defined 
to favor long smooth contours. Its form is also designed to facilitate distributed 
multistage optimization, as discussed below. 
To define F(A) of a chain of elements A: {el, ..., era}, consider first a single element 
el, and the n preceding elements in the chain. We use first a quantity s,(i) which 
is the contribution of the n preceding elements to el, which is: 
i 
= 
j=max{1,i--n} 
aj is defined as 1 when ej corresponds to a contour fragment in the image and 0 for 
gaps. s, (i) is therefore simply a weighted sum of the contributions of the elements 
in the chain. The weighting factor Cij is taken to be a decreeing function of the 
total curvature of the path 7ij between elements ei and ej. This will lead to a 
grouping that prefers curves with small overall curvature over wiggly ones. Cii is 
given by the formula: 
Gj = e f,,iSds 
The exponent is the squared total curvature of the path between elements ei and e, 
and the resulting C 0 lies between 0 (highly curved contour) and 1 (straight line). 
For a discrete sampling of the curve, Cii can be approximated by the product' 
%: c. = 1 
p=i 
where fp,q is referred to as the coupling constant between adjacent elements ep and 
eq and is given by fr,q = e--atari  where a is the angle menuring the orientation 
difference between p and q [3]. In a similar manner, one can define s.i), the 
contribution of the n elements following e in the chain. &(i) = s.(i) + S.(i) - 
ai menures the contribution to element ei kom both direction. This incre.es 
monotonically with the leugth and low total curvature of the curve psing through 
element ei. Then then the overall quality of the chain A is finally given by 
i=1 
F,(A) increases quadratically with the size of A and is non-linear with respect to 
the total curvature of A. Maximizing  F(Aj) over all possible groupings will, 
therefore, prefer groups that are long and smooth. As n increases, the measure F, 
will depend on larger portions of the curve surrounding each element, resulting in 
a finer discrimination betweeu groups. In practice, we limit the measure to a finite 
n, and the optimal grouping is defined as: 
m 
In = arg max E Fn(Ai) 
m,A,...,Am i=1 
where the max is taken over all possible groupings. That is, we are looking for 
a grouping that will maximize the overall criterion function based on length and 
smoothness. 
338 Shashua and Ullman 
3 The Optimization Approach 
Optimizing I, is a nonlinear problem with an energy landscape that can be quite 
complex making it difficult to find a global optimum, or even good local optima, 
using straightforward gradient descent methods. We define below a computation 
that proceeds in two stages, saliency and pairing stages, of n steps each. In the 
saliency stage we compute, by iterating a local computation, optimal values of 
$n(i) for all elements in the graph. These values are an upper-bound on the saliency 
values achievable by any grouping. In the pairing stage we further update $,(i) by 
repeatedly forming local pairings of elements at each node of the graph. The details 
of both stages are given below. 
3.1 Saliency Stage 
For any given grouping A1, ..., Am, because they are disjoint, we have that 
m N 
y. F,(Ai) =  $,.,(i) _< 
j=l i=1 
where N is the number of elements in the graph and 7i is a curve passing through 
element ei. We denote S..(i) to be the saliency of element ei with respect to a curve 
7i. We therefore have that the maximal saliency value $.*(i) '-- max., S.(i) is an 
upper-bound on the saliency value element ei receives on the optimal grouping I.. 
We define a local computation on the grid of elements such that each element ei 
computes maximal S,(i) by iterating the following simple computation, at each 
step taking the maximal contribution of its neighbors. 
so(i) = ai 
s,+(i) = ai + m.axs.(j)fij (1) 
$ 
where this computation is performed by all elements in parallel. It can be shown 
that at the n'th iteration s(i) is maximal over all possible curves of length n, 
having any number of gaps, that come into ei. Since $,(i)= s,(i)+ ,(i)-ai, we 
have found the maximal S (i) as well. For further details on the properties of this 
computation, see [3]. Note that since the computation is carried by all elements of 
the net, including gaps (r equals 0), the gaps are filled-in as a by-product of the 
computation. One can show that the filling-in contour between two end-elements 
has the smallest overall curvature, and therefore has the shape of a cubic spline. 
3.2 Pairing Stage 
Given the optimal saliency values $*(i) computed at the saliency stage we would like 
next to find a near-optimal grouping I,. We first note the one-to-one correspondence 
between a grouping and a pairing of elements at each node of the r. aph. We define 
a pairing to be a partition of the k elements around node P into [7] disjoint pairs. 
A pairing performed over all nodes of the net creates an equivalence relation over 
the elements of the net and therefore, by transitivity, determines a grouping. We 
therefore proceed by selecting a pairing at each node of the net that will yield a 
near optimal grouping I. 
Given s,(i), the optimal saliency values computed by (1), and a pairing at node P 
Grouping Contours by Iterated Pairing Network 339 
we update the saliency values by 
s.+(i) = ai + s.(j)fii 
(2) 
where ei and ej are pairs determined by the pairing. This computation is exactly 
like (1) with the exception that (2) is applied to a fixed pairing while in (1) each 
element selects the neighbor with maximal contribution. Further applications of 
pairing followed by (2) allows the result of pairing decisions to propagate along 
curves and influence other pairing decisions. This gives rise to the notion of iterated 
pairings, a repetitive pairing procedure applied simultaneously over all nodes of the 
graph followed by saliency computation (2). We define below a pairing procedure 
that identifies salient groups in contour images. 
For every node P in the graph with elements el, ..., e} coming into P, we have that 
s,(i) i = 1,...,k computed by (1) are measured along optimal, not necessarily 
disjoint, curves A, ..., Ak of length n each. An optimal pairing at node P is defined 
as a disjoint pairing that concatenates A1,...,A into [] curves such that the 
sum of their quality measure F(-) is maximal. Because F is defined to prefer 
smooth curves and because of its non-linearity with respect to total curvature, an 
optimal pairing agrees with the notion of forming salient groups on the expense 
of potentially less salient ones. The following proposition shows that an optimal 
pairing can be determined locally without the need to evaluate the quality measure 
of the concatenated curves. 
Proposition I For a given nde P, let e,...,e be the elements around P, 
A,...,A be curves comi7g into P that are associated with the non-zero saliency 
values s(n),...,s(n) with suJficicntly large n (at least twice the largest chain A), 
r be a permutation oftbe inices(1,...,k) and J = {(1,2),(3,4),...,(k-l,k)), then 
argmax  Fn(Ar.Ar,)=argmax  w.,,., 
71' 71' 
(i,j)6J (i,j)eJ 
where AiAj stands for the concatenation of curves Ai,Aj, and wij = 
fij (s,(i)c,(j) + s,(j)c(i)) where c, is defined as c,(i) = Z Cj where k is taken 
over all elements in the chain Aj. 
Proof: This is merely a calculation. F, (AiAj), the measure of group-saliency of the 
chain AAj, is equal to F,,(A) + F,(Aj) + wj. Finally, without loss of generality, 
we can assume that k is even, because we can always add another element with zero 
weights attached to it.  
Proposition 1 shows that an optimal pairing of elements can be determined locally on 
the basis of the saliency values computed in (1). One way to proceed is therefore the 
following. The quantities c and therefore wij can be accumulated and computed 
during computation (1). Then, the optimal pairing is computed at every node. 
Finding an optimal pairing is equivalent to finding an optimal weighted match in a 
general graph [2], with weights wij. The weighted matching problem on graphs has 
a polynomial algorithm due to Edmonds [1] and therefore its implementation is not 
unwieldly. 
Below we describe an alternative and more biologically plausible scheme that can 
be implemented in a simple network using iterative local computations. The com- 
putation is in fact almost identical to the saliency computation described in (1). 
340 Shashua and Ullman 
Since the saliency values s, computed by (1) are an upper-bound on the final values 
achievable in any grouping, we would like to find a pairing that will preserve these 
values as closely as possible. Suppose that at P, ei receives its maximal contri- 
bution from e j, and at the same time ei provides the maximal contribution to ej 
('mutual neighbors'). When performing local pairing at P, it is reasonable to select 
ei and ej as a pair. Note that although this is a local decision at P, the values 
s,(i) and s,(j) already take into account the contribution of extended curves. The 
remaining elements undergo another round of saliency selection and pairing of mu- 
tual neighbors, until all elements at P are paired. The following proposition shows 
that this pairing process is well behaved in the sense that at each selection round 
there will always be at least one pair that mutually select each other. We therefore 
have that the number of selection rounds is bounded by [], where k is the number 
of elements having non-zero saliency value coming into node P. 
proposition 2 Let xl , ..., x be k positive real numbers, {dij -- {dji be positive weights 
i, j = 1, ..., k and 5i = arg maxj xjcoij, then 3i, j such that 5i = j and 5j = i (i and 
j are mutual neighbors). 
Proof: by induction on k. For k = 3 assume there exists a cycle in the selection 
pattern. For any given cycle we can renumber the indecis such that 51 = 2, 52 = 3 
and 53 = 1. Let wi stand for Wi-l,i where wl = wk,1. We get (i) x2w2 > x3wi, (it) 
x2w3 > xw2 and (iii) XWl > x23. From (it) and (iii) we get an inequality that 
contradicts (i). For the induction hypothesis, assume the claim holds for arbitrary 
k - 1. We must show that the claim holds for k. Given the induction hypothesis 
we must show that there is no selection pattern that will give rise to a cycle of 
size k. Assume in contradiction that such a cycle exists. For any given cycle of 
size k we can renumber the indecis such that 5i = i + 1 and 5k = 1 which implies 
that CiCO i  CjCOij for all j  i. In particular we have the following k inequalities: 
ildi  i_2Wi_ 1 where i = 1,...,k. From the k- 1 inequalities corresponding 
to i = 2, ..., k we get, by transitivity, that XlWl < Xk_lWk which contradicts the 
remaining inequality that correspouds to i = 1. ] 
3.3 Summary of Computation 
The optimization is mapped onto a locally connected network with a simple uniform 
computation. The computation consists of the following steps. (i) Compute the 
saliency S,* of each line element using the computation defined in (1). (it) At each 
node perform a pairing of the line elements at the node. The pairing is performed 
by repeatedly selecting mutual neighbors. (iii) Update at each node the values s, 
based on the newly formed pairing (eq. 2). (iv) Go back to step 2. 
These iterated pairings allow pairing decisions to propagate along maximally salient 
curves and influence other pairing decisions. In the implementation, the number of 
iterations n is equal iu both stages and as n increases, the finer the pairing would 
be, resulting in a finer discrimination between groups. During the computation, the 
more salient groups emerge first, the less salient groups require additional iterations. 
Although the process is not guaranteed to converge to an optimal solution, it is a 
very simple computation that yields in practice good results. Some examples are 
shown in the next section. 
Grouping Contours by Iterated Pairing Network 341 
Figure 2: Results after 30 iterations of saliency and pairing on a net of size 128 x 128 with 
16 elements per node. Images from left to right display the saliency map following the 
saliency and pairing stages and a number of strongest groups. The saliency of elements in 
the display is represented in terms of brightness and width -- increased saliency measure 
corresponds to increase in brightness and width of element in display. 
3.4 Examples 
Fig. 2 shows the results of the network applied to the images in Fig. 1. The 
sallehey values following the saliency and pairing stages illustrate that perceptually 
salient curves are also associated with high saliency values (see also [3]). Finally, 
in these examples, the highest saliency value of each group has been propagated 
along all elements of the group such that each group is now associated with a single 
saliency value. A number of strongest groups has been pulled out showing the close 
correspondence of these groups to objects of interest in the images. 
Acknowledgments 
This work was supported by NSF grant IRI-8900267. Part of the work was done 
while A.S. was visiting the exploratory vision group at IBM research center, York- 
town Heights. 
References 
[1] J. Edmonds. Path trees and flowers. Can. J. Math., 1:263-271, 1965. 
[2] C.H. Papadimitriou and K. Steiglitz. Combinatorial Optimization: Algorithms 
and Complexity. Prentice-Hall, New Jersey, 1982. 
[3] A. Shashua and S. Ullman. Structural saliency: The detection of globally salient 
structures using a locally connected network. In Proceedings of the nd Inter- 
national Conference on Computer Vision, pages 321-327, 1988. 
