Contour Organisation with the EM 
Algorithm 
J. A. F. Leite and E. R. Hancock 
Department of Computer Science 
University of York, York, Y01 5DD, UK. 
Abstract 
This paper describes how the early visual process of contour organ- 
isation can be realised using the EM algorithm. The underlying 
computational representation is based on fine spline coverings. Ac- 
cording to our EM approach the adjustment of spline parameters 
draws on an iterative weighted least-squares fitting process. The 
expectation step of our EM procedure computes the likelihood of 
the data using a mixture model defined over the set of spline cover- 
ings. These splines are limited in their spatial extent using Gaus- 
sian windowing functions. The maximisation of the likelihood leads 
to a set of linear equations in the spline parameters which solve the 
weighted least squares problem. We evaluate the technique on the 
localisation of road structures in aerial infra-red images. 
1 Introduction 
Dempster, Laird and Rubin's EM (expectation and maximisation) [1] algorithm was 
originally introduced as a means of finding maximum likelihood solutions to prob- 
lems posed in terms of incomplete data. The basic idea underlying the algorithm 
is to iterate between the expectation and maximisation modes until convergence is 
reached. Expectation involves computing a posteriori model probabilities using a 
mixture density specified in terms of a series of model parameters. In the max- 
imisation phase, the model parameters are recomputed to maximise the expected 
value of the incomplete data likelihood. In fact, when viewed from this perspective, 
the updating of a posteriori probabilities in the expectation phase would appear 
to have much in common with the probabilistic relaxation process extensively ex- 
ploited in low and intermediate level vision [9, 2]. Maximisation of the incomplete 
Contour Organisation with the EM Algorithm 881 
data likelihood is reminiscent of robust estimation where outlier reject is employed 
in the iterative re-computation of model parameters [7]. 
It is these observations that motivate the study reported in this paper. We are 
interested in the organisation of the output of local feature enhancement operators 
into meaningful global contour structures [13, 2]. Despite providing one of the clas- 
sical applications of relaxation labelling in low-level vision [9], successful solutions 
to the iterative curve reinforcement problem have proved to be surprisingly elusive 
[8, 12, 2]. Recently, two contrasting ideas have offered practical relaxation operat- 
ors. Zucker et al [13] have sought biologically plausible operators which draw on 
the idea of computing a global curve organisation potential and locating consistent 
structure using a form of local snake dynamics [11]. In essence this biologically 
inspired model delivers a fine arrangement of local splines that minimise the curve 
organisation potential. Hancock and Kittler [2], on the other hand, appealed to a 
more information theoretic motivation [4]. In an attempt to overcome some of the 
well documented limitations of the original Rosenfeld, Hummel and Zucker relaxa- 
tion operator [9] they have developed a Bayesian flamework for relaxation labelling 
[4]. Of particular significance for the low-level curve enhancement problem is the un- 
derlying statistical framework which makes a clear-cut distinction between the roles 
of uncertain image data and prior knowledge of contour structure. This framework 
has allowed the output of local image operators to be represented in terms of Gaus- 
sian measurement densities, while curve structure is represented by a dictionary of 
consistent contour structures [2]. 
While both the fine-spline coverings of Zucker [13] and the dictionary-based re- 
laxation operator of Hancock and Kittler [2] have delivered practical solutions to 
the curve reinforcement problem, they each suffer a number of shortcomings. For 
instance, although the fine spline operator can achieve quasi-global curve organisa- 
tion, it is based on an essentially ad hoc local compatibility model. While being 
more information theoretic, the dictionary-based relaxation operator is limited by 
virtue of the fact that in most practical applications the dictionary can only realist- 
ically be evaluated over at most a 3x3 pixel neighbourhood. Our aim in this paper 
is to bridge the methodological divide between the biologically inspired fine-spline 
operator and the statistical flamework of dictionary-based relaxation. We develop 
an iterative spline fitting process using the EM algorithm of Dempster et al [1]. 
In doing this we retain the statistical flamework for representing filter responses 
that has been used to great effect in the initialisation of dictionary-based relaxa- 
tion. However, we overcome the limited contour representation of the dictionary by 
drawing on local cubic splines. 
2 Prerequisites 
The practical goal in this paper is the detection of line-features which manifest 
themselves as intensity ridges of variable width in raw image data. Each pixel 
is characterised by a vector of measurements, z_ i where i is the pixel-index. This 
measurement vector is computed by applying a battery of line-detection filters of 
various widths and orientations to the raw image. Suppose that the image data 
is indexed by the pixel-set I. Associated with each image pixel is a cubic spline 
parameterisation which represents the best-fit contour that couples it to adjacent 
feature pixels. The spline is represented by a vector of parameters denoted by 
882 J. A. E Leite and E. R. Hancock 
3 T 
-qi = (q' q' q' qi )  
indexed i. The spline 
the pixel indexed j is 
j. We can write the 
2 3 
S_i, j = (1, $i,j, $i,j, $i,j 
the predicted vertical 
Let (xi,yi) represent the position co-ordinates of the pixel 
variable, $i,j --- Xi --Xj associated with the contour connecting 
the horizontal displacement between the pixels indexed i and 
cubic spline as an inner product F(si,j, _qi) - _qiT._si,j where 
)T. Central to our EM algorithm will be the comparison of 
spline displacement with its measured value ri,j -- Yi -- Yj. 
In order to initialise the EM algorithm, we require a set of initial spline probabilities 
which we denote by x(_q?). Here we use the multi-channel combination model 
recently reported by Leite and Hancock [5] to compute an initial multi-scale line- 
feature probability. Accordingly, if E is the variance-covariance matrix for the 
components of the filter bank, then 
[ 1 Tv,-1 ] 
r(_q?) = 1 -- exp [--z_ i z. z_ i 
(1) 
The remainder of this paper outlines how these initial probabilities are iteratively re- 
fined using the EM algorithm. Because space is limited we only provide an algorithm 
sketch. Essential algorithm details such as the estimation of spline orientation and 
the local receptive gating of the spline probabilities are omitted for clarity. Full 
details can be found in a forthcoming journal article [6]. 
3 Expectation 
Our basic model of the spline organisation process is as follows. Associated with 
each image pixel is a spline parameterisation. Key to our philosophy of exploiting 
a mixture model to describe the global contour structure of the image is the idea 
that the pixel indexed i can associate to each of the putative splines residing in a 
local Gaussian window Ni. We commence by developing a mixture model for the 
conditional probability density for the filter response _z i given the current global 
spline description. If (n) _ {97),V i 6 I} is the global spline description at 
iteration n of the EM process, then we can expand the mixture distribution over a 
set of putative splines that may associate with the image pixel indexed i 
)(_qj ) (2) 
jNi 
The components of the above mixture density are the conditional measurement 
densities p(z_il () .__j ). The conditional 
_qj ) and the spline mixing proportions 
measurement densities represent the likelihood that the datum z_ originates from the 
spline centred on pixel j. The mixing proportions, on the other hand, represent the 
fractional contribution to the data arising from the jth parameter vector i.e. _q). 
Since we are interested in the maximum likelihood estimation of spline parameters, 
we turn our attention to the likelihood of the raw data, i.e. 
p(z_i, Vi  I[(b ) = IIP(z_il(I, ) (3) 
The expectation step of the EM algorithm is aimed at estimating the log-likelihood 
using the parameters of the mixture distribution. In other words, we need to av- 
erage the likelihood over the space of potential pixel-spline assignments. In fact, 
Contour Organisation with the EM Algorithm 883 
it was Dempster, Laird and Rubin [1] who observed that maximising the weighted 
log-likelihood was equivalent to maximising the conditional expectation of the like- 
lihood for a new parameter set given an old parameter set. For our spline fitting 
problem, maximisation of the expectation of the conditional likelihood is equivalent 
to maximising the weighted log-likelihood function 
(n) In p(z_il_q(j + ) 
(_% Iz_i) ) 
(4) 
The a posteriori probabilities P(95 M [z_i) may be computed from the corresponding 
components of the mixture density p(z_i[q M) using the Bayes formula 
q!,) ]x(q!,)] (5) 
- YkeN, P(Z-i[_, , ,_, , 
For notational convenience, and to make the weighting role of the a posteriori prob- 
 (M = p(q}MIz_i). Once updated parameter 
abilities explicit we use the shorthand .vi, j _ 
estimates _q?) become available through the maximisation of this criterion, im- 
proved estimates of the mixture components may be obtained by substitution into 
equation (6). The updated mixing proportions, (_q?+)), required to determine 
the new weights  (M 
'vi, j are computed from the newly available density components 
using the following estimator 
I_qi ) (_qi) 
7r (n+l) 
)= p(z 
I_q ) 
jeN, Y-kelP(Z-j (n) (n) 
(6) 
In order to proceed with the development of a spline fitting process we require 
a model for the mixture components, i.e. p(z_il_q}M ). Here we assume that the 
required model can be specified in terms of Gaussian distribution functions. In 
other words, we confine our attention to Gaussian mixtures. The physical variable 
of these distributions is the squared error residual for the position prediction of the 
ith datum delivered by the jth spline. Accordingly we write 
19} )- exp 
where fi is the inverse variance of the fit residuals. Rather than estimating fi, we 
use it in the spirit of a control variable to regulate the effect of fit residuals. 
Equations (5), (6) and (11) therefore specify a recursire procedure that iterates the 
weighted residuals to compute a new mixing proportions based on the quality of 
the spline fit. 
4 Maximisation 
The maximisation step aims to optimize the quantity Q((+)l (')) with respect 
to the spline parameters. Formally this corresponds to finding the set of spline 
parameters which satisfy the condition 
('+) = arg max Q(I(') ) (8) 
884 J. A. E Leite and E. R. Hancock 
We find a local approximation to this condition by solving the following set of linear 
equations 
= 0 (9) 
0(q/k) ( n+l ) 
for each spline parameter (q/k)(n+l) in turn, i.e. for k=0,1,2,3. Recovery of the 
splines is most conveniently expressed in terms of the following matrix equation for 
the components of the parameter-vector 
_q(=+) (=)-2 (=) 
i =(Ai) _X i 
(10) 
The elements of the vector X(M are weighted cross-moments between the parallel 
and perpendicular spline distances in the Gaussian window, i.e. 
(11) 
The elements of the matrix A? ) on the other hand, are weighted moments com- 
puted purely in terms of the parallel distance si,j. If k and I are the row and column 
indices, then the (k,/)th element of the matrix A? ) is 
-- Wi,j i,j 
jNi 
(12) 
5 Experiments 
We have evaluated our iterative spline fitting algorithm on the detection of line- 
features in aerial infra-red images. Figure la shows the original picture. The initial 
feature probabilities (i.e. (q0))) assigned according to equation (1) are shown 
in Figure lb. Figure lc shows the final contour-map after the EM algorithm has 
converged. Notice that the line contours exhibit good connectivity and that the 
junctions are well reconstructed. We have highlighted a subregion of the original 
image. There are two features in this subregion to which we would like to draw 
attention. The first of these is centred on the junction structure. The second 
feature is a neighbouring point on the descending branch of the road. 
Figure 2 shows the iterative evolution of the cubic splines at these two locations. 
The spline shown in Figure 2a adjusts to fit the upper pair of road segments. Notice 
also that although initially displaced, the final spline passes directly through the 
junction. In the case of the descending road-branch the spline shown in Figure 2b 
recovers from an initially poor orientation estimate to align itself with the underlying 
road structure. Figure 2c shows how the spline probabilities (i.e. (q?))) evolve 
with iteration number. Initially, the neighbourhood is highly ambiguous. Many 
neighbouring splines compete to account for the local image structure. As a result 
the detected junction is several pixels wide. However, as the fitting process iterates, 
the splines move from the inconsistent initial estimate to give a good local estimate 
which is less ambiguous. In other words the two splines illustrated in Figure 2 have 
successfully arranged themselfs to account for the junction structure. 
Contour Organisation with the EM Algorithm 885 
(a) Original image. 
(b) Probability map. 
(c) Line map. 
Figure 1: Infra-red aerial picture with corresponding probability map showing region 
containing pixel under study and correspondent line map. 
6 Conclusions 
We have demonstrated how the process of parallel iterative contour refinement can 
be realised using the classical EM algorithm of Dempster, Laird and Rubin [1]. 
The refinement of curves by relaxation operations has been a preoccupation in the 
literature since the seminal work of Rosenfeld, Hummel and Zucker [9]. However, 
it is only recently that successful algorithms have been developed by appealing 
to more sophisticated modelling methodologies [13, 2]. Our EM approach not 
only delivers comparable performance, it does so using a very simple underlying 
model. Moreover, it allows the contour re-enforcement process to be understood in 
a weighted least-squares optimisation framework which has many features in com- 
mon with snake dynamics [11] without being sensitive on the initial positioning of 
control points. Viewed from the perspective of classical relaxation labelling [9, 4], 
the EM framework provides a natural way of evaluating support beyond the im- 
mediate object neighbourhood. Moreover, the flamework for spline fitting in 2D is 
readily extendible to the reconstruction of surface patches in 3D [10]. 
References 
[1] Dempster A., Laird N. and Rubin D., "Maximum-likelihood from incomplete data 
via the EM algorithm", J. Royal Statistical Soc. Set. B (methodological),39, pp 1-38, 
1977. 
[2] Hancock E.R. and Kittier J., "Edge Labelling using Dictionaxy-based Probabilistic 
Relaxation", IEEE PAMI, 12, pp. 161-185, 1990. 
[3] Jordan M.I. and Jacobs R.A, "Hieraxchical Mixtures of Experts and the EM Al- 
gorithm", Neural Computation, 6, pp. 181-214, 1994. 
[4] Kittler J. and Hancock, E.R., "Combining Evidence in Probabilistic relaxation", 
International Journal of Pattern Recognition and Artificial Intelligence, 3, N1, pp 
29-51, 1989. 
[5] Leite J.A.F. and Hancock, E.R., "Statistically Combining and Refining Multichannel 
Information", Progress in Image Analysis and Processing III: Edited by S Impedovo, 
World Scientific, pp. 193-200, 1994. 
[6] Leite J.A.F. and Hancock, E.R., "Iterative curve organisation with the EM al- 
gorithm", to appear in Pattern Recognition Letters, 1997. 
886 J. A. E Leite and E. R. Hancock 
<v> / (v) ] (v9 
$() $(> $(> 
<rio <viii) (ix) 
s() $ , 
(> (xD 
(a) 
(b) 
(c) 
Figure 2: Evolution of the spline in the fitting process. The image in (a) is the 
junction spline while the image in (b) is the branch spline. The first spline is shown 
in (i), and the subsequent ones from (ii) to (xi). The evolution of the corresponding 
spline probabilities is shown in (c). 
[7] Meet P., Mintz D., Rosenreid A. and Kim D.Y., "Robust Regression Methods for 
Computer Vision - A Review", International Journal of Computer vision, 6, pp. 
59-70, 1991. 
[8] Peleg S. and Rosenreid A., "Determining Compatibility Coefficients for curve en- 
hancement relaxation processes", IEEE SMC, $, pp. 548-555, 1978. 
[9] Rosenreid A., Hummel R.A. and Zucker S.W., "Scene labelling by relaxation opera- 
tions", IEEE Transactions SMC, SMC-6, pp400-433, 1976. 
[10] Sander P.T. and Zucker S.W., "Inferring surface structure and differential structure 
from 3D images", IEEE PAMI, 12, pp 833-854, 1990. 
[11] Terzopoulos D., "Regularisation of inverse problems involving discontinuities", IEEE 
PAMI, $, pp 129-139, 1986. 
[12] Zucker, S.W., Hummel R.A., and Rosenfeld A., "An application of relaxation labelling 
to line and curve enhancement", IEEE TC, C-26, pp. 394-403, 1977. 
[13] Zucker S., David C., Dobbins A. and Iverson L., "The organisation of curve de- 
tection: coarse tangent fields and fine spline coverings", Proceedings of the Second 
International Conference on Computer Vision, pp. 577-586, 1988. 
