Newsgroups: sci.image.processing
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!udel!gatech!howland.reston.ans.net!EU.net!Germany.EU.net!unibwh.unibw-hamburg.de!unibwh!e_meier
From: e_meier@unibw-hamburg.de (Wilhelm Meier)
Subject: Re: Estimation of pdf - summary
Message-ID: <E_MEIER.94Nov8161945@siegen.unibw-hamburg.de>
Sender: news@unibw-hamburg.de
Reply-To: e_meier@unibw-hamburg.de
Organization: University of the Federal Armed Forces, Hamburg
Date: Tue, 8 Nov 1994 15:19:45 GMT
Lines: 111


Thanks to all contributors !
I got the following replies:

pham@iron.ece.jhu.edu (Dzung Pham) writes:

	From what I've heard, the method of moments is intractable for n>=3.  The 
	most common way to solve this problem is using the EM algorithm (Dempster, 
	J. Roy Stats, 1979-1981 I believe) which is an interative maximum 
	likelihood approach, (sorry).  If you want something faster with better 
	convergence properties, Bezdek (Pattern Rec. Letters, ~1985) showed that 
	the c-means (aka k-means) clustering algorithm provides a suboptimal MLE 
	that is faster and more robust with respect to initial conditions.  
	Several recent papers employ some derivative of the c-means clustering 
	algorithm to provide initial conditions for some derivative of the EM 
	algorithm resulting in a hybrid approach which is generally robust and 
	optimal.

EM algorithm is a very good choice ( see below for reference ).

bens@watson.ibm.com (Ben Shahshahani) writes:

	If I understand it correctly, you are trying to fit a multi-variate mixture
	density to some data. Method of moments is very messy, for the case
	of two multivariate gaussians see the paper by Fukunaga and Flick:
	"estimation of parameters of a Gaussian mixture using the method of moments" in
	IEEE Trans. Pattern Analysis and Machine Intelligence 1983, no 4, pp 410-416.
	The method of choice is usually maximum likelihood through the use of
	the EM algorithm. For more info also I suggest the following book:
	Statistical Analysis of Finite Mixture Distributions by
	D.M. Titterington, Smith, and Makov.

popat@image.mit.edu (Kris Popat) writes:

	Karl Pearson considered your problem for N=2, and suggested the
	method of moments approach in a paper published in 1894.

	Since then, there has been an enormous amount of research effort
	expended on this problem of mixture density estimation.

	Based on what I've read and what I've tried on my own, my feeling is
	that you will not find the non-iterative procedure you seek.

	However, there is a much more suitable iterative technique than the
	one you described, and it is widely considered to be the "right thing
	to do" to estimate the parameters in a mixture density.  It is usually
	known by the name "EM algorithm."

	It is usually credited to L.E. Baum (in the late sixties), but it has
	been discovered independently by lots of people.

	The comprehensive writeup on EM in the general setting is:

	Dempster, Laird, Rubin.  "Maximum likelihood from incomplete data
	via the EM algorithm," J. Royal Stat. Soc., Series B, v. 39, 1977,
	pp. 1-38.

	Specialized to the mixture estimation problem, a good writeup is:

	Redner, Walker.  "Mixture densities, maximum likelihood, and the
	EM algorithm,"  SIAM Review, v. 26, no. 2, April 1984, pp. 195-239.

Dempster et al. and Redner, Walker are good references.

"John Derrico" <derrico@pixel.kodak.com> writes:

	Sorry, but there is no simple way of solving the problem for the
	general case of N gaussians without recourse to an optimization.
	I've written code which uses a constrained optimizer that has
	estimated as many as ten gaussians, but as I remember, that
	tended to take up to about 30 minutes on a VAX. 

	I'll give you a hint if you intend to go the optimization route.

	Use a good constrained optimization package. I used one called
	VMCON. Constraints are important to prevent the optimization
	from diverging or getting lost in local minima. 

	Some constraints you might use are:

	1. specify a minimum distance between gaussian means.
	2. variances are positive and less than some reasonable maximum.
	3. mixture proportions are all positive and sum to unity.
	4. all gaussian means must fall within the bounds of your histogram,
	or if this is not the case, specify some reasonable bounds in which
	they must fall.

This exactly the method which I used for the last time. It works well, but is 
very time-consuming with many gaussians.
The EM gives comparable results, but is much faster if implemented to work
with histograms, not with distinct observations.
-- 
                             _\\|//_ 
                             ( O-O )
---------------------------o00--(_)--00o------------------------------
Dipl.-Ing. Wilhelm Meier	Phone:	(49)(0)40 / 65412524
UniBwH, FB ET			FAX:	(49)(0)40 / 65412822
Holstenhofweg 85		email:	e_meier@unibw-hamburg.de
D-22043	Hamburg, GERMANY


--
                             _\\|//_ 
                             ( O-O )
---------------------------o00--(_)--00o------------------------------
Dipl.-Ing. Wilhelm Meier	Phone:	(49)(0)40 / 65412524
UniBwH, FB ET			FAX:	(49)(0)40 / 65412822
Holstenhofweg 85		email:	e_meier@unibw-hamburg.de
D-22043	Hamburg, GERMANY


