Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!gatech!swrinde!cs.utexas.edu!utnut!info.physics.utoronto.ca!cannon.ecf!srodula
From: srodula@ecf.toronto.edu (SRODULA  EWA ANNA)
Subject: RESPONSES: Levenburg-Marquardt algorithm
X-Nntp-Posting-Host: spark21.ecf
Message-ID: <D7x7CC.Ix@ecf.toronto.edu>
Sender: news@ecf.toronto.edu (News Administrator)
Organization: University of Toronto, Engineering Computing Facility
Date: Mon, 1 May 1995 22:19:24 GMT
Lines: 122

I wish to thank everyone who responded to my request for reference/
pointers on the L-M algorithm (now, available MATLAB NN Toolbox v2.0)
I also requested for comparison with the Conjugate Grad. method
used by Timothy Masters in his book "Practical Neural Network
Recipes in C++".
Here is a compillation of the responses (excuse me if I left
anyone out):


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hi Thas
Try  M.T.Hagan, M.B.Menhaj:Training Feedforward Networks with the Marquardt
Algorithm, IEEE Transactions on Neural Networks, Vol.5, No 6, Nov. 1994, p.
989-993
.....and tell me if you succeeded in understanding the paper.
Regard
Marcel Loher
Marcel Loher, Dr. sc.nat.
St.Gallen School of Engineering ISG
CH-9000 St.Gallen
Phone: ..71 30 87 24    Fax: ..71 30 87 87     e-mail: loher@dial-switch.ch

++++++++++++++++++++++++++++++++++++++++++++++++++++
From:	B Amavasai <B.Amavasai@sheffield.ac.uk>

You must understand that Masters implementation is used to understand
the algorithm and probably not as optimised as the LM algorithm
in Matlab's NN toolbox. Also, try fiddling around with the LM
incrementing factor in Master's implementation (I think that it
is the variable lambda). Master's recommends a value of 2 but
most mathematical text recommends 10.

I like the paper published by Martin Moller in 1993 entitled "Scaled 
Conjugate Gradient Descent...." in the Neural Network journal (I forget 
what issue).

Hope the info helps.

Regards,
Bala

+++++++++++++++++++++++++++++++++++++++++++++++++++++
From:	"Sameer M. Prabhu" <samprabh@acpub.duke.edu>


\bibitem{moll} Moller, M. F. ``A scaled conjugate gradient algorithm for
fast supervised learning'', {\it Neural Networks}, Vol. 6, No. 4,
pp. 525-533, 1993.


I believe that's Levenberg-Marquardt.

I think Marquardt's original paper appeared in a SIAM (Society for Industrial
and Applied Mathematics) Journal somewhere around 1975.

As I recall:

Basically this is a compromise between steepest descent (very slow convergence,
but converges from anywhere) and Newton's method (very fast convergence,
but converges only close to the optimum).  Newton's method also has the
disadvantage of requiring computation of the Hessian matrix (the second
derivative with respect to all pairs of variables x_i, x_j), or actually
it's inverse.  This is a very expensive computation.

The Newton direction is often not the same direction as the steepest
descent.  Levenberg-Marquardt attempts to use Newton (or perhaps a variant of
Newton, using a positive definite approximation to the Hessian) wherever
it gives good descent results, and steepest descent where it doesn't.
There is a parameter which causes the update direction to vary between
the two directions, and which also controls step size.  The algorithm
continuously varies this parameter to maintain descent as speedy as
possible while ensuring that the steps are indeed descents.


A reference for the Levenberg-Marquardt algorithm is:

M. Hagan and M. Menhaj, "Training Feedforward Networks with the Marquardt
Algorithm," IEEE Transactions on Neural Networks, Vol. 5, No. 6, November
1994, pp 989-993.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
From: mhagan@master.ceat.okstate.edu (hagan m t)

A reference for the Levenberg-Marquardt algorithm is:

M. Hagan and M. Menhaj, "Training Feedforward Networks with the Marquardt
Algorithm," IEEE Transactions on Neural Networks, Vol. 5, No. 6, November
1994, pp 989-993.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
From: saswss@hotellng.unx.sas.com (Warren Sarle)


Levenberg, K. (1944) "A method for the solution of certain problems
in least squares," Quart. Appl. Math., 2, 164-168.

Marquardt, D. (1963) "An algorithm for least-squares estimation of
nonlinear parameters," SIAM J. Appl. Math., 11, 431-441.

More\', J.J. (1977) "The Levenberg-Marquardt algorithm:
implementation and theory," in Watson, G.A., ed., _Numerical
Analysis_, Lecture Notes in Mathematics 630, Springer-Verlag,
Heidelberg, 105-116.

-- 
Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
From: el3bpa@sunc.sheffield.ac.uk (B Amavasai)
Tim Masters will be releasing a book entitled "Advanced Algorithms for
Neural Networks", published by Wiley. It should be out in a weeks time
in the US and around May/June in the UK. The book provides a practical
approach to the LM algorithm and other advance algorithms like 
Probabilistic NNs.
Tim can be contacted via e-mail at 74010.767@compuserve.com if you
require more info.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++



-Thas 
