Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!news.mathworks.com!news.kei.com!nntp.coast.net!swidir.switch.ch!swsbe6.switch.ch!news.unige.ch!usenet
From: orsier@cuisun38.unige.ch (Bruno Orsier)
Subject: Re: Second Order Methods
In-Reply-To: Bradley Warner's message of 25 Jan 1996 22:09:49 GMT
Message-ID: <ORSIER.96Feb1133053@cuisun38.unige.ch>
Lines: 46
Sender: usenet@news.unige.ch
Reply-To: orsier@cui.unige.ch
Organization: Centre Universitaire d'Informatique, University of Geneva
References: <4e8v3d$a6k@tali.UCHSC.edu>
Date: Thu, 1 Feb 1996 12:30:52 GMT

>>>>> "Bradley" == Bradley Warner <bw> writes:

Bradley> I am trying to use a Newton method to estimate the weights in a
Bradley> neural network with one hidden layer.  It appears that for several
Bradley> different data sets, that the Hessian is ill conditioned.  I am
Bradley> wondering if this is the result of the formulation of the neural
Bradley> network, ie. will neural networks always have an ill conditioned
Bradley> Hessian?  Does anyone have experience with a Newton-Raphson approach
Bradley> to optimizing neural networks and have you observed this near
Bradley> singularity of the Hessian?

You may be interested by the Scaled Conjugate Gradient algorithm
which is able to handle non positive definite Hessians. It is based on a 
Levenberg-Marquart approach.

See

@Article{MOLLER93,
  author = 	 "Martin F. Moller",
  title = 	 "A Scaled Conjugate Gradient Algorithm for Fast Supervised
		  Learning",
  journal =	 "Neural Networks",
  year =	 1993,
  volume =	 6,
  pages =	 "525-533"
}


@Book{BISH95,
  author = 	 "Christopher M. Bishop",
  title = 	 "Neural Networks for Pattern Recognition",
  publisher = 	 "Oxford University Press",
  year = 	 1995
}

An implementation of Scaled Conjugate Gradient is available with
Stuttgart Neural Network Simulator v4.1. See
 http://vasarely.informatik.uni-stuttgart.de/snns/snns.html

Bruno Orsier
--
-- 

Bruno Orsier		     			    E-mail: orsier@cui.unige.ch
University of Geneva	  WWW:http://cuiwww.unige.ch/AI-group/staff/orsier.html

