Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!agate!dog.ee.lbl.gov!newshub.nosc.mil!news!news
From: "R. Scott Starsman" <r_starsman@nise-p.nosc.mil>
Subject: Re: why back prop?
Message-ID: <1995Jan17.173255.3138@nosc.mil>
Sender: news@nosc.mil
Organization: NCCOSC RDT&E Division, San Diego, CA
References: <DavidC.16.2F19E40B@ise.canberra.edu.au> <3fcv84$750@cantaloupe.srv.cs.cmu.edu>
Date: Tue, 17 Jan 1995 17:32:55 GMT
Lines: 29

sef@CS.CMU.EDU (Scott Fahlman) wrote:
> Once you have dE/dw at the current point in weight space, the next
> question is how to adjust the weights so as to reach a local minimum
> (ideally jumping over small craters) in reasonable time.  Simple
> gradient descent is the simplest scheme.  It's easy to implement, easy
> to understand, and lends itself to case-at-a-time training.  Faster
> optimization methods are known, as you point out.  Neural net people
> working on big problems commonly use some form of conjugate-gradient
> descent or a method such as Quickprop that uses (an approximation to)
> the second derivative of the error function.  I consider these all to
> be forms of "back propagation" learning.
> 
> -- Scott
> 
> ===========================================================================
> Scott E. Fahlman			Internet:  sef+@cs.cmu.edu
> Principal Research Scientist		Phone:     412 268-2575
> School of Computer Science              Fax:       412 268-5576
> Carnegie Mellon University		Latitude:  40:26:46 N
> 5000 Forbes Avenue			Longitude: 79:56:55 W
> Pittsburgh, PA 15213			Mood:      :-)
> ===========================================================================
> 
What is Quickprop and how is it related to Backprop+Momentum?  How much faster is it?
I'm looking to improve my BP training time and any help you could provide would be
greatly appreciated.

Scott
r_starsman@nise-p.nosc.mil
