Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!nntp.sei.cmu.edu!news.cis.ohio-state.edu!math.ohio-state.edu!howland.reston.ans.net!swrinde!newsfeed.internetmci.com!in2.uu.net!news.interpath.net!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Backprop ERROR question
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <DosJGB.HDH@unx.sas.com>
Date: Sun, 24 Mar 1996 21:24:11 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References: <Dooxur.CJ3@unx.sas.com> <4j3bs7$9j$1@mhadf.production.compuserve.com>
Organization: SAS Institute Inc.
Lines: 43


Someone wrote:
|> The numerical analysis camp has a vast store of much better algorithms
|> all figured out. Look back over some of the articles about "conjugate
|> gradient" that were posted recently. 

saswss@hotellng.unx.sas.com (Warren Sarle) wrote:
|> I agree, but for someone who is still trying to get their first NN
|> program to work, there's a lot to be said for simplicity, even for
|> standard backprop!
   
In article <4j3bs7$9j$1@mhadf.production.compuserve.com>, Will Dwinnell <76743.1740@CompuServe.COM> replied:
|> I have never understood why everyone thinks backprop is so simple 
|> (to implement).  I would think that something like simulated 
|> annealing would be alot easier to put together.

The source code peculiar to standard backprop (excluding the code
for reading data, forward propagation, computing derivatives, etc.,
that is required for any gradient method) is about 3 statements in
C, Fortran, or any similar language.  Rprop is about 10. Quickprop
is about 20.  Conjugate gradient or Levenberg-Marquardt could run
into 100s of statements.

Simulated annealing relieves you of the need for derivatives, but
anyone who passed first-year calculus can compute the derivatives
for a single-hidden-layer network even if they've never heard of
backpropagation.

I would say that the big advantage for a beginner of standard
backprop over simulated annealing is that you can tell right away if
standard backprop is working (in batch mode with a small learning
rate) by seeing if the error function goes down on each epoch.
Determining whether simulated annealing is working correctly is more
problematic--never having programmed simulated annealing myself, I
can't say right off how you _would_ verify a simulated annealing
algorithm.


-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
