Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!udel!gatech!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: proof for backprop ??
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D180BI.LJ6@unx.sas.com>
Date: Thu, 22 Dec 1994 16:26:06 GMT
References:  <3d71e4$rn4@macculloch.loria.fr>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Keywords: backprop
Lines: 37


In article <3d71e4$rn4@macculloch.loria.fr>, pican@loria.fr (Nicolas Pican) writes:
|> ...
|> Many proof of convergence of the backprop algorithm (and its
|> optimization) has been known for a long time and can be found in :
|>
|> Broyden C. G. : The convergence of a class of double-rank minimization
|> algorithms 2: the new algorithm, in Journal Institute of Math. and its
|> Appl. 6, pp. 222-231. 1970
|> ...
[Numerous references to quasi-Newton, conjugate gradient, and other
irrelevant things deleted]

The generalized delta rule, aka standard backprop, is very different
from most of the well-known optimization methods found in the numerical
analysis literature such as steepest descent, quasi-Newton, and
conjugate gradient methods. Convergence proofs for such methods do not
apply to standard backprop. In fact, online standard backprop does _not_
converge--you need a variable learning rate as discussed in the article
that Mark Craven cited:

   @incollection{mangasarian.nips6
     ,author       = "O. L. Mangasarian and M. V. Solodov"
     ,title        = "Backpropagation convergence via deterministic perturbed minimization"
     ,booktitle    = "Advances in Neural Information Processing Systems"
     ,editor       = "J. Cowan and G. Tesauro and J. Alspector"
     ,year         = 1994
     ,volume       = 6
     ,pages        = "383--390"
     ,publisher    = "Morgan Kaufmann"
     ,address      = "San Francisco, CA"

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
