Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!udel!gatech!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: proof for backprop ??
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D16C2y.I5L@unx.sas.com>
Date: Wed, 21 Dec 1994 18:44:58 GMT
References: <1994Dec12.122304.44261@yogi> <3d4s37$48n@aplcomm.jhuapl.edu> <3d4vsr$t03@spool.cs.wisc.edu> <D14KDI.91D@unx.sas.com> <3d9c2t$hi9@aplcomm.jhuapl.edu>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 24


In article <3d9c2t$hi9@aplcomm.jhuapl.edu>, randy@aplcorejhuapl.edu (Randall C. Poe) writes:
|>
|> You can't in general prove better than convergence to a stationary point.
|> I think that's true even for second-order methods like Newton's and its
|> variants (quasi-Newton, Levenberg-Marquart, etc).  What the NN community needs
|> is a proof that the stationary points ARE reasonably good local minima.

Some of them assuredly are not. There are usually numerous saddle
points. E.g., setting all weights to zero in an MLP gives you a saddle
point.

|> As I said in a previous post, if you stick to the "batch" version of backprop
|> (update after summing over all patterns) and don't use a momentum term, you
|> are using plain vanilla steepest-descent, and the convergence proof for this
|> is well known (again, to a stationary point).  So there's a proof for backprop.

Except that steepest descent is usually used with a line search.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
