Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!newsfeed.internetmci.com!howland.erols.net!news.mathworks.com!newsgate.duke.edu!interpath!news.interpath.net!news.interpath.net!sas!newshost.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: RE: NN vs. Linear Regression
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <E578uF.D20@unx.sas.com>
Date: Thu, 6 Feb 1997 20:56:39 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References:  <5d827q$27s@topgun.es.dupont.com>
Organization: SAS Institute Inc.
Keywords: Over-fitting, Degrees of Freedom, "Memorization"
Lines: 29


In article <5d827q$27s@topgun.es.dupont.com>, owens@slivova.es.dupont.com (Aaron J. Owens) writes:
|> ...
|> "Hon" Kim replied, and e-mailed to me ...
|> ...
|> K>I thought that an extreme case of over-training happens when
|> K>the number of hidden units are same as the number of input
|> K>patterns, where the network memorizes (not learns!) all the
|> K>input patterns by mapping each input pattern to each hidden
|> K>unit. Thus, at most 20 hidden units will work no matter what 
|> K>the input patterns are. I still doubt that 4 hidden units are
|> K>enough for classification of 20 patterns.
|> 
|> I have heard this stated before. It is wrong. "Memorization"
|> occurs when the number of WEIGHTS, not the number of hidden units,
|> equals the number of patterns.

Some article, which I cannot locate at the moment, claimed that if
there are at least as many hidden units as training cases, then the
error surface has no bad local optima. Perhaps that is what Hon Kim
was thinking of.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
 *** Do not send me unsolicited commercial or political email! ***

