Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!news.sprintlink.net!redstone.interpath.net!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: NN Vs Stats......
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D3Fqpq.9Hu@unx.sas.com>
Date: Fri, 3 Feb 1995 17:46:38 GMT
References: <1995Jan11.145719.1@ulkyvx.louisville.edu> <3f9r6i$imn@newsbf02.news.aol.com> <3fa1j6$1kf@usenet.INS.CWRU.Edu> <3frq6t$6fc@maui.cs.ucla.edu> <3gaahk$q1m@usenet.INS.CWRU.Edu> <3gb54t$3sd@vixen.cso.uiuc.edu> <D3CDAC.E1u@unx.sas.com>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 45


In article <GEORGIOU.95Feb2164044@wiley.csusb.edu>, georgiou@wiley.csusb.edu (George M. Georgiou) writes:
|> WS> If this is an unrelated question about whether the original perceptron
|> WS> has an analog in statistics, then yes, a perceptron is a linear
|> WS> discriminant function, and the two most common training methods are
|> WS> logistic regression and the analytic solution assuming multivariate
|> WS> normal classes with equal covariance matrices.
|>
|> That was an unrelated question.  Sorry for not being clear.  The
|> property that the perceptron algorithm has, and my guess is that there
|> is nothing analogous in statistics, is that it can find a linear
|> discriminant that separates the two classes with zero classification
|> error, if possible to do so.   The linear discriminant methods in
|> statistics that I am aware of, e.g. Fisher's, regression,
|> entropy-based, etc., cannot guarantee exact separation.
|>
|> Also, the perceptron algorithm, makes no distribution assumptions --
|> absolutely.

Normal-theory linear discriminant analysis does not necessarily yield
perfect separation for linearly-separable classes.

Logistic regression is a neural net with no hidden layer, a logistic
output activation function, and usually a cross-entropy training
criterion.  Training can be done by any of the usual algorithms. I have
never known Levenberg-Marquardt to fail to separate linearly-separable
classes, but I'm not aware of a proof that it will always do so. You
could use a global optimization algorithm instead.

The perceptron learning rule is, of course, guaranteed to separate
linearly-separable classes, but if the classes are not
linearly-separable, the perceptron learning rule may not even converge.
Hence it would be unsuitable for statistical applications.

The linear programming approach is guaranteed to separate
linearly-separable classes and it always terminates in finite time. But
the training criterion that is used with linear programming has no
virtues that I know of other than making the problem soluble by linear
programming.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
