Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!news.alpha.net!uwm.edu!spool.mu.edu!howland.reston.ans.net!cs.utexas.edu!news.sprintlink.net!redstone.interpath.net!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Performance on SONAR data
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D65yxJ.AKn@unx.sas.com>
Date: Tue, 28 Mar 1995 18:48:55 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References:  <xY+53eU.predictor@delphi.com>
Organization: SAS Institute Inc.
Lines: 43


In article <xY+53eU.predictor@delphi.com>, Will Dwinnell <predictor@delphi.com> writes:
|> I am curious as to how well other researchers (and their systems) have
|> done modeling the SONAR data.  With BrainMaker Pro, I was fairly
|> successful, and ModelWare did quite well.

I got the sonar data from the CMU archive, where it is divided into a
training set and validation set of 104 cases each. I used only the
training set for all aspects of training, subdividing it further into
actual training and test sets for stopped training. The validation set
was used only for estimating the final error (misclassification) rate.
Since the validation set is so small, error rate estimates are accurate
only to about 10 percentage points based on 95% confidence intervals.
The training criterion was cross-entropy (maximum likelihood for a
Bernoulli distribution).

The first obvious thing to try is logistic regression, which is a
feedforward net with no hidden layer and the usual logistic output
activation function. Since the data are linearly separable, the training
error is zero, but the final solution is not quite unique. With
Levenberg-Marquartd, I got a validation error rate of 24%; with
conjugate gradients, 23%.

There are 60 inputs and 1 binary target. With such a small ratio of
cases to inputs, some sort of regularization or pruning is indicated. I
haven't implemented weight decay (ridging) in my macros yet, so I used
stopped training with the logistic regression model with the 104
training cases divided into 75% actual training cases and 25% test
cases. I ran 5 random divisions and got validation error rates of 21%,
21%, 22%, 24%, and 27%.

I repeated the stopped training with 10 hidden units and got validation
error rates of 17%, 18%, 21%, 22%, and 26%, suggesting that slightly
better generalization is obtained with the hidden units. Whether this
result generalizes to other training/validation splits, I don't know.

OK, Will, so what are "fairly successful" and "quite well"?

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
