Newsgroups: comp.ai.edu,comp.ai.fuzzy,comp.ai.neural-nets,comp.ai
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!fas-news.harvard.edu!newspump.wustl.edu!news.ecn.bgu.edu!movietone.ils.nwu.edu!newsfeed.acns.nwu.edu!math.ohio-state.edu!howland.reston.ans.net!newsfeed.internetmci.com!news.netrail.net!barney.gvi.net!redstone.interpath.net!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Help: "cross validation"
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <DJJCGM.LGo@unx.sas.com>
Date: Wed, 13 Dec 1995 17:26:46 GMT
Distribution: inet
X-Nntp-Posting-Host: hotellng.unx.sas.com
References:  <Pine.PMDF.3.91.951206175308.1076091808A-100000@vms.ucc.okstate.edu>
Organization: SAS Institute Inc.
Lines: 79
Xref: glinda.oz.cs.cmu.edu comp.ai.edu:3045 comp.ai.fuzzy:6282 comp.ai.neural-nets:28670 comp.ai:35325


In article <Pine.PMDF.3.91.951206175308.1076091808A-100000@vms.ucc.okstate.edu>, lim@okstate.edu writes:
|>
|>      What is "cross validation", and "10-fold cross validation"
|> when dealing with data sets especially on machine learning
|> algorithms? Any pointers to books, homepages, advices, or explanations
|> are welcome. Thanks for all the help.

Cross-validation and related methods for estimating generalization
error are described in the comp.ai.neural-nets rerun below.

Newsgroups: comp.ai.neural-nets
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Small training sets
Date: Tue, 10 Oct 95 02:07:42 EDT

In article <44se0c$rre@usc.edu>, "J. Michael Tyszka, Ph.D." <tyszkaj@csmc.edu> writes:
|> ...
|> If you have a training set of say only 30 patterns and you want to make
|> maximum use of its training potential, while determining as much as
|> possible about the classifying power of the trained network, can you
|> pick one pattern as a test, train the net with the remaining 29
|> patterns, then repeat for the next pattern. This would lead to a total
|> of 30 tests of a net trained with 29 patterns each time. What about
|> dividing the training set into two halves, training with one half,
|> testing with the other, then swapping for two tests of 15 patterns on a
|> network trained on 15 patterns each time.

Both of those methods are examples of cross-validation. In k-fold
cross-validation, you divide the data into k subsets of equal size.
You train the net k times, each time leaving out one of the subsets
from training but using only the omitted subset to compute whatever
error criterion interests you. If k equals the sample size, this is
called leave-one-out cross-validation. A more elaborate and expensive
version of cross-validation involves leaving out all possible subsets
of a given size.

Note that cross-validation is quite different from the "split-sample"
or "holdout-sample" method that is commonly used for stopped training
in neural nets. In the split-sample method, only a single subset is
used to estimate the error function, instead of k different subsets.
While various people have suggested that cross-validation be applied
to stopped training, the proper way of doing that is not at all
obvious

|> Which if any of these approaches gives the best information about the
|> classifying power of the network. Are the results dependent in some way
|> ??

Leave-one-out cross-validation often works well for continuous error
functions such as the mean squared error, but it may perform poorly
for noncontinuous error functions such as the number of misclassified
cases. In the latter case, k-fold cross-validation is preferred. But
if k gets too small, the error estimate is pessimistically biased
because of the difference in sample size between the full-sample
analysis and the cross-validation analyses.

Bootstrapping generally works better than cross-validation. In the
simplest form of bootstrapping, instead of repeatedly analyzing
subsets of the data, you repeatedly analyze subsamples of the data.
Each subsample is a random sample with replacement from the full
sample. Depending on what you want to do, anywhere from 200 to 2000
subsamples might be used. For details, see (in order of increasing
complexity):

   Weiss, S.M. & Kulikowski, C.A. (1991), _Computer Systems That
   Learn_, Morgan Kaufmann.

   Efron, B. and Tibshirani, R.J. (1993), _An Introduction to the
   Bootstrap_, London: Chapman & Hall.

   Hjorth, J.S.U. (1994), _Computer Intensive Statistical Methods_:
   Validation, Model Selection, and Bootstrap_, London: Chapman & Hall.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
