Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!news.duke.edu!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: He who knows what he does not know is wise
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <Cz4G9o.MAo@unx.sas.com>
Date: Fri, 11 Nov 1994 21:13:00 GMT
References: <parkCyxFB0.5Ko@netcom.com> <Cyz0MF.Jx3@unx.sas.com> <x05XzZ-.predictor@delphi.com> <Cz0yJq.GFJ@unx.sas.com> <RE610yU.predictor@delphi.com> <3a0142$6vs@lyra.csx.cam.ac.uk>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 32


In article <3a0142$6vs@lyra.csx.cam.ac.uk>, srw1001@wolf.eng.cam.ac.uk (Steve Waterhouse) writes:
|> I presume the sort of thing we're after here is a network that gives confidence measures
|> about it's predictions (regression) or classiifications. There is some work in this area,
|> particularly by MacKay (for classification, via Bayesian frameworks) and Weigand (for predction,
|> via a method similar to
|> the one you suggest - i.e. using an auxillary network to estimate the variance of
|> the prediction).

One more time: the idea of an auxillary network does _NOT_ address the
original question, which was:
|> What are some good ways to get a neural network to report that the inputs
|> you gave it are too different from its training set to permit it to
|> give you an accurate answer?

If a pattern is too far from the training data for the primary net to
generalize well, then it is almost certainly too far from the training
data for the auxillary net to generalize well. The only way around this
is to construct training data for the auxillary net that cover the
entire input space, which is infeasible for more than a few inputs.
Even if you could do that, if you can construct auxillary training data
based on the actual error of prediction, then you should have used those
data to train the primary net. If you can't construct auxillary training
data based on the actual error of prediction, how _do_ you construct
those data?  You are back to the original question, and the argument is
circular.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
