Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!gatech!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Multicollinearity and outliers
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D0o1vr.J7o@unx.sas.com>
Date: Sun, 11 Dec 1994 21:47:51 GMT
References:  <3cdmm6$70c@celebrian.otago.ac.nz>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 36


In article <3cdmm6$70c@celebrian.otago.ac.nz>, argray@rivendell.otago.ac.nz writes:
|>      I recently posted to this group regarding the effect of
|> multicollinearity in training data on neural networks, specifically
|> backpropagation trained networks.  ...  I would also be interested in the
|> effects of multiple outliers in the dataset on the trained network.

For neural nets that are linear models, such as some RBF nets, functional
link nets, higher-order nets, etc., see any textbook on linear
regression such as:

   Sanford Weisberg (1985), _Applied Linear Regression_, NY: Wiley

   Raymond H. Myers (1986), _Classical and Modern Regression with
      Applications_, Boston: Duxbury Press

In nonlinear feedforward nets, the same general principles apply as with
linear models except that in all the formulas, you replace the matrix of
inputs by the Jacobian matrix of partial derivatives of the outputs with
respect to the weights.

The effect of outliers on nonlinear feedforward nets varies quite a bit.
If there are lots of hidden units, the net will probably just learn the
outliers, with little effect on the rest of the training space. If you
have just enough hidden units to learn the good data, then the outliers
may distort the general fit of the net, or they may usurp some hidden
units and cause underfitting of the rest of the data. You can use robust
estimation criteria for training neural nets just as you can in linear
regression, but I am not aware of any sound studies on how effective
such methods are for neural nets.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
