Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!bloom-beacon.mit.edu!uhog.mit.edu!news.mathworks.com!news.duke.edu!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Degrees of freedom in a net
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <Cz3F73.HBq@unx.sas.com>
Date: Fri, 11 Nov 1994 07:52:15 GMT
References:  <Cz294r.LJt@cs.dal.ca>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 40


In article <Cz294r.LJt@cs.dal.ca>, ab340@cfn.cs.dal.ca (John Shimeld) writes:
|> I'm wondering how to determine the number of degrees of freedom
|> in a neural network. ... In my simple minded manner, I would
|> say that the number of degrees of freedom is equal to the total number
|> of weights in the net. ..
|>
|> A problem is that each weight is really not an independent parameter.
|> Is it feasibly possible, then, to determine f? 

It depends on what you want to do with the "degrees of freedom". There
is no single generalization of the concept of "degrees of freedom" in
a linear model to the nonlinear case. If you are interested in
estimating prediction error, see:

   Moody, J.E. (1992), "The Effective Number of Parameters: An Analysis
   of Generalization and Regularization in Nonlinear Learning Systems",
   NIPS 4, 847-854.

But Moody's "Effective Number of Parameters" does not necessarily
apply for other purposes such as hypothesis testing. In fact, if you
want to test the hypothesis that a certain hidden node is superfluous,
the usual nonlinear theory breaks down because if the input-to-hidden
weights are zero, the hidden-to-output weights are indeterminate and
vice versa. See:

   Kuan, C.-M. and White, H. (1994), "Artificial Neural Networks: An
   Econometric Perspective", Econometric Reviews, 13, 1-91.

However, for some practical applications, you can get reasonably
accurate results by taking the number of weights to be the "degrees of
freedom" if the number of training cases is at least twice as large as
the number of weights. But the only way I know of to be sure of that
is to run simulations and see.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
