Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!pipex!uunet!zib-berlin.de!news.th-darmstadt.de!fauern!rrze.uni-erlangen.de!faulrs!msn!smurf.noris.de!hugis!venus!alex
From: alex@venus.nbg.sub.org (Alexander Adolf)
Subject: Re: Degrees of freedom in a net
Message-ID: <5Z3UB71C@venus.nbg.sub.org>
Organization: Nuernberg, Germany
X-Newsreader: TIN [version 1.2 PL2]
References: <Cz294r.LJt@cs.dal.ca> <9QJUBJCC@venus.nbg.sub.org> <CzMs0o.K39@unx.sas.com> <X4UUB7DB@venus.nbg.sub.org> <D03Cyz.C8D@unx.sas.com>
Date: Mon, 5 Dec 1994 23:27:42 GMT
Lines: 50

Warren Sarle (saswss@hotellng.unx.sas.com) wrote:

: In article <X4UUB7DB@venus.nbg.sub.org>, alex@venus.nbg.sub.org (Alexander Adolf) writes:
: |> ...
: |> Though I still don't have any idea what information I could obtain
: |> from a result as "the degree of freedom for this net is 5". Do you
: |> have any idea wether this could be useful?

: "Degrees of freedom" (DF) is a technical term in statistics that relates
: to many important issues such as estimating the noise variance,
: estimating generalization error, and selecting model complexity.  In a
: linear model, the total number of cases N is broken down into model DF,
: which equals the number of parameters to be estimated, say P, and error
: DF, which is N-P. If the sum of squared errors is SSE, then the unbiased
: estimate of error variance is MSE=SSE/(N-P). Under certain assumptions,
: the unbiased estimate of generalization error is given by
: SSE*(N+P)/(N-P)*N, which goes by various names inclusing Final
: Prediction Error. DF appear in many other statistical formulas as well.

I had these in mind but didn't dare to apply them to a highly
nonlinear system. Neither did I dare to assign any meaning to them
when being applied to a hihgly nonlinear system.


: In a nonlinear model, the above formulas may hold approximately, but
: there is no known exact formula for DF, and different uses, such as MSE
: and FPE, may require different formulas. One approach to computing DF in
: neural networks for a different method of estimating generalization
: error is given in Moody, J.E. (1992), "The Effective Number of
: Parameters: An Analysis of Generalization and Regularization in
: Nonlinear Learning Systems", NIPS 4, 847-854.

Hm. Propably should have a look at that. But with all the
different architectures (means combinations of topology, neuron model
and learning algorithm), could we expect such a computation to yield
generally usable results?


: Apparently, Alexander means something different by "degrees of freedom".
: Is there some accepted definition of the term in neural netese?

Curious...

  -- Alexander Adolf

-- 
                                              #include <std-disclaimer.h>
Alexander Adolf ---------------------------------- alex@venus.nbg.sub.org
Georg-Simon-Ohm Polytech.Univ. Nuernberg/FRG --- Department of Electrical
Engineering -------- Computer Science and Information Technology Division
