Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!pipex!uunet!zib-berlin.de!prise.nz.dlr.de!news.dfn.de!zeus.rbi.informatik.uni-frankfurt.de!jurpool0.rz.uni-frankfurt.de!terra.wiwi.uni-frankfurt.de!news.th-darmstadt.de!fauern!rrze.uni-erlangen.de!faulrs!msn!smurf.noris.de!hugis!venus!alex
From: alex@venus.nbg.sub.org (Alexander Adolf)
Subject: Re: Degrees of freedom in a net
Message-ID: <9QJUBJCC@venus.nbg.sub.org>
Organization: Nuernberg, Germany
X-Newsreader: TIN [version 1.2 PL2]
References: <Cz294r.LJt@cs.dal.ca>
Date: Fri, 18 Nov 1994 15:42:57 GMT
Lines: 43

John Shimeld (ab340@cfn.cs.dal.ca) wrote:


: I'm wondering how to determine the number of degrees of freedom
: in a neural network.  Consider a simple backpropagation net with 
: three inputs, a single hidden layer of four nodes, and a one output.
: The input layer node is connected to each hidden layer node, and 
: each hidden layer node is connected to the output.  The hidden layer
: nodes are not interconnected.  In my simple minded manner, I would 
: say that the number of degrees of freedom is equal to the total number 
: of weights in the net. That is, f = 3x4+4 = 16. 

The (most?) general definition propably is the one that states that
the degree of freedom is the number of state parameters os the system
that can be altered simultaneously wihtout each of them influencing
any other parameter of the system. Right?


: A problem is that each weight is really not an independent parameter.

You seem to have had the same idea. And... hit the problem already!
Obviously that depends mostly on the topology of the network. In a
fully coonected pure feed-forward net, I'd suggest that the degree of
freedom is the number of links between the input layer and the first
hidden layer. All other links won't count because the're influenced by
the input layer's links. For different topologies (i.e. feedbacks) you
have to completely reconsider that. Another point is the kind of
node-model you use. There are models that comprise excitation of the
closest neighbours (even without links to them) upon an activation
limit, etc. Starting from our above definition of f one could propably
come up with some reasonable way of determining f for a give
topology/node-model combination but I doubt that it would be of much
information value.



  -- Alexander Adolf

-- 
                                              #include <std-disclaimer.h>
Alexander Adolf ---------------------------------- alex@venus.nbg.sub.org
Georg-Simon-Ohm Polytech.Univ. Nuernberg/FRG --- Department of Electrical
Engineering -------- Computer Science and Information Technology Division
