Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!nntp.club.cc.cmu.edu!news.duq.edu!newsgate.duke.edu!news.mathworks.com!EU.net!CERN.ch!news
From: john nigel gamble <gamble@dxcoms.cern.ch>
Subject: Re: Beginners questions
X-Nntp-Posting-Host: dxcoms.cern.ch
Content-Type: text/plain; charset=us-ascii
To: "Stefan C. Kremer" <stefan.kremer@crc.doc.ca>
Message-ID: <329C1714.167E@dxcoms.cern.ch>
Sender: news@news.cern.ch (USENET News System)
Content-Transfer-Encoding: 7bit
Organization: CERN European Lab for Particle Physics
References: <32998DF2.41C6@dxcoms.cern.ch> <57d1eo$pik@crc-news.doc.ca> <329AE2A8.41C6@dxcoms.cern.ch> <57euvb$fgt@crc-news.doc.ca>
Mime-Version: 1.0
Date: Wed, 27 Nov 1996 10:25:24 GMT
X-Mailer: Mozilla 3.01Gold (X11; I; OSF1 V3.2 alpha)
Lines: 65

Stefan C. Kremer wrote:
> 
> In article <329AE2A8.41C6@dxcoms.cern.ch>, gamble@dxcoms.cern.ch says...

Big Snip!

> >2). How close does the analogy hold. For example In the kangaroo's
> >    land I understood weights to be used as if they were equivalent
> >    to the axis. North-South is orthogonal to East-West.
> >    But is this true in "weight-space" .. is this a stupid question?
> >    I have difficulty picturing it, but maybe the number of "axis" (in
> >    an n-dimentional coordinate system) that represents weight space
> >    is not the same as the number of weights.
> 
> I think the analogy holds at this level.  Each weight (and bias) in
> the network represents an orthogonal direction in weight space (since
> each weight can be varied independently of every other).  The error
> value for a particular point in weight space adds one more dimension.
> I.e. if there are two weights in your network (a one-input, one-output,
> one-weight, one-bias network), then the two weights dimensions can be
> thought of as NS and EW (in a flat-earth sense).  The error function
> of the network can be represented as a third dimesion--altitude.
> 

I am still intrigued by the mapping of the weights themselves into the
"problem" space of the NN.  As I understand things (don't forget I'm a
beginner), one  tries  to  solve  a  problem  in  the  real  world  by
"inventing"  a  certain  NN  topology.  There  can  be many topologies
invented to solve the same problem, they will all have the same inputs
and outputs .. but internally the networks can be different.

It isn't obvious to me that weights really represent  orthogonal  axis
of  the problem space (you can add nodes to hidden layers .. its still
the same problem).

Imagine a two dimensional space.  In a given  Network  topology  maybe
changing  one of the weights moves you along a 45 degree line from the
"real" axis and the other a 5 degree line.

Suppose this was the case, then maybe the "optimum" NN topology is one
where the weights do represent orthogonal axis.

Let me try to visualise this.  The Xor problem cannot be solved  until
a  hidden  layer  is introduced.  Without the hidden layer the weights
are only capable of mapping  a  problem  space  with  some  dimensions
missing.  One  node  in the hidden layer doesn't change this, although
you have actually incresed the number of "axis" that  you  can  "use".
Two  nodes in the hidden layer provide the required "dimensionality" -
maybe this is the optimum orthogonal case.  Three nodes in the  hidden
layer  also  works (doesn't it? anyone tried?) - but the third node is
introducing extra weight axis that are maybe not orthogonal.

Has anyone thought about this?  Maybe I should keep  quiet!!!  and  go
and study maths.

The "model" has interesting consequencies ..  Too  many  weights  axis
would tend to make the "landscape" (back to kangaroos) rougher - since
you could end up in the same "problem"  place  with  different  weight
coordinates.

Would there be any way of "testing" this?  Like being able to tell  if
weight axis are orthogonal or not? and adjust the toplogy accordingly.


John (a curious beginner).
