Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!nntp.club.cc.cmu.edu!news.duq.edu!newsfeed.pitt.edu!portc02.blue.aol.com!howland.erols.net!EU.net!CERN.ch!news
From: john nigel gamble <gamble@dxcoms.cern.ch>
Subject: Re: Beginners questions
X-Nntp-Posting-Host: dxcoms.cern.ch
Content-Type: text/plain; charset=us-ascii
To: "Stefan C. Kremer" <stefan.kremer@crc.doc.ca>
Message-ID: <329AE2A8.41C6@dxcoms.cern.ch>
Sender: news@news.cern.ch (USENET News System)
Content-Transfer-Encoding: 7bit
Organization: CERN European Lab for Particle Physics
References: <32998DF2.41C6@dxcoms.cern.ch> <57d1eo$pik@crc-news.doc.ca>
Mime-Version: 1.0
Date: Tue, 26 Nov 1996 12:29:28 GMT
X-Mailer: Mozilla 3.01Gold (X11; I; OSF1 V3.2 alpha)
Lines: 149

Thanks Stefan for taking the time to reply. Much appreciated.

The kangaroo analogy is very descriptive, but yes, its upside down.
So I will continue with  a kangaroo looking for deep water-holes. I
think someone already suggested a frog in the FAQ.

>>For
>>example, from the peak can I map out the  surrounding valleys?

>How would you or the kangaroo do that.  Keep in mind that the only
>information you have to go on is the slope of the surface at the points
>you have visited.  You can't see any distance (even from the top of a
>local minimum, you're jumping around in the dark.

Being a programmer rather than mathematician, (with regret) I
considered that having found a (peak) valley I have a set of
coordinates in weight space that is the local minimum. I could simply
"hop" along each of the coordinate axis in turn till I started going
up-hill. (e.g in two dimensions go -North, South, East and West from
the minimum). This is just feed-forward (isn't it?).

Now, if the analogy holds, the further I go the more ground I am likely
to miss (the area between the axis) - so maybe I shouldn't go too far.

There again, by looking at the local surface gradient as I hop I can
build up some confidence concerning the local terrain. (I don't see
how to do this easily).
If the optimum is really an isolated steep-sided hole in a hill, then
it is going to be difficult for any step-wise approximating algorithm
to find it.

>Of course, there is the possibility that "one side" of the mountain may
>extend to infinity, dropping in altitude as one or more weights are
>made larger (or smaller).  I suspect this case would be very prevalent
>for binary output problems (and maybe continous ones as well).

But thats OK isn't it. I was trying to eliminate areas (volumes is
perhaps more correct) of weight space that I don't need to look at.

>arbitrary contour and even contain holes).  If my supposition about
>arbitrarily complex regions is true, then in order to be sure that there
>are no other mountains hiding in what you think might be the region of
>your current mountain.  You would have to actually visit every point
>in the hypothesised region.  If the weight space is continuous then
>the problem of identifying the region of a mountain is impossible.

I agree (I think) to this as a generalised mathematical statement, but
doesn't this observation apply to all step-wise algorithms?
If we are looking for a minimum, and this happens to be the crater of
a volcano, then I don't see how any gradient based algorithm can
find it - without an exhaustive search - or good luck!.

I have (at least) two subsequent questions

1). what other algorithms exist that don't "follow" the surface?
2). How close does the analogy hold. For example In the kangaroo's
    land I understood weights to be used as if they were equivalent
    to the axis. North-South is orthogonal to East-West.
    But is this true in "weight-space" .. is this a stupid question?
    I have difficulty picturing it, but maybe the number of "axis" (in
    an n-dimentional coordinate system) that represents weight space
    is not the same as the number of weights.
3). I guess all this also depends on the training set being complete.
    Do you have an analogy for the landscape when the training set is
    incomplete?

Out of my depth .... John.

P.S. Thanks again for your time.

============================================= original mail
Stefan C. Kremer wrote:
> 
> In article <32998DF2.41C6@dxcoms.cern.ch>, gamble@dxcoms.cern.ch says...
> >
> >Hi,
> >
> > I am a beginner in Neural Networks - its become a hobby. After learning
> >the  hard  way about the effects of  the  initial choice of weights I
> >discovered the superb description of the kangaroo in the NN  FAQ.
> 
> Its nice to see someone read the FAQ first, and then post!
> 
> >Of
> >course  this  just stimulated many more questions.  Taking the kangaroo
> >analogy, I parachute my kangaroo onto the landscape and he/she finds a
> >local mountain. There is clearly? a region where all landings of my
> >kangaroo will find the same mountain.
> 
> I prefer to think of kangaroos jumping downhill in a landscape where
> altitude equates to error.  Here the kangaroos seek out valleys which
> represent solutions to the problem you are trying to solve.  There
> is indeed a region where kangaroo landings find the same valley.
> This is called a "basin of attraction".  I don't know what you'ld call
> it in the up-hill version of the metaphor.  The uphill metaphor
> also doesn't make much sense when talking about local minima (which
> I think is what you're getting at with all of this discussion).
> Besides, don't kangaroos like flat, low-lying areas (instead of high
> peaks)?  Anyways, I'll try to stick with the original terminology
> for the remainder of this discussion.
> 
> >Is there any simple algorithm that
> >maps out this region so that I know not to parachute there again?
> 
> Not that I know of, nor do I think there could be (see below).
> 
> >For
> >example, from the peak can I map out the  surrounding valleys?
> 
> How would you or the kangaroo do that.  Keep in mind that the only
> information you have to go on is the slope of the surface at the points
> you have visited.  You can't see any distance (even from the top of a
> local minimum, you're jumping around in the dark.
> 
> >Does the
> >analogy  hold  ..  i.e.  will a set of weight ranges map the boundaries
> >(contour) of a local mountain? or is the math more complex than this?.
> 
> The contour of a local mountain could be described by an equation whose
> variables are the network weights and whose solutions represent points
> on the contour.
> 
> E.g.:  consider a nice circular mountain in a 2-dimensional weight space
> whose contour might be described by an equation like w1^2+w2^2=1.
> 
> The region of the mountain can then be described by a similar inequality.
> 
> Of course, there is the possibility that "one side" of the mountain may
> extend to infinity, dropping in altitude as one or more weights are
> made larger (or smaller).  I suspect this case would be very prevalent
> for binary output problems (and maybe continous ones as well).
> 
> The real problem is that I believe the region from which kangaroos can
> climb to the top of the same mountain can be arbitrarily
> complex depending on the problem presented.  E.g.  if you give me any
> connected region in a weight space, there exists some problem for which
> there is a mountain with that region (i.e. the region can have an
> arbitrary contour and even contain holes).  If my supposition about
> arbitrarily complex regions is true, then in order to be sure that there
> are no other mountains hiding in what you think might be the region of
> your current mountain.  You would have to actually visit every point
> in the hypothesised region.  If the weight space is continuous then
> the problem of identifying the region of a mountain is impossible.
> 
> Hope that made sense.
> 
> >John. A curious beginner.
> 
>         -Stefan.  A curious veteran.
