Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!newsfeed.internetmci.com!howland.reston.ans.net!psinntp!psinntp!psinntp!psinntp!megatest!news
From: Dave Jones <djones>
Subject: Re: THRESHOLD FUNCTION?
Content-Type: text/plain; charset=us-ascii
Message-ID: <DMoq9D.10y@Megatest.COM>
Sender: news@Megatest.COM (News Admin)
Nntp-Posting-Host: pluto
Content-Transfer-Encoding: 7bit
Organization: Megatest Corporation
References: <8BA44EB.01CE0053F1.uuout@almac.co.uk>
Mime-Version: 1.0
Date: Mon, 12 Feb 1996 22:53:37 GMT
X-Mailer: Mozilla 1.1N (X11; I; SunOS 5.4 sun4m)
X-Url: news:8BA44EB.01CE0053F1.uuout@almac.co.uk
Lines: 46

neil.slater@almac.co.uk (NEIL SLATER) wrote:
>I have a simple (I hope) question about Threshold Functions.
>
>I am hoping to write my own simple library of NN objects. I don't 
>understand the derivation of the Backpropagation formulae, and I 
>wondered how tied they were to the sigmoid function:
>
>f(x) = 1 / (1 + exp(-x))
>
>Is it just the general shape of the function that is required in BP 
>networks, or does the value need to be known accurately?

I'm puzzling this stuff out also.

Here's what little I know about it. The function above is called the
"logistic" function. Any function with that general shape will do as
a sigmoid activation. How is it tied to the backpropogation algorithm?
Well, several of the most common backpropagation algorithms will need to know
the derivative of it. Some regression algorithms also need its inverse. The
derivative of the logistic function (and of tanh, which is the same thing
squished and tugged a bit) has a particularly convenient form.

>
>For speed, I was hoping to use a relatively small lookup table with 
>linear interpolation to make the function continuous. Does this make 
>sense?

Yes. I am using a lookup table with 256 points (plus one more for all
extreme values), which codes for one side of the (approximate) logistic
function, ranging from 0 to 16.0. Thus the x-values of the sample points are
exactly representable in the machine. [Actually I use two tables, one for the
sigmoid, another for differences between points.] The derivatives and inverses
are therefore not exact, but it doesn't seem to matter. Things are seldom exact
with floating point arithmetic in any case.

> Would I get any mileage from trying a half-sinusoid function 
>like:
>
>f(x) = (1 + sin(x/c)), =0 with x < -2c/pi, =1 with x > 2c/pi.
>

I don't think so.


            Dave

