Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news4.ner.bbnplanet.net!cam-news-hub1.bbnplanet.com!news.mathworks.com!newsfeed.internetmci.com!howland.erols.net!news.sprintlink.net!news-stk-200.sprintlink.net!news.sprintlink.net!news-stk-11.sprintlink.net!interpath!news.interpath.net!sas!newshost.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Apply sigmoid activation fun
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <DwI9wL.73I@unx.sas.com>
Date: Wed, 21 Aug 1996 20:50:45 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References: <4unt8m$31m@hecate.umd.edu> <Pine.SOL.3.91.960815170437.24262A-100000@suma3.reading.ac.uk> <4vcmgv$9r2@hecate.umd.edu>
Organization: SAS Institute Inc.
Lines: 48


In article <4vcmgv$9r2@hecate.umd.edu>, coral@csc.umd.edu (Jian-Zheng Zhou) writes:
|> ... 
|>    My main question is that how do you test a NN after training. 
|> How can we compare the real output values which can be ranged beyond 0 to 1
|> with the calculated values which are only between 0 and 1 due to sigmoid 
|> transformation.

No, no, no. You do NOT use an output activation function with a range
of (0,1) if the correct outputs can be outside that range. There are
three things you can do:

 1. Use an identity ("linear") output activation function (as 
    Jeff Hannan said) or some other unbounded activation function.
 2. If you know upper and lower bounds for the correct outputs,
    scale the target values as (Target-Lowerbound)/(Upperbound-Lowerbound)
    so the correct outputs will be in [0,1].
 3. If you know upper and lower bounds for the correct outputs,
    scale the result of the output activation function as
    Activation*(Upperbound-Lowerbound)+Lowerbound so the range of
    the network output will include the correct outputs.

|> Can we just compare the real output values with the calculated
|> values without sigmoid transformation in the stage of testing NN ?

Absolutely not.

|>   I have no problem for using sigmoid or any other kind of transformations in
|> training  NN.  I know the sigmoid transformation at least is a better way to
|> control overflow during the training. 

Overflow is rarely a problem with any good training algorithm. Standard
backprop is so slow it would take forever to overflow unless you used
a ridiculously high learning rate.

Use of a bounded output activation function does avoid the danger of
the outputs taking values far outside the desired range. While preventing
wild outputs is necessary for getting good generalization, it is far
from sufficient, and there are a variety of methods for improving
generalization that usually avoid wild outputs (although perhaps not
100% of the time, as Bill Armstrong would be quick to point out while
extolling the virtues of ALNs).

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
