Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!scramble.lm.com!news.math.psu.edu!news.cse.psu.edu!uwm.edu!math.ohio-state.edu!howland.reston.ans.net!swrinde!newsfeed.internetmci.com!in2.uu.net!news.interpath.net!sas!newshost.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: y (approximation) is close to t (target) ?
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <DsI12D.7nF@unx.sas.com>
Date: Tue, 4 Jun 1996 23:25:25 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References:  <DsCC04.C3u@zeno.fit.edu>
Organization: SAS Institute Inc.
Lines: 69


In article <DsCC04.C3u@zeno.fit.edu>, ram82001@zach.fit.edu (Gary Russel /ADVISOR L. FAUSETT) writes:
|> A neural net of the form y = w(1)*z(1) + ... + w(4)*z(4),
|> where z(j) = z(v(j)) = exp(-v(j))           
|>            -(x(1) - c(1,j))^2 - (x(2) - c(2,j))^2
|> and v(j) = -------------------------------------- ,
|>                                  2 
|>                            2*s(j)
|> 
|> has approximated t = exp(-(x(1) - .25)^2 - (x(2) - .75)^2)
|>                    + exp(-(x(1) - .75)^2 - (x(2) - .25)^2).
|> 
|> Of course, y could equal z exactly if
|> w(1) = 1, w(2) = 1, w(3) = 0, w(4) = 0,
|> s(1) = sqrt(0.5), s(2) = sqrt(0.5), s(3) <> 0, s(4) <> 0,
|> c(1,1) = .25, c(1,2) = .75, c(1,3) = ?, c(1,4) = ?,
|> c(2,1) = .75, c(2,2) = .25, c(2,3) = ?, c(2,4) = ?
|> 
|> But here is the set of weights y learned:
|> w(1) = -.4493, w(2) = .9760, w(3) = 1.0480, w(4) = -.3181,
|> s(1) = .5537, s(2) = .6952, s(3) = .8496, s(4) = .0445,
|> c(1,1) = .9595, c(1,2) = .5839, c(1,3) = .6049, c(1,4) = -.1189,
|> c(2,1) = .9875, c(2,2) = .5858, c(2,3) = .6161, c(2,4) = 1.0027 
|> 
|> Numerically, one can show how close y is to t
|> (e.g. t(0,0) = 1.0705, y(0,0) = 1.0814),

The two Gaussians in the target function are so close to each other that
the problem is ill conditioned (fitting two Gaussians, the condition
number of the Hessian at the optimum is nearly 1000). As Bill Armstrong
has mentioned before, the usual neural net algorithms cannot compute
accurate weights for ill-conditioned problems (most people don't seem to
care how accurate the weights are). As Greg Heath pointed out, fitting 4
Gaussians to a function consisting of two Gaussians only makes things
worse.

|> Must one investigate optimizing (t - y)^2 ?

If you want to get accurate weights, you need to use a numerically
stable optimization algorithm. Using the NLP procedure in the SAS/OR
product, I get results similar to these in a few iterations with any of
a variety of algorithms:

          Two Gaussians                   Four Gaussians

     Parameter        Estimate       Parameter          Estimate  
  -----------------------------   --------------------------------
     1  C11           0.750000       1   C11            1.139590  
     2  C12           0.250000       2   C21           -0.199607  
     3  C21           0.250000       3   C12            0.250000  
     4  C22           0.750000       4   C22            0.750000  
     5  W1            1.000000       5   C13            0.405166  
     6  W2            1.000000       6   C23            0.488305  
                                     7   C14            0.750000  
                                     8   C24            0.250000  
                                     9   W1         2.988946E-13  
                                    10   W2             1.000000  
                                    11   W3         6.827258E-14  
                                    12   W4             1.000000  

You should also be able to get similar results with any neural net
software using a good implementation of Levenberg-Marquardt, conjugate
gradients, etc.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
