Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!newsfeed.internetmci.com!usenet.eel.ufl.edu!arclight.uoregon.edu!dispatch.news.demon.net!demon!sunsite.doc.ic.ac.uk!warwick!bham!sunserver1.aston.ac.uk!cortex!mclachla
From: mclachla@aston.ac.uk (Alan McLachlan)
Subject: Re: Thin plate spline function
X-Nntp-Posting-Host: cortex.aston.ac.uk
Message-ID: <Dq413s.DJu@aston.ac.uk>
Sender: usenet@aston.ac.uk
Organization:  Neural Computing Research Group, Aston University
X-Newsreader: TIN [version 1.2 PL2]
References: <DprC2z.20v@uns.bris.ac.uk>
Date: Fri, 19 Apr 1996 12:52:40 GMT
Lines: 96

M.J.Ratcliffe@bris.ac.uk wrote:
:>Hi,

:>Several (and there are not many compared to the number of MLP/backprop 
:>refs...) RBF references blandly state that 'The Thin Plate Spline 
:>function does not require a spread constant...'. Why the bloody hell not? 

:>Also does anybody have any references on the differences between using 
:>different RBF functions - Gaussian, TPSF, multi-quadric, teapot, etc. 

:>Ideally these would be neural net-based RBF refs, rather than statistical 
:>theory.

:>Ta,
:>Max.


If you can get a copy, have a look at

"On the Use of Nonlocal and Non Positive Definite Basis Functions in
Radial Basis Function Networks"

David Lowe, 4th IEE International Conference on Artificial Neural Networks,
IEE publications no. 409, 1995, pp 206-211.


A *brief* summary :

1/ Non positive definiteness of the basis functions

The first layer of an RBF attempts to model the unconditional distribution of
the data.

Any kernel-based density estimator constructed out of positive kernels and a 
finite set of samples cannot be unbiased, though the bias decreases 
asymptotically as the sample size increases.

Allowing kernels with negative regions improves the asymptotic convergence
and potentially gives smaller bias (zero bias if you're lucky).

2/ Non locality

Even with unbounded basis functions, the RBF network can give a localised
response (if you view things in terms of an appropriate basis). In the 
particular case where you attempt to model a polynomial using an RBF with a 
regular grid of centres, you *must* have unbounded basis functions.


From the point of view of both bias and universality, unbounded, non-positive
definite basis functions have the potential to outperform everyone's 
favourite (bounded, positive-definite) Gaussian. The thin-plate spine 
(r*r*logr) is a good candidate on both counts.



As for the width parameter, this is a hyperparameter which is not essential
for the purposes of universality in *any* radial basis function, and some 
radial basis function references simply discuss the Gaussian basis function 
exp(-r*r) directly. In the Gaussian case, an adaptable width parameter can 
generally help you get by with fewer kernels (although each kernel is now 
using more parameters), as the width can be increased to extend the domain
of influence of the Gaussian if needed. You can occasionally have numerical 
problems with basis functions collapsing to a point, and unbounded basis
functions gain little benefit from an adjustable width.



Finally, regarding the question of weight-decay in RBF's (mentioned in a 
previous submission), you can do weight decay on the second layer weights
in an RBF network in precisely the same manner as you would for an MLP, as
these weights specify how much use is being made of the network's 
approximation of the input data probability density when attempting to
replicate the targets.

Furthermore, the fact that the RBF output is linear in the weights means 
that the sum-of-squares error is quadratic in the weights (unlike an MLP).
Therefore, if you wish to do a Bayesian treatment a la David MacKay 
(Chris Bishop's book has a chapter on this), you then have the advantage 
that what would be a computationally intensive approximation for an MLP 
becomes a trivial exact procedure for the RBF.


Hope some of this makes sense!


--
Dr Alan McLachlan                |  Tel: (+44/0) 121 359 3621 x4244
Neural Computing Research Group  |  Fax: (+44/0) 121 333 6215
Department of Computer Science   |---------------------------------
Aston University                 |  a.mclachlan@aston.ac.uk
Birmingham B4 7ET, UK            |  http://www.ncrg.aston.ac.uk/





