Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!news.mathworks.com!news.kei.com!news.texas.net!news.sprintlink.net!news-fw-6.sprintlink.net!interpath!news.interpath.net!sas!newshost.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Haykin's RBF Chapter
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <Dws056.Bsn@unx.sas.com>
Date: Tue, 27 Aug 1996 02:55:53 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References: <Pine.SUN.3.91.960820132749.21955A-100000@soma.med.utah.edu> <DwIAAJ.7nv@unx.sas.com> <321DD517.41C67EA6@stats.ox.ac.uk>
Organization: SAS Institute Inc.
Lines: 72


In article <321DD517.41C67EA6@stats.ox.ac.uk>, Prof Brian Ripley <ripley@stats.ox.ac.uk> writes:
|> ...
|> Perhaps Warren Sarle should declare that the FAQs he is recommending are
|> his own account (and opinion).  I think there is much more (and many
|> more
|> references) to the questions of normalization and inference for RBFs
|> that are discussed there.  

If there are any papers besides the two mentioned in the FAQ that
compare ordinary and normalized RBF networks, I would very much like
to know about it so I can add them to the FAQ. Prof Ripley's excellent
book says only that normalized RBFs are "sometimes considered". Most
articles on RBF networks omit any prominent mention of which type of
RBF is being used, and you often have to look at the formulas to tell.

I am not at all surprised by people accusing me of bias towards
statistical methodology, or against books that neglect generalization,
but I hardly expected my discussion of RBF networks to be
controversial!  Slightly unorthodox, yes, but the prevailing
terminology is imprecise. The FAQ does rather neglect non-Gaussian
RBFs, but I rarely see anybody using non-Gaussian RBFs in the neural
net literature.

|> I am a purist, and use RBFs in the sense that
|> the term is used in numerical approximation (and where Broomhead and
|> Lowe
|> got it from).

If you use "RBF network" to mean the form used in numerical
approximation, and "normalized RBF network" to mean the normalized
from, how do you refer to the union of the two?

|> RBFs are a class of functions, and can be fitted by
|> least-squares, as a log-linear model or as a normalized linear model
|> (what Sarle calls softmax, although it is softmax on a log
|> transformation
|> of an RBF, not softmax as defined by its author, Bridle.

Bridle's formula (p. 213 in NIPS 2) takes a vector of input values
from one training case as its argument. I treat softmax as an
activation function. But Bridle says that softmax is a "multi-input
generalisation of the logistic" so he also seems to be thinking of
softmax as an activation function. And what the FAQ describes as
softmax certainly is a "multi-input generalisation of the logistic".
So I don't see any conflict between Bridle's treatment and my
treatment of softmax.

|> His notation is
|> also differs from the original in working with squared distances not
|> distances, so
|> Gaussians become `exponential' `activation' functions.)

Is this a problem?  It seemed a very convenient rearrangement of the
algebra to me.

|> Another
|> groups of authors in the NN community use RBF (and sometimes also PNN)
|> as a name for fitting mixtures of Gaussian densities: these end up
|> modelling
|> posterior probabilities by normalized RBFs, BUT with a different fitting
|> algorithm.  (I do think Bishop is rather misleading over this.)

Yes, a very different fitting algorithm. I don't recall this issue
being raised in the newsgroup before, so it doesn't seem like a prime
candidate for the FAQ.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
