Suzy  1.0  -  Radial basis function classifier system.
(Tom Grove, 1993)

(Note:  Turbo  Vision is a registered trademark  of  Borland
International)

Introduction
~~~~~~~~~~~~

Suzy  is  a simple classifier system written in Borland  C++
3.0  and  Turbo Vision 2.0 . It uses radial basis  functions
with    simple   non-linear   step   functions   to    solve
classification problems. It is not a terribly  sophisticated
program (training facts/samples/examples/patterns) must only
belong  to one class, for example)  and is not intended  (as
it  stands) for any serious practical applications. However,
it  is relatively easy to use  (thanks to the interface) and
could  be  of  some use to someone.  I can't guarantee  that
this  program  is bug-free (or even a correct implementation
of   RBF  - though I hope it is!)  but would appreciate  any
comments, advice, etc. I appreciate that many aspects of the
program  are rather poor, and I now have better ideas  about
how  to do things - but .....  there does not seem to be any
RBF PD software available for  PCs, so I thought I'd make  a
contribution. The next thing I write will be better.

Please  mail  comments,  advice,  bug   reports,  etc.   to:

	tgrove@psycho.fme.vutbr.cz

Included files
~~~~~~~~~~~~~~
     suzy.exe - executable
     spirals.dat - two  spirals benchmark problem
     widget.dat  - simple visual pattern recognition
     xor.dat     - xor problem
     demo.rep - example report file (see below)
     suzy.txt - this file
     source.zip  - source code and project file for  Borland
     C++ 3.0 and Turbo Vision


Radial basis functions (RBF)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RBF   nets  are  three  layer  (input-hidden-output)   fully
connected  nets  (as  per  Back propagation).  Unlike  back-
propagation  there is no global learning rule and  only  the
output layer units adapt during training. The weights of the
hidden  layer units are normally fixed when the  network  is
created. These weights define the unit's centre - a point in
n-dimensional space (where n is the number of  input  units,
and  consequently weights, of a unit) which along with  it's
radius  and  output (or kernel) function define how  a  unit
responds to input patterns. A unit's output is calculated by
measuring  the  distance between it's centre and  the  input
pattern  using a suitable metric (normally Euclidean).  This
is  then fed into a suitable transfer function (in the  case
of  Suzy this is currently a non-linear step function) which
should  give  a  high output if the input is  close  to  the
centre and a low (or zero) output if the input falls outside
of  the radius of the unit. The weights of the output  layer
are found using the delta-rule.

Finding  good  centres  for the hidden  units  is  the  most
important task - there seem to be two main approaches:

Prototyping,  where  some (or all) of the  patterns  in  the
training set are used as centres - the radius of these units
being  set  to  some  suitable value based  on  the  average
distance between patterns (or the n nearest neighbours of  a
particular unit).


Clustering,  where  hidden units are  positioned  so  as  to
respond to clusters of inputs.

Suzy  can  use  either of these approaches - the  particular
clustering  algorithm  used is class  based  (clusters  only
contain  members  of  the  same  class)  as  opposed  to   a
clustering  algorithm  that groups patterns  solely  on  the
basis   of   their   geometric  distance.   This   procedure
(HiddenLayer::Init in RBF.CPP)   is quite badly written  and
hard to understand. This is the algorithm followed:


1.  Starting  with no hidden units (H=0) ,  and  all  inputs
unclustered:

     2.  Are there any unclustered inputs?

	  No: clustering complete. End.

     Yes:  create  a new hidden unit  centred on  the  first
     unclustered input. Then ...

	  3. FOR ALL  remaining  unclustered inputs:

	  4.  provisionally  add this input  to  the  hidden
	  unit's   cluster creating a new centre (mean)  for
	  this unit.

	  5. Find the distance of the furthest input in this
	  units  class (D1)

	  6.  Find the distance of the nearest input in  the
	  other  classes(D2)

	  IF  D1>D2 :

		  7.  reject  the  this input, and recover  the  old
		  centre.

	  ELSE:

		  8. Accept the merge.

     9.   Increase the number of hidden units by one (H=H+1)
     and GO TO 2.




Guide to Suzy
~~~~~~~~~~~~~

Suzy  should be relatively intuitive to use. All  operations
are  accessible from the menu bar using either the mouse  or
hot  keys. The list below described the functions associated
with the menu items.

File
     - Open training set
	  load a training/test  set  (*.dat) - see below for
	  format
     - Open Network
	  load a previously saved network (*.net)
     - Save Network
	  save the current network
     - Exit

Suzy
     - Start training
	  start training the current network. A window  will
	  be created to display the current error.
     - Suspend
	  suspend training
     - Network parameters
	  Opens  a  dialog  box  where  the  learning  rate,
	  training   tolerance   and   maximum   number   of
	  iterations  can  be  set.  The  normal/slow  radio
	  buttons  refer  to a feature whereby  the  program
	  constructs  a look-up table for the hidden  units.
	  This is possible in RBF because the weights in the
	  hidden  layer  do  not  change,  so  it  is   only
	  necessary   to calculate once how each  unit  will
	  respond  to  each pattern in the set.  This  costs
	  memory,  though, and may cause problems if  memory
	  is tight. Slow (without using lookup tables) is  ,
	  unfortunately, slow.

	  This  dialog  box may be used during training  (if
	  your  patient) to alter the learning rate  of  the
	  net.  This  is  helpful if you wish to  alter  the
	  learning rate.
     - Init.
	  Create a new network based on the current training
	  set  (a  training  set must  be  loaded  before  a
	  network   can   be  created).  The  initialisation
	  strategy (clustered or prototyped) can be selected
	  using the radio buttons.
     - Test
	  Create  a test report file (*.REP) using the  test
	  data  (if  available). This file has the following
	  format:

	  Test  fact  X  of  class Y classified  as  Z  with
	  certainty C   <YES>/<NO>

	  X  is  the test fact number, Y is X's class. Z  is
	  the  class  assignment given  by  the  program.  C
	  reflects  the  difference in  output  between  the
	  output  unit  which gave the highest response  and
	  the  second highest. The closer to 1 this is,  the
	  better.  <YES>/<NO>  is simply refers  to  whether
	  this    classification   was   correct.    Another
	  possibility  is  <REJECTED> which  indicates  that
	  this test pattern is too far from the training set
	  for   classification   to   be   attempted.   This
	  conclusion  is  reached when none  of  the  hidden
	  units  respond to a test pattern. Some statistical
          information  (no.  rejects, %  correct,  etc.)  is
          added at the end of  the file.
          

Formats
~~~~~~~
Inaddition to *.REP files, Suzy makes use of two other types
of  files:  training  set files (*.DAT)  and  network  files
(*.NET).  The  former contain training and test  information
whilst the latter hold saved networks.

Training set format
~~~~~~~~~~~~~~~~~~~

(see examples XOR.DAT, WIDGETS.DAT and SPIRALS.DAT )

Training sets have the following format:
(order is important)


     size (integer - must be at least one)

     test  (integer - may be 0)

     attributes  (integer  - number of  attributes  in  each
     pattern being classified)
     classes   (integer - number of classes)
     values (number of symbols)

     value (string) = (float) (symbol definitions)
     :
     :

     attribute,attribute   ...........   class     (patterns
     separated by spaces or commas. patterns may
     occupy more than one line)

The  parser  should generate a useful error  message  if  it
fails to parse a training set. Training
sets  may  include comments starting with a ':' (colon)  and
ending with a new line. See the examples for details.

Saved network format
~~~~~~~~~~~~~~~~~~~~

Networks are saved as ascii files in the following format:
(Each item is stored on a separate line)

     no. of  units in hidden layer
     no. of weights/unit
     < weight  0 unit 0>
     < weight  1 unit 0>
     :
     :
     <weight x  of unit y>
     <radius of unit 0>
     :
     <radius of unit y>
     no. of  units in output layer
     no. of   weights/unit
     <weight 0 of unit 0>
     <weight 1 of unit o>
     :
     :
     <weight x of unit  y>
     <bias of unit o>
     :
     <bias of unit y>
     EOF


