
Genetic Algorithms Digest   Friday, May 29 1992   Volume 6 : Issue 18

 - Send submissions to GA-List@AIC.NRL.NAVY.MIL
 - Send administrative requests to GA-List-Request@AIC.NRL.NAVY.MIL
 - anonymous ftp archive: FTP.AIC.NRL.NAVY.MIL (see v6n5 for details)

Today's Topics:
	- Paradigmatic over-fitting
	- Two new GA papers available
	- Genetic Neural Network
	- GA research centres, need information

**********************************************************************

CALENDAR OF GA-RELATED ACTIVITIES: (with GA-List issue reference)

 COGANN, Combinations of GAs and NNs, @ IJCNN-92 (v5n31)      Jun 6,     1992
 ARTIFICIAL LIFE III, Santa Fe, NM                            Jun 15-19, 1992
 Evolution as a computational process, Monterey (v6n9)        Jun 22-24, 1992
 ML-92, Machine Learning Conference, Aberdeen (v6n8)          Jul  1-3,  1992
 10th National Conference on AI, San Jose,                    Jul 12-17, 1992
 FOGA-92, Foundations of Genetic Algorithms, Colorado (v5n32) Jul 26-29, 1992
 COG SCI 92, Cognitive Science Conference, Indiana, (v5n39)   Jul 29-1,  1992
 ECAI 92, 10th European Conference on AI (v5n13)              Aug  3-7,  1992
 Parallel Problem Solving from Nature, Brussels, (v5n29)      Sep 28-30, 1992
 SAB92, From Animals to Animats, Honolulu (v6n6)              Dec  7-11, 1992

 (Send announcements of other activities to GA-List@aic.nrl.navy.mil)

**********************************************************************
----------------------------------------------------------------------

From: belew%FRULM63.BITNET@pucc.Princeton.EDU (Rick BELEW)
Date: Fri, 15 May 92 14:55:04 +0200
Subject: Paradigmatic over-fitting

   There has been a great deal of discussion in the GA community concerning
   the use of particular functions as a "test suite" against which different
   methods (e.g., of performing cross-over) might be compared.  The GA is
   perhaps particularly well-suited to this mode of analysis, given the way
   arbitrary "evaluation" functions are neatly cleaved from characteristics
   of the algorithm itself.  We have argued before that dichotomizing the
   GA/evaluation function relationship in this fashion is inapprorpiate [W.
   Hart & R. K. Belew, ICGA'91].  This note, however, is intended to focus on
   the general use of test sets, in any fashion.

   Ken DeJong set an ambitious precident with his thesis [K. DeJong, U.
   Michigan, 1975].  As part of a careful empirical investigation of several
   major dimensions of the GA, DeJong identified a set of five functions that
   seemed to him (at that time) to provide a wide variety of test conditions
   affecting the algorithm's performance.  Since then, the resulting "De Jong
   functions F1-F5" have assumed almost mythic status, and continue to be one
   of the primary ways in which new GA techniques are evaluated.

   Within the last several years, a number of researchers have re-evaluated
   the DeJong test suite and found it wanting in one respect or another.
   Some have felt that it does not provide a real test for cross-over, some
   that the set does not accurately characterize the general space of
   evaluation functions, and some that naturally occuring problems provide a
   much better evaluation than anything synthesized on theoretical grounds.
   There is merit to each of these criticisms, and all of this discussion has
   furthered our understanding of the GA.

   But it seems to me that somewhere along the line the original DeJong suite
   became villified.  It isn't just that "hind-sight is always 20-20."  I
   want to argue that DeJong's functions WERE excellent, so good that they
   now ARE a victim of their own success.  My argument goes beyond any
   particular features of these functions or the GA, and therefore won't make
   historical references beyond those just sketched.  \footnote{If I'm right
   though, it would be an interesting exercise in history of science to
   confirm it, with a careful analysis of just which test functions were
   used, when, by whom, with citation counting, etc.} It will rely instead on
   some fundamental facts from machine learning.

   Paradigmatic Over-fitting

   "Over-fitting" is a widely recognized phenonemon in machine learning (and
   before that, statistics).  It refers to a tendancy by learning algorithms
   to force the rule induced from a training corpus to agree with this data
   set too closely, at the expense of generalization to other instances.  We
   have all probably seen the example of the same data set fit with two
   polynomials, one that is correct and a second, higher-order one that also
   attempts to fit the data's noise.  A more recent example is provided by
   some neural networks, which generalize much better to unseen data if their
   training is stopped a bit early, even though further epochs of training
   would continue to reduce the observed error on the training set.

   I suggest entire scientific disciplines can suffer a similar fate.  Many
   groups of scientists have found it useful to identify a particular data
   set, test suite or "model animal" (i.e., particular species or even
   genetic strains that become {\em de rigueur} for certain groups of
   biologists).  In fact, collective agreement as the validity and utility of
   scientific artifacts like this are critically involved in defining the
   "paradigms" (ala Kuhn) in which scientists work.  Scientifically, there
   are obvious benefits to coordinated use of common test sets.  For example,
   a wide variety of techniques can be applied to common data and the results
   of these various experiments can be compared directly.  But if science is
   also seen as an inductive process, over-fitting suggests there may also be
   dangers inherent in this practice.

   Initially, standardized test sets are almost certain to help any field
   evaluate alternative methods; suppose they show that technique A1 is
   superior to B1 and C1.  But as soon as the results of these experiments
   are used to guide the development ("skew the sampling") of new methods (to
   A2, A3 and A4 for example), our confidence in the results of this second
   set of experiments as accurate reflections of what will be found generally
   true, must diminish.  Over time, then, the same data set that initially
   served the field well can come to actually impeed progress by creating a
   false characterization of the real problems to be solved.  The problem is
   that the time-scale of scientific induction is so much slower than that of
   our computational methods that the biases resulting from "paradigmatic
   over-fitting" may be very difficult to recognize.

   Machine learning also offers some remedies to the dilema of over-training.
   The general idea is to use more than one data set for training, or more
   accurately, partition available training data into subsets.  Then,
   portions of the training set can be methodically held back in order to
   compare the result induced from one subset with that induced from another
   (via cross validatation, jack-knifing, etc.).

   How might this procedure be applied to science?  It would be somewhat
   artificial to purposefully identify but then hold back some data sets,
   perhaps for years.  More natural strategies with about the same effect
   seem workable, however.  First, a field should maintain MULTIPLE data
   sets, to minimize aberations due to any one.  Second, each of these can
   only be USED FOR A LIMITED TIME, to be replaced by a new ones.

   The problem is that even these modest conventions require significant
   "discipline discipline."  Accomplishing any coordination across
   independent-minded scientists is difficult, and the use of shared data
   sets is a fairly effortless way to accomplish useful coordination.  Data
   sets are difficult to obtain in the first place, and convincing others to
   become familiar with them ever harder; these become intertial forces that
   will make scientists reluctant to part with the classic data sets they
   know well.  Evaluating results across multiple data sets also makes new
   problems for reviewers and editors.  And, because the time-scale of
   scientific induction is so long relative to the careers of the scientists
   involved, the costs associated with all these concrete problems, relative
   to theoretical ones due to pardigmatic over-fitting, will likely seem
   huge: "Why should I give up familiar data sets when we previously agreed
   to their validity, especially since my methods seem to be working better
   and better on them?!"

   Back to the GA

   The GA community, at present, seems fairly healthy according to this
   analysis.  In addition to De Jong's, people like Dave Ackley have
   genereated very useful sets of test functions.  There are now test suites
   that have many desirable properities, like being "GA-hard,"
   "GA-deceptive," "royal road," "practical" and "real-world."  So there are
   clearly plenty of tests.

   For this group, my main point is that this plurality is very desirable.  I
   too am dissatisfied with De Jong's test suite, but I am equally
   dissatisfied with any ONE of the more recently proposed alternatives.  I
   suggest it's time we move beyond debates about whose tests are most
   illuminating.  If we ever did pick just one set to use for testing GAs it
   would --- like De Jong's --- soon come to warp the development of GAs
   according to ITS inevitable biases.  What we need are more sophisticated
   analyses and methodologies that allow a wide variety of testing
   procedures, each showing something different.

   Flame off,
	   Rik Belew

   [I owe the basic insight --- that an entire discipline can be seen to
   over-fit to a limited training corpora --- to conversations with Richard
   Palmer and Rich Sutton, at the Santa Fe Institute in March, 1992.  Of
   course, all blame for damage occuring as the neat, little insight was
   stretched into this epistle concerning GA research is mine alone.]


       Richard K. Belew
       Computer Science & Engr. Dept (0014)
       Univ. California - San Diego
       La Jolla, CA 92093

       rik@cs.ucsd.edu

   From now until about 20 July I will be working in Paris:
   Status: RO

       c/o J-A. Meyer
       Groupe de BioInformatique
       URA686. Ecole Normale Superieure
       46 rue d'Ulm
       75230 PARIS Cedex05
       France
       Tel: 44 32 36 23
       Fax: 44 32 39 01

       belew@wotan.ens.fr

------------------------------

From: <EFKWEPES%HMARL5.bitnet@CUNYVM.CUNY.EDU>
Date: Tue, 12 May 92 23:01 N
Subject: Two new GA papers available

   There is a new paper available that continues the research of N.L.J.
   Ulder, E.H.L. Aarts, H.L. Bandelt, P.J.M. van Laarhoven, and E. Pesch:
   Genetic local search algorithms for the traveling salesman problem. Proc.
   1st. Int. Workshop on Parallel Problem Solving from Nature (H.-P. Schwefel
   and R. Maenner, eds.), Lecture Notes in Computer Science 496 (1991),
   109-116

   The paper is: A. Kolen and E. Pesch: Genetic local search in combinatorial
   optimization, Discrete Applied Math. (to appear) Abstract: The most common
   application of genetic algorithms to combinatorial optimization problems
   has been restricted to the traveling salesman problem.  We review some of
   these ideas and present some new results, especially in th cas e that
   severe time constraints are imposed on the running time of the algorithm.

   Another paper is available:

   U. Dorndorf and E. Pesch: Evolution based learning in a job shop
   scheduling environment.

   Abstract: A class of approximation algorithms is described for solving the
   minim um makespan problem of job shop scheduling. A common basis of these
   algorithms is the underlying genetic algorithm that serves as a
   meta-strategy to guide an optimal design of local decision rule sequences.
   Computational experiments show that our algorithm can find shorter
   makespans than the shifting bottleneck heuristic or a simulated annealing
   approach with the same running time.

   Requests to
   Erwin Pesch
   Faculty of Economics, KE
   University of Limburg
   P.O. Box 616
   6200 MD Maastricht
   The Netherlands
   e-mail:EFKWEPES@HMARL5.Bitnet

------------------------------

From: Frederic Gruau <gruau@drfmc.CENG.cea.fr>
Date: 19 May 92 11:58:24+0100
Subject: Genetic Neural Network

   The Laboratoire de l' Informatique du Parallelisme at The Ecole Normale
   Superieure de Lyon is pleased to announce the availability of the
   following technical report:


	      Cellular Encoding of Genetic Neural Network

			     Gruau Frederic
	     Laboratoire de l'Informatique du Parallelisme
		     Ecole Normale Superieure de Lyon
			   47 Allee d'Italie
			  69007 Lyon, France


			       ABSTRACT

   It is now recognized that the key point in successful application of
   Genetic Algorithms (GA) to an optimization problem, is the scheme used to
   encode solutions on the structures manipulated by the GA.  Despite the
   fundamental importance of the encoding, there has not yet been any attempt
   to establish a description of which theoretical properties this encoding
   should have in the particular case of neural network optimization.  In the
   first part of this report, seven theoretical and verifiable properties
   that rate an encoding of neural networks are defined.  The properties are:
   completeness, compactness, closure, modularity, scalability, power of
   expression, abstraction.  An encoding called ``cellular encoding'' with
   all these properties is described.  It is an extension of an encoding
   scheme that has already given surprising experimental results. The
   analysis of the seven properties explains this empirical success.

   In the second part, continuing the presentation of cellular encoding, the
   alphabet of the code is extended so as to form a neural network machine
   language.  A compiler for this machine language is described.  It compiles
   programs written with a subset of pascal.  This fact allows to complete
   the properties of cellular encoding.

   Copies of the report can be obtained by contacting:

   Gruau Frederic
   Laboratoire de l'Informatique du Parallelisme
   Ecole Normale Superieure de Lyon
   47 Allee d'Italie
   69007  en troiLyon, France
   <<email>> gruau@drfmc.ceng.cea.fr

------------------------------

From: CCA91584@IBM3090.COMPUTER-CENTRE.BIRMINGHAM.AC.UK
Date: Tue, 19 May 92 15:59:23 BST
Subject: GA research centres, need information

   Hi, my name is Andy Pryke, and I am currently doing an MSc in Cognitive
   Science at Birmingham University, England. I am about to embark on a GA /
   A-Life project. I have a computing background and would like to do a PhD
   or other research in GA's / A-Life, so I am interested in finding out what
   institutions (UK or elsewhere) are currently active in this field. Any
   information would be gratefully received.

		  Thanks in advance,
				    Andy Pryke

	 (Email : CCA91584@ibm3090.computer-centre.birmingham.ac.uk)

   Ps. Does anyone (ed?) have the A-Life digest email address).

------------------------------
End of Genetic Algorithms Digest
******************************
