Newsgroups: comp.ai.genetic
Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!news.mathworks.com!news.alpha.net!uwm.edu!msunews!harbinger.cc.monash.edu.au!lugb!latcs1!lengers
From: lengers@latcs1.lat.oz.au (Roeland Lengers)
Subject: Using GA to optimize Neural Nets
Message-ID: <D68Gt7.FrF@latcs1.lat.oz.au>
Organization: Comp Sci, La Trobe Uni, Australia
Date: Thu, 30 Mar 1995 03:10:19 GMT
Lines: 136

Hello everyone,

I noticed that there where a few questions posted about using Genetic
Algorithms to optimize Neural Networks (weights as well as topology).
I am cuurently doing research on doing just this, although I haven't
figured out yet what technique to use exactly. However, I gathered
some references that might be usefull to others as well.

Here goes:

[Yao, 1993]  Xin Yao: A Review of Evolutionary Artificial Neural
Networks. In: International Journal of Intteligent Systems, Vol. 8,
pages 539-567, John Wiley & Sons, Inc., 1993.
This is a very useful article, it contains lots of references to GA/NN
combinations. It describes three uses of the combination: using GA's
to evolve the weights, the topology and the learning algorithm. Very
useful.


A list on articles in the 1993 International Conference on Neural
Networks. Just to give you an idea where you could get your material.

[Adler, 1993]  Dan Adler: Genetic Algorithms and Simulated Annealing:
A marriage Proposal. In: 1993 IEEE International Conference on Neural
Networks, pages 1104-1109.

[Peck, 1993]  Charles C. Peck and Atam P. Dhawan: Genetic Algorithm
based Input Selection for a Neural Network Function Approximator with
Application to SSME Health Monitoring. In: 1993 IEEE International
Conference on Neural Networks, pages 1115-1122.

[Oliker, 1993]  S. Oliker, M. Furst and O. Maimon: Design
Architectures and Training of Neural Networks with a Distributed
Genetic Algorithm. In: 1993 IEEE International Conference on Neural
Networks, pages 119-202.

[Koza, 1993]  John R. Koza, Martin A . Keane and James P. Rice:
Performance Improvement of Machine Learning via Automatic Discovery of
Facilitating Functions as Applied to a problem of Symbolic System
Identification. In: 1993 IEEE International Conference on Neural
Networks, pages 191-198.

[McInerney, 1993]  Michael McInerney and Atam P. Dhawan: Use of
Genetic Algorithms with Back Propagation in Training of Feed-Forward
Neural Networks. In: 1993 IEEE International Conference on Neural
Networks, pages 203-208.

[Ichikawa, 1993]  Yoshiaki Ichikawa and Yoshikazu Ishii:  Retaining
Diversity of Genetic Algorithms for Multivariable Optimization and
Neural Network Learning. In: 1993 IEEE International Conference on
Neural Networks, pages 1110-1114.

[Sin, 1993]  Sam-Kit Sin and Rui J.P. deFigueoredo: A Method for the
Design of Evolutionary Multilayer Neural Networks. In: 1993 IEEE
International Conference on Neural Networks, pages 869-870.

[McDonnel, 1993]  John R. McDonnel & Don Waagen: Evolving Neural
Network Connectivity. In: 1993 IEEE International Conference on Neural
Networks, pages 863-868.



Then a few articles that can be found in the neuroprose directory on
archive.cis.ohio-state.edu. Some very useful articles, just get the
INDEX file from the subdir, and check out the keywords.

[Schiffman, 1992]  W. Schiffman, M. Joost and R. Werner: Synthesis and
Performance of Multilayer Neural Network Architectures. Technical
Report 16/1992. University of Koblenz, Institute fur Physics,
Koblenz, Germany, 1992. FTP: archive.cis.ohio-state.edu in
/pub/neuroprose/schiff.gann.ps.Z.

[Boers, 1992]  Egbert J.W. Boers and Herman Kuiper: Biological
Metaphors and the Design of Modular Artificial Neural Networks.
Master's Thesis for Departements of Computer Science and Experimental
and Theoretical Psychology at Leiden University, the Netherlands. FTP:
archive.cis.ohio-state.edu in
/pub/neuroprose/boers.biological-metaphors.ps.Z.
This article descibes evolving a topology, not the weights.

[Perrone, 1992]  Michael P. Perrone and Leon N. Cooper: When Networks
Disagree: Ensemble Methods for Hybrid Neural Networks. In: "Neural
Networks for Speech and Image Processing" R.J. Mammone, ed.
Chapman-Hall, 1993. FTP: archive.cis.ohio-state.edu in
/pub/neuroprose/perrone.MSE-averaging.ps.Z.

And for a change: a book.
[Michalewicz, 1992]  Zbigniew Michalewicz: Genetic Algorithms + Data
Structures = Evolution Programs. Springer-Verlag, Berlin, 1992.
I put this in because it gives you some idea why you shouldn't use a
binary representation for your net. Altough I still haven't figured
out how I'm going to represent my NN. Most likely I'm going to use
Evolutionary Programming instead of the "Pure GA". Other articles in
this list refer to the same problem.

And for a bit more of the biological side of things:
[Nolfi, 1992]  Stefano Nolfi and Domenico Parisi: Growing Neural
Networks.  Technical Report PCIA-91-15, Institute of Psychology,
National Research Council, Rome. FTP: kant.irmkant.rm.cnr.it (or
150.146.7.5) /pub/econets/nolfi.growing.ps.Z.

[Nolfi, 1994]  Stefano Nolfi and Domenico Parisi: `Genotypes' for
Neural Networks.  In M.A. Arbib, ed.: The Handbook of Brain Theory and
Neural Networks. Bradford Books, MIT Press. The draft is in: Technical
Report 94-06. Institute of Psychology, National Research Council,
Rome. FTP: kant.irmkant.rm.cnr.it (or 150.146.7.5)
/pub/econets/nolfi.genes.ps.Z.


Another stochastical optimization technique to use for evolving the
neural net is simulated annealing. Look at this article to get an
idea of its usefulness.

[Ingber, 1992]  Lester Ingber and Bruce Rosen: Genetic Algorithms and
Very Fast Simulated Reannealing: A Comparison. Mathematical and
Computer Modelling, 16(11), pages 87-100, 1992. FTP:
alumni.caltech.edu /pub/ingber/asa92_saga.ps.Z.


And the last reference is about a masters thesis by James Spofford. It
describes a combined weights/topology evolution of a NN. It has some
drawbacks, that will hopefully be solved in version 2 of GANNET. This
version will be announced soon by Dr. Kenneth Hintz (look in the file
/gannet/thesis/readme (if I remember correctly).

[Spofford, 1990]  Jason Joseph Spofford: Evolving Neural Networks with
a Genetic Algorithm. Masters Thesis, George Mason University, Fairfax,
Virginia, 1990. FTP: fame.gmu.edu /gannet/thesis/thesis1..5,a,b.ps.



Hope someone out there in cyberspace finds this useful. If so, or if
someone else has got some more references that might be useful, please
mail them to me.

Thanks.
