Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!vixen.cso.uiuc.edu!uwm.edu!reuter.cse.ogi.edu!netnews.nwnet.net!serval.net.wsu.edu!NewsWatcher!user
From: berg@falcon.flight.wpafb.af.mil (Jordan M. Berg)
Subject: Re: What is"simulated annealing"?
Message-ID: <berg-211194134840@134.131.68.86>
Followup-To: comp.ai.neural-nets
Sender: news@serval.net.wsu.edu (News)
Organization: NRC
References: <19941118130823.BKAMP@pi0192.kub.nl> <3aqg0b$d3t@hearst.cac.psu.edu>
Date: Mon, 21 Nov 1994 13:48:39 GMT
Lines: 60

In article <3aqg0b$d3t@hearst.cac.psu.edu>, caj@jerry.psu.edu wrote:

> In article <19941118130823.BKAMP@pi0192.kub.nl>, BKAMP@kub.nl  (Kamp B.) writes:
> >Hi,
> >I am quite familiar with NN jargon, but I don't know the meaning
> >of "simulated annealing".
> >What is this?
> >Bart
> 
> 
> "Simulated Annealing" is a term not developed in NN theory or exclusive to
> it so you may not see info about it in NN areas.
> 
> "Annealing" refers to a manufacturing process where metals (or other malleable
> stuff) if shaped under high temperature.  When very hot, the metal can be
> shaped into the general form deisired.  It is then cooled somewhat to hold
> this general form but to allow smaller refinements.  Finally it is cooled
> more for final touches.  You use the hot temps to get the shape into the
> right ballpark, then cool it down to do the final adjustments.

Not to nitpick, but I don't think that annealing itself is a manufacturing
process. When a piece of metal is shaped, hot or cold, the internal
crystalline structure is disrupted. Some of the energy of the shaping is
absorbed by the material in imperfections in the crystal lattice. The new
structure is typically harder ("strain hardened"), but also more prone to
fracture. Therefore if more shaping is to be done, the original crystal
structure must be restored.

This is done by heating the material above half its melting point, and
holding it there for some time, then slowly cooling it. The thermal energy
at high temp and the slow cooling allows the metal atoms to fall into their
lowest energy configuration, namely perfect crystals (well, at least large,
crystalline, grains). So annealing is done _between_ shaping stages. It is
aimed at microstructural transformation, not shape change. Annealing itself
has only a small effect on shape.

As I understand the analogy with the computational technique, it is the use
of thermal energy to free the atoms (solutions) from their local energy
minimum (the imperfect lattice) and the slow cooling, to allow them to
settle into their global energy minimum (the perfect lattice).

I know the original poster probably doesn't give a damn about (unsimulated)
annealing, but since I am doing some work in using neural nets to control
microstructure during forging, I felt the need to respond. And since I'm a
controls type, and not a metallurgist, I would be very happy to be
corrected.

> Simulated Annealing
> In computer models with adjustable parameters (such as NNs) you would do your
> initial training allowing the weights to change by large amounts
> (i.e. simulating high temperature.  Some methods even call the variable
> controlling the amount of change, t or temperature).  Then, during further
> training you reduce the amount the parameters (NN weights) are allowed to
> change (cool temperature).  The idea is that the high temp training avoids
> falling into local minima.  Then once in the global minima region, the 
> cooler temp training refines the solution.

Sounds good to me!

Jordan Berg
