This file contains a sample run of the Backprop simulator.  Loading
and running procedures are shown by example.

% # copy the files and compile it.
% cp /q1/ug/ics172ta/public/backprop/{backprop-sim.c,xor.pat,xor.wgt,encode4.pat,encode4.wgt} .
% scc backprop-sim.c -o backprop-sim 

% backprop-sim
Basic Net Simulator BSN.1, 17 Apr 89
Simple BackProp Net Simulator.
SBp> #
SBp> # Don't forget - here's how to get help!
SBp> #
SBp> help
MakeNet MFILE
  create the specified network structures and read the patterns in MFILE
net Abbreviation for 'MakeNet'.
ShowPat PAT_NO
  Show the network state vis-a-vis a given I/O pattern
sp Abbreviation for 'ShowPat'.
ShowPats Show the network state for all defined I/O patterns
sps Abbreviation for 'ShowPats'.
ReadWeights MFILE
        Read the connection weights from MFILE.
readw Abbreviation for 'ReadWeights'.
ShowWeights Show the connection weights of the network.
sw Abbreviation for 'ShowWeights'.
Strain S_TIMES
    Train the network by performing the back-propagation cycle S_TIMES
    [S_TIMES defaults to 30]
st Abbreviation for 'Strain'.
help print commands and their descriptions
h print commands and their descriptions
quit end the simulation
q end the simulation
source FILE
  read commands from FILE. They can be nested.
so abrev for 'source'
SBp> #
SBp> # First we bring into the program the 4 I/O patterns
SBp> # describing the operation of the XOR function.  You
SBp> # can develop your own examples, and it is recommended
SBp> # that you try running the corresponding example for
SBp> # encoding found in the "encode.pat" and "encode.wgt"
SBp> # files.  Note that activation values are given in
SBp> # percentages (100 = 100% = 1).
SBp> #
SBp> MakeNet xor.pat
INPunits 2 HIDunits 2 OUTunits 1
reading patterns
    0    0 ==>     0 : pat 0
    0  100 ==>   100 : pat 1
  100    0 ==>   100 : pat 2
  100  100 ==>     0 : pat 3
SBp> #
SBp> # Next, we get the initial weights for the network.  The files
SBp> # for patterns and weights are separated so you may exercise
SBp> # different combinations.  In this case, the listing shows
SBp> # that two connections exist from the input units to the first
SBp> # hidden unit with weights of 51 & 52.  The bias of the first
SBp> # hidden unit is 51.  Similiarly for the next hidden unit (3).
SBp> # Unit 4 is an output unit and has two connections to it from
SBp> # hidden units 2 and 3.  The 2->4 connection has weight 51 and
SBp> # the 3->4 connection has weight 49.  The bias on the output
SBp> # unit is 51.
SBp> #
SBp> ReadWeights xor.wgt
reading weights ...
to  2:     51    52  bias=     51
to  3:     50    49  bias=     50
to  4:     51    49  bias=     51
SBp> #
SBp> # Now we show a table describing the state of the network.
SBp> # The leftmost column (Iact) shows the input activation
SBp> # for each input unit.  The next two columns show the
SBp> # hidden unit activation levels and delta_j values.  The
SBp> # next two columns (Oact and Odta) show the activation
SBp> # and delta_j values for the output unit.  Also given
SBp> # are the target output, the output error, and the
SBp> # squared error (again, as percentages!).  The four
SBp> # groups (vertically) in the table correspond to the
SBp> # four I/O patterns given above.
SBp> #
SBp> ShowPats
   Iact  Hact  Hdta  Oact  Odta Targt  Oerr err^2
-     0    62   -17    76  -139     0   -76    57
      0    62   -16                        

-     0    74     4    78    39   100    22     5
    100    73     4                        

-   100    73     4    78    39   100    22     5
      0    73     4                        

-   100    82   -10    79  -131     0   -79    63
    100    82   -10                        

sum of sqr errs =    130
SBp> #
SBp> # Now lets run the back-propagation procedure to train the
SBp> # network.  This will alter the weights so that for any of
SBp> # the given inputs, the network will produce the proper 
SBp> # output.  The program prints either a "." for each cycle
SBp> # or a number (divisible by 5) and the squared error at that
SBp> # point.  If the procedure is working, the error should go
SBp> # toward 0.
SBp> #
SBp> Strain
0=130.....5=109.....10=100.....15=103.....20=104.....25=101.....
SBp> #
SBp> # Here are the weights after these 30 cycles:
SBp> #
SBp> ShowWeights
  from:     0     1
to  2:     35    36  bias=      6
to  3:     33    32  bias=      5

  from:     2     3
to  4:     17    13  bias=    -20
SBp> #
SBp> # And here is the network performance for each of the
SBp> # patterns.  Notice that only a small amount of progress
SBp> # has been made over these 30 cycles.
SBp> #
SBp> ShowPats
   Iact  Hact  Hdta  Oact  Odta Targt  Oerr err^2
-     0    52    -5    49  -122     0   -49    24
      0    51    -4                        

-     0    60     5    49   127   100    51    26
    100    59     4                        

-   100    60     5    49   127   100    51    26
      0    59     4                        

-   100    69    -4    50  -125     0   -50    25
    100    67    -4                        

sum of sqr errs =    100
SBp> #
SBp> # To speed up the process, lets train the network over a
SBp> # few hundred cycles to get convergence:
SBp> #
SBp> Strain 400
0=100.....5=100.....10=100.....15=100.....20=100.....25=100.....30=100.....35=100.....40=100.....45=100.....50=100.....55=100.....60=100.....65=100.....70=100.....75=100.....80=100.....85=100.....90=99.....95=99.....100=99.....105=98.....110=98.....115=97.....120=95.....125=93.....130=90.....135=86.....140=82.....145=77.....150=73.....155=69.....160=64.....165=59.....170=52.....175=42.....180=32.....185=21.....190=14.....195=9.....200=6.....205=5.....210=4.....215=3.....220=2.....225=2.....230=2.....235=2.....240=2.....245=1.....250=1.....255=1.....260=1.....265=1.....270=1.....275=1.....280=1.....285=1.....290=1.....295=1.....300=1.....305=1.....310=1.....315=1.....320=1.....325=1.....330=1.....335=1.....340=1.....345=1.....350=1.....355=1.....360=1.....365=1.....370=1.....375=1.....380=1.....385=1.....390=0.....395=0.....
SBp> #
SBp> # Notice that the convergence rate is very slow and asymptotic.
SBp> # It takes many more cycles to go from an error of 1 to 0 than
SBp> # from 2 to 1.  Also, due to the number of cycles with error = 100,
SBp> # it appears that some "ridge" in weight space is being followed
SBp> # before finding the "valley" leading to error = 0.
SBp> #
SBp> # The final weights are given below, along with the state of the
SBp> # network vis-a-vis the trained patterns.
SBp> #
SBp> ShowWeights
  from:     0     1
to  2:    629   629  bias=   -276
to  3:    411   411  bias=   -631

  from:     2     3
to  4:    825  -904  bias=   -372
SBp> ShowPats
   Iact  Hact  Hdta  Oact  Odta Targt  Oerr err^2
-     0     6    -1     4    -1     0    -4     0
      0     0     0                        

-     0    97     0    97     1   100     3     0
    100    10    -1                        

-   100    97     0    97     1   100     3     0
      0    10    -1                        

-   100   100     0     3    -1     0    -3     0
    100    87     1                        

sum of sqr errs =      0
SBp> # 
SBp> # Let's get outta here!
SBp> # 
SBp> quit
% 
