Homework # 5

15-486/782: Artificial Neural Networks Dave Touretzky, Fall 2006

Assigned November 6, 2006. Due November 13, 2006.

Consider the Hopfield network shown above; it has four units and five connections. Assume the units use +1/ − 1 states. We can denote a state α of the network as a whole as a four-tuple ⟨S_{A}, S_{B}, S_{C}, S_{D}⟩.

1. By hand, calculate the energy of the state ⟨−1, +1, +1, −1⟩. Show all your work.

2. Calculate the energies of the eight states where B has the value +1. You can either do this by hand, or write code to do it for you. Show each state and its energy.

3. Now that you know the energies of eight of the network’s sixteen states, how can you find the energies of the remaining eight states without doing any more additions or multiplications? (Hint: what do you know about “reversal states” in Hopfield nets?)

4. In a Boltzmann machine at equilibrium, the probabilities of the states satisfy the Boltzmann distribution:

\frac{P (α)}{P (β)} = \frac{e^{−E_{α}/T}}{e^{−E_{β}/T}}

From this relationship, show how to derive the formula for P (α) given the energies of all the states. Hint: note that ∑_{β} P (β) must be 1.

5. Suppose we want to use stochastic units, i.e., make the network a Boltzmann machine instead of a Hopfield net. Given the energies of all 16 states, and a temperature T, calculate the probability P (α) of each state α. Do this for three temperatures: 100, 5, and 0.1.

Display your results as an array called Probabilities with 16 rows and 3 columns. Also plot the result, and hand in the plot.

The simplest way to plot the result is to write plot(Probabilities). However, another cool way to look at the result is bar(Probabilities,'stacked'). Use whichever one you like best. Use the legend command to label your plot.

6. It’s possible to add bias connections to a Hopfield net. Let each unit have a bias connection b_{i}, so that its net input is:

net_{i} = b_{i} + ∑_{j} w_{ij}s_{j}

You can think of the bias connection as just an ordinary connection to an anonymous unit whose output is fixed at +1. Based on this idea, write down the energy function E for a Hopfield net with bias connections.

7. Assume a +1/−1 Hopfield net with states s_{i} and weights w_{ij}. Derive a 1/0 Hopfield net with states S_{i}, weights W_{ij}, and bias connections B_{i} that has the same energy landscape, i.e., the same energy values for corresponding states. Note: you will need to add a constant term C to the energy function.