Code

plotTree=Function[{labels},
Module[{coords},
coords = {{1,2},{2.5,3},{2,2},{3,2},{4,2},{1,0},{2.5,0},{4,0}};
TreePlot[{{2->1,"2"},{3->1,"1"},{4->1,"-3"},{5->1,"-2"},
{6->3,"1"},{6->4,"2"},{6->5,"3"},
{7->3,"-3"},{7->4,"2"},{7->5,"1"},
{8->3,"1"},{8->4,"-2"},{8->5,"1"}},
VertexLabeling->True,DirectedEdges->True,VertexRenderingFunction->(Text[Panel[Style[labels[[#2]],Small]],#1]&),
VertexCoordinateRules-> coords]]];
perceptronPlot=Module[{labels},
labels={"g[(∑_(i = 0)^3)(Subscript[w, i]) (Subscript[X, i])]","(Subscript[X, 0])=1","(Subscript[X, 1])","(Subscript[X, 2])"};
TreePlot[{{2->1,"(Subscript[w, 0])"}, {3->1,"(Subscript[w, 1])"}, {4->1,"(Subscript[w, 1])"}},VertexLabeling->True,DirectedEdges->True,VertexRenderingFunction->(Text[Panel[Style[labels[[#2]],Medium]],#1]&)]];

plt =Module[{labels, coords},labels={"out(x)=g[(∑_(i = 0)^3)(Subscript[u, i]) (Subscript[Z, i])]","(Subscript[Z, 0])=1","(Subscript[Z, 1])=g[(∑_(i = 0)^2)(Subscript[w, i]^1)(Subscript[X, i])]","(Subscript[Z, 2])=g[(∑_(i = 0)^2)(Subscript[w, i]^2)(Subscript[X, i])]","(Subscript[Z, 3])=g[(∑_(i = 0)^2)(Subscript[w, i]^3)(Subscript[X, i])]","(Subscript[X, 0])=1","(Subscript[X, 2])","(Subscript[X, 1])"};
coords = {{1,2},{2.5,3},{2,2},{3,2},{4,2},{1,0},{2.5,0},{4,0}};

TreePlot[{{2->1,"(Subscript[u, 0])"},{3->1,"(Subscript[u, 1])"},{4->1,"(Subscript[u, 2])"},{5->1,"(Subscript[u, 3])"},
{6->3,"(Subscript[w, 0]^1)"},{6->4,"(Subscript[w, 1]^1)"},{6->5,"(Subscript[w, 2]^1)"},
{7->3,"(Subscript[w, 0]^2)"},{7->4,"(Subscript[w, 1]^2)"},{7->5,"(Subscript[w, 2]^2)"},
{8->3,"(Subscript[w, 0]^3)"},{8->4,"(Subscript[w, 1]^3)"},{8->5,"(Subscript[w, 2]^3)"}},
VertexLabeling->True,DirectedEdges->True,VertexRenderingFunction->(Text[Panel[Style[labels[[#2]],Small]],#1]&),
VertexCoordinateRules->coords]
];
dynamicDemo =DynamicModule[{w10,w11,w12,w20,w21,w22,w30,w31,w32,u0,u1,u2,g},
r={-3,3};
g=Function[x,(1/(1+Exp[-x]))];
Grid[{
{MatrixForm[{{Slider[Dynamic[w10],r],Slider[Dynamic[w11],r],Slider[Dynamic[w12],r]},
{Slider[Dynamic[w20],r],Slider[Dynamic[w21],r],Slider[Dynamic[w22],r]},
{Slider[Dynamic[w30],r],Slider[Dynamic[w31],r],Slider[Dynamic[w32],r]}}]},
{MatrixForm[{{Slider[Dynamic[u0],r],Slider[Dynamic[u1],r],Slider[Dynamic[u2],r]}}]},
{Dynamic[
Plot3D[
g[Total[Flatten[
{{u0,u1,u2}} . g[{{w10,w11,w12},{w20,w21,w22},{w30,w31,w32}} .{{1},{a},{b}}]]
]],
{a,-10,10},{b,-10,10}]
]}}]
];

Neural Networks

Joseph E. Gonzalez

We are going to go through Neural Networks and review the process of back propagation.

Experimental Mathematica based presentation.

Single Perceptron

The Perceptron

perceptronPlot

There are several parts

Link Function

g[x]=(1/(1+(e^(-x))))

g=Function[x,(1/(1+Exp[-x]))];
Plot[g[x],{x,-8,8}]

Demo

Manipulate[
g=Function[x,(1/(1+Exp[-x]))];
Plot3D[ g[w0 + w1 x1 + w2 x2],{x1,-3,3},{x2,-3,3}],
{{w0,0},-3,3},{{w1,2},-3,3},{{w2,-2},-3,3}
]

Neural Network with Multiple Hidden Layers

Lets Consider what this network looks like

plt

Matlab Style Forward Propagation

Lets define a matrix Was:

W=(({{(Subscript[w, 0]^1), (Subscript[w, 1]^1), (Subscript[w, 2]^1)}, {(Subscript[w, 0]^2), (Subscript[w, 1]^2), (Subscript[w, 2]^2)}, {(Subscript[w, 0]^3), (Subscript[w, 1]^3), (Subscript[w, 2]^3)}}))

We can multiply this matrix by X where we have added a 1

W·(({{1}, {(Subscript[X, 1])}, {(Subscript[X, 2])}}))=(({{(Subscript[w, 0]^1), (Subscript[w, 1]^1), (Subscript[w, 2]^1)}, {(Subscript[w, 0]^2), (Subscript[w, 1]^2), (Subscript[w, 2]^2)}, {(Subscript[w, 0]^3), (Subscript[w, 1]^3), (Subscript[w, 2]^3)}}))·(({{1}, {(Subscript[X, 2])}, {(Subscript[X, 3])}}))=(({{(Subscript[w, 0]^1)+(Subscript[w, 1]^1)(Subscript[X, 1])+(Subscript[w, 2]^1)(Subscript[X, 2])}, {(Subscript[w, 0]^2)+(Subscript[w, 1]^2)(Subscript[X, 1])+(Subscript[w, 2]^2)(Subscript[X, 2])}, {(Subscript[w, 0]^3)+(Subscript[w, 1]^3)(Subscript[X, 1])+(Subscript[w, 2]^3)(Subscript[X, 2])}}))

Lets define function application as element wise.  Then we obtain:

g[W·(({{1}, {(Subscript[X, 2])}, {(Subscript[X, 3])}}))]=(({{g[(Subscript[w, 0]^1)+(Subscript[w, 1]^1)(Subscript[X, 1])+(Subscript[w, 2]^1)(Subscript[X, 2])]}, {g[(Subscript[w, 0]^2)+(Subscript[w, 1]^2)(Subscript[X, 1])+(Subscript[w, 2]^2)(Subscript[X, 2])]}, {g[(Subscript[w, 0]^3)+(Subscript[w, 1]^3)(Subscript[X, 1])+(Subscript[w, 2]^3)(Subscript[X, 2])]}}))=(({{(Subscript[Z, 1])}, {(Subscript[Z, 2])}, {(Subscript[Z, 3])}}))

We can then prepend a 1 to the result to obtain:

out(X)=g[(({{(Subscript[u, 0]), (Subscript[u, 1]), (Subscript[u, 2]), (Subscript[u, 3])}}))·(({{1}, {(Subscript[Z, 1])}, {(Subscript[Z, 2])}, {(Subscript[Z, 3])}}))]=g[(Subscript[u, 0])+(∑_(i=1)^3)(Subscript[u, i])(Subscript[Z, i])]

Forward Propagation (Example) #1

What is the value of (Subscript[Z, 1])

plotTree[{"out(x)=g[?]","(Subscript[Z, 0])=1","(Subscript[Z, 1])=g[?]","(Subscript[Z, 2])=g[?]","(Subscript[Z, 3])=g[?]","(Subscript[X, 0])=1","(Subscript[X, 2])=2","(Subscript[X, 1])=3"}]

What is the value of (Subscript[Z, 2])?

plotTree[{"out(x)=g[?]","(Subscript[Z, 0])=1","(Subscript[Z, 1])=g[1-6+3]=0.88","(Subscript[Z, 2])=g[?]","(Subscript[Z, 3])=g[?]","(Subscript[X, 0])=1","(Subscript[X, 2])=2","(Subscript[X, 1])=3"}]

What is the value of (Subscript[Z, 3])?

plotTree[{"out(x)=g[?]","(Subscript[Z, 0])=1","(Subscript[Z, 1])=0.88","(Subscript[Z, 2])=g[2+4-6]=0.5","(Subscript[Z, 3])=g[?]","(Subscript[X, 0])=1","(Subscript[X, 2])=2","(Subscript[X, 1])=3"}]

What is the value of out(X)?

plotTree[{"out(x)=g[?]","(Subscript[Z, 0])=1","(Subscript[Z, 1])=0.88","(Subscript[Z, 2])=0.5","(Subscript[Z, 3])=g[3+2+3]=1","(Subscript[X, 0])=1","(Subscript[X, 2])=2","(Subscript[X, 1])=3"}]

Done!

plotTree[{"out(x)=g[2+0.88-1.5-2]=0.35","(Subscript[Z, 0])=1","(Subscript[Z, 1])=0.88","(Subscript[Z, 2])=0.5","(Subscript[Z, 3])=1","(Subscript[X, 0])=1","(Subscript[X, 2])=2","(Subscript[X, 1])=3"}]

Demo

dynamicDemo

Generalized Back Propagation

plt

Suppose we want to find the best model out(x;U,W) with respect to the parameters W and U.  How can we quantify best? Lets considered mean squared error.

E=(∑_(i=1)^n)(((out((Subscript[X, i]))-(Subscript[Y, i])))^2)

There are many ways to do this.  One of the most common (and least effective) methods is to use gradient descent. This corresponds to the update rule:

(Subscript[u, i]^((t+1)))⟵(Subscript[u, i]^((t)))-η ((∂E)/(∂(Subscript[u, i])))|({{ }, {(Subscript[u, i]^((t)))}})

(Subscript[w, ij]^((t+1)))⟵(Subscript[w, ij]^((t)))-η ((∂E)/(∂(Subscript[w, ij])))|({{ }, {(Subscript[w, ij]^((t)))}})

Recall we have the following graph:

plt

Lets first derive the update rule for U:

E=(((out(X)-Y))^2)

Taking the derivative we get (stuck?):

((∂E)/(∂(Subscript[u, k])))=(∂/(∂(Subscript[u, k])))(((out(X)-Y))^2)

Applying the infamous chain rule:

(∂/(∂x))f(g(x))=((∂/(∂u))f(u)|({{ }, {u=g(x)}}))((∂/(∂x))g(x))

((∂E)/(∂(Subscript[u, k])))=(Underscript[2(out(X)-(Subscript[Y, i])), (Underscript[⏟, (∂/(∂x))(x^2)= 2x])])((∂/(∂(Subscript[u, k])))out(X))

(∂/(∂x))f(g(x)) = f'(g(x)) g'(x)

Now we need to take the derivative of the neural network.  Lets first replace out with the function from the top perceptron

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)((∂/(∂(Subscript[u, k])))g[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])])

Chain rule again

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)g'[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])]((∑_(i=0)^3)(∂/(∂(Subscript[u, k])))(Subscript[u, i])(Subscript[Z, i]))

We know that only one term in the (Subscript[Z, i]) sum will remain and that is (Subscript[Z, i=k])

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)g'[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])] (Subscript[Z, k])

Done thats it!!!  Sort of.  Lets look at the derivative of g[x]=(1/(1-Exp[-x]))

g'[x]=(∂/(∂x))(((1+Exp[-x]))^(-1))

g'[x]=-(((1+Exp[-x]))^(-2))(∂/(∂x))(1+Exp[-x])

g'[x]=-(((1+Exp[-x]))^(-2))((∂/(∂x))1+(∂/(∂x))Exp[-x])

g'[x]=-(((1+Exp[-x]))^(-2))(0+Exp[-x](∂/(∂x))(-x))

g'[x]=-(((1+Exp[-x]))^(-2))(0-Exp[-x])

g'[x]=(((1+Exp[-x]))^(-2))Exp[-x]

With some manipulation we get:

g'[x]=((Exp[-x])/((1+Exp[-x])))(1/((1+Exp[-x])))

g'[x]=((Exp[-x])/(1+Exp[-x]))g[x]

g'[x]=((1+Exp[-x]-1)/(1+Exp[-x]))g[x]

g'[x]=(((1+Exp[-x])/(1+Exp[-x]))+((-1)/(1+Exp[-x])))g[x]

g'[x]=(1-(1/(1+Exp[-x])))g[x]

g'[x]=(1-g[x])g[x]

Recall that we earlier had:

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)g'[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])](Subscript[Z, k])

we can make a simple substitution to get:

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)(1-g[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])])g[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])](Subscript[Z, k])

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)(1-out(X))out(X)(Subscript[Z, k])

plt

Gradient of W

That wasn't too bad.  How about the next layer.  We again start with:

E=(((out(X)-Y))^2)

Taking the derivative with respect to (Subscript[w, k]^r) (and applying the chain rule)

((∂E)/(∂(Subscript[w, k]^r)))=((∂E)/(∂out(X)))((∂out(X))/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(∂/(∂(Subscript[w, k]^r)))out(X)

Expanding out we get:

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(∂/(∂(Subscript[w, k]^r)))g[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])]

Chain rule:

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(g'[(∑_(i=0)^3)(Subscript[u, i])(Subscript[Z, i])])((∑_(i=0)^3)(∂/(∂(Subscript[w, k]^r))) (Subscript[u, i])(Subscript[Z, i]))

Recall that g'[x]=(1-g[x])g[x]

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)((∑_(i=0)^3)(∂/(∂(Subscript[w, k]^r))) (Subscript[u, i])(Subscript[Z, i]))

Remember that each of the (Subscript[Z, i]) is connected to all the perceptrons from the lower level so we must take the derivative of each.

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)((∑_(i=0)^3) (Subscript[u, i])(∂/(∂(Subscript[w, k]^r)))(Subscript[Z, i]))

Which becomes:

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)((∑_(i=0)^3) (Subscript[u, i])(∂/(∂(Subscript[w, k]^r)))g[(∑_(s=0)^3)(Subscript[w, s]^i) (Subscript[X, s])])

Another application of the chain rule:

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)(∑_(i=0)^3) (Subscript[u, i])g'[(∑_(s=0)^3)(Subscript[w, s]^i) (Subscript[X, s])](∂/(∂(Subscript[w, k]^r)))(∑_(s=0)^3)(Subscript[w, s]^i) (Subscript[X, s])

Taking the final derivative we have:

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)((∑_(i=0)^3) (Subscript[u, i]) (1-(Subscript[Z, i]))((Subscript[Z, i])))(Subscript[X, k])

Why is it called back propagation?

Lets look at the following two equations:

((∂E)/(∂(Subscript[u, k])))=2(out(X)-Y)(1-out(X))out(X)(Subscript[Z, k])

((∂E)/(∂(Subscript[w, k]^r)))=2(out(X)-Y)(1-out(X))out(X)(∑_(i=0)^3) (Subscript[u, i]) (1-(Subscript[Z, i]))((Subscript[Z, i]))(Subscript[X, k])

plt

We propagate the derivative information backwards to the inputs:

((∂E)/(∂(Subscript[u, k])))=(2(out(X)-Y)(1-out(X))out(X))(Subscript[Z, k])

((∂E)/(∂(Subscript[w, k]^r)))=(2(out(X)-Y)(1-out(X))out(X))(∑_(i=0)^3) (Subscript[u, i]) (1-(Subscript[Z, i]))((Subscript[Z, i]))(Subscript[X, k])

((∂E)/(∂(Subscript[w, k]^r)))=((2(out(X)-Y)(1-out(X))out(X))(∑_(i=0)^3) (Subscript[u, i]) (1-(Subscript[Z, i]))((Subscript[Z, i])))(Subscript[X, k])

Questions?

Any feedback about Mathematica style presentations is welcome.