Fall 2006  Course Syllabus
Last modified: Fri Dec 1 04:18:23 EST 2006



Monday, August 28.
Organizational meeting; introduction to neural nets.
[pdf]
 Hertz, Krogh & Palmer, chapter 1.
Wednesday, August 30.
Perceptrons and the LMS Algorithm.
[pdf]
 Hertz, Krogh & Palmer, sections 5.1 through 5.4.
 Optional enrichment: Anderson, J. A., and Rosenfeld,
E. (1998) Talking Nets: An Oral History of Neural Networks, chapter
3, by Bernard Widrow. Cambridge, MA: MIT Press. Available online
at CogNet.
Problem set 1: learning with
linear units.
Monday, Sep. 4. No class. Labor day.
Wednesday, Sep. 6. Backpropagation Learning. [pdf]
 Hertz, Krogh &
Palmer sections 5.5, and 6.16.3.
 Handout: Derivation of the backprop learning rule
 Optional enrichment: Rumelhart, D. E., Hinton, G. E, and Williams,
R. J. (1986) Learning
internal representations by error propagation. In D. E. Rumelhart
and J. L. McClelland (eds.), Parallel Distributed Processing:
Explorations in the Microstructure of Cognition, chapter 8.
Cambridge, MA: MIT Press. (local copy)
Problem set 1 due.
Problem set 2:
backpropagation learning.
Monday, Sep. 11.
VisuallyGuided Robot Control.
[pdf]
Note: class now meets in NSH 3002
Wednesday, Sep. 13.
Optimization Techniques.
[pdf]
 Bishop, C. M. (1995) Neural Networks for Pattern
Recognition, chapter 7.
Oxford University Press.
Problem set 2 due.
Problem set 3: ALVINN.
Monday, Sep. 18.
Overfitting, CrossValidation, and Early Stopping.
[pdf]
 Caruana, R., Lawrence, S., and Giles, C. L. (2001) Overfitting
in neural networks trained with backpropagation, conjugate gradient,
and early stopping. In T. K. Leen, T. G. Dietterich, and V. Tresp
(eds.), Advances in Neural Information Processing Systems 13. MIT
Press.
 Weigend, A. S. (1993) On
overfitting and the effective number of hidden units. In M. Mozer,
P. Smolensky, D. Touretzky, J. Elman, and A. Weigend (eds.),
Proceedings of the 1993 Connectionist Models Summer School,
pp. 335342.
Wednesday, Sep. 20.
Simple Recurrent Networks.
[pdf]
Monday, Sep. 25.
Language Processing Models.
[pdf]
Wednesday, Sep. 27.
Pattern Classification I.
[pdf]
 Bishop, C. M. (1995) Neural Networks for Pattern
Recognition, chapter 1.
Oxford University Press.
Problem set 3 due.
Monday, Oct. 2. Pattern Classification II. [pdf]
 Bishop, C. M. (1995)
Neural Networks for Pattern Recognition, chapter 2, sections 2.1 through
2.5. Oxford University Press.
Wednesday, Oct. 4.
Radial Basis Functions.
[pdf]
Monday, Oct. 9.
The EM (ExpectationMaximization) Algorithm.
[pdf]
 Bishop, chapter 2, just section 2.6.
 Optional enrichment: Ueda, N., Nakano, R., Ghahramani, Z.,
and Hinton, G. E. (2000) SMEM
algorithms for mixture models. Neural Computation
12:21092128.
 Optional enrichment: Williamson, J. R. (1997) A constructive,
incrementallearning network for mixture modeling and
classification. Neural Computation
9(7):15171543.
Wednesday, Oct. 11.
Neural Networks for Control.
[pdf]
 Jordan, M. I., and Rumelhart, D. E. (1992) Forward models: supervised learning
with a distal teacher. Cognitive Science
16(3):307354.
 Optional enrichment: Nguyen, D., and Widrow, B. (1990) The truck backerupper: an example of
selflearning in neural networks. In W. T. Miller III,
R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for
Control, ch. 12, pp. 287299. Cambridge, MA: MIT Press.
Monday, Oct. 16.
Support Vector Machines. [Guest lecturer: Mark Fuhs]
[PowerPoint]
 Haykin, S. (1999) Neural Networks: A Comprehensive Foundation,
2nd edition. Chapter 6, Support Vector Machines,
sections 6.16.4 and 6.9. PrenticeHall.
Wednesday, October 18.
Midterm Exam.
Monday, Oct. 23.
Time Series Prediction.
[pdf]
Wednesday, Oct. 25.
Shared Weight Networks.
 LeCun, Y., Bottou, L, Bengio, Y., and Haffner, P. (1998) Gradientbased learning applied to
document recognition, sections I through III. Proceedings
of the IEEE, November 1998.
 LeNet5 demos by Yann LeCun,
at http://yann.lecun.com/exdb/lenet/index.html.
Problem set 4: Robot arm control.
Monday, Oct. 30.
Competitive Learning and Kohonen Nets.
[pdf]
Wednesday, Nov. 1.
Hebbian Learning and Principal Components Analysis.
[pdf]
Problem set 4 due.
Monday, Nov. 6.
Hopfield Nets and Boltzmann Machines.
[pdf]
 Hertz, Krogh & Palmer, chapter 2 and section 7.1.
 Optional enrichment: Kirkpatrick, S., Gelatt, C. D., Jr.,
and Vecchi, M. P. (1983) Optimization by simulated
annealing. Science 220:671680.
Problem set 5: Hopfield networks.
Wednesday, Nov. 8.
Mean Field Approximation
[pdf]
Monday, Nov. 13.
Helmholtz Machines; Minimum Description Length.
[pdf]
 Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995)
The Helmholtz machine.
Neural Computation, 7(5):889904.
 Hinton, G. E., Dayan, P. Frey, B. J., and Neal, R. M. (1995)
The wakesleep algorithm for
unsupervised neural networks. Science,
268:11581160.
 Brief
tutorial on information theory, by Dave Touretzky.
Problem set 5 due.
Wednesday, Nov. 15.
Bayesian Networks.
[pdf]
 Bishop, chapter 10, sections 10.1, 10.2, and 10.6.
Monday, Nov. 20.
Computational Learning Theory.
[pdf]
 Kearns, M. J., and Vazirani, U. V. (1994) An introduction
to Computational Learning Theory, chapter 1. Cambridge, MA: MIT
Press.
Wednesday, Nov. 22. No class. Thanksgiving Holiday.
Monday, Nov. 27.
Connectionist Symbol Processing.
[pdf]
Wednesday, Nov. 29.
Reinforcement Learning.
[pdf]
Monday, Dec. 4.
Neurophysiology for Computer Scientists. [Guest lecturer: Mark Fuhs]
Wednesday, Dec. 6.
No class.
Tuesday, Dec. 12. Final exam, 14 pm, Wean Hall 5403.
Dave Touretzky