Spring 2004 - Course Syllabus
Last modified: Sun May 2 23:18:10 EDT 2004
Monday, Jan. 12.
Organizational meeting; introduction to neural nets.
[ps, pdf]
- Hertz, Krogh & Palmer, chapter 1.
Wednesday, Jan. 14.
Perceptrons and the LMS Algorithm.
[ps, pdf]
- Hertz, Krogh & Palmer, chapter 5.
- Optional: Anderson, J. A., and Rosenfeld, E. (1998) Talking Nets:
An Oral History of Neural Networks, chapter 3. Cambridge, MA: MIT Press.
Problem set 1: learning with
linear units.
Monday, Jan. 19. No class. Martin Luther King's birthday.
Wednesday, Jan. 21.
Pattern Recognition I.
[ps, pdf]
Problem set 1 due.
- Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 1.
Oxford University Press.
Monday, Jan. 26.
Pattern Recognition II.
[ps, pdf]
- Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 2,
sections 2.1 through 2.5. Oxford University Press.
Wednesday, Jan. 28.
Backpropagation Learning.
[ps, pdf]
- Hertz, Krogh & Palmer sections 5.4, and 6.1-6.3.
- Optional: Rumelhart, D. E., Hinton, G. E, and Williams, R. J. (1986) Learning
internal representations by error propagation. In D. E. Rumelhart and J.
L. McClelland (eds.), Parallel Distributed Processing: Explorations in
the Microstructure of Cognition, chapter 8. Cambridge, MA: MIT Press.
Problem set 2:
backpropagation learning.
Monday, Feb. 2.
Visually-Guided Robot Control.
[ps, pdf]
- Pomerleau, D.A. (1991) Efficient training of artificial neural networks
for autonomous navigation. Neural Computation, 3(1):88-97.
- Pomerleau, D.A. (1994) Neural network based vision for precise
control of a walking robot. Machine Learning, 15(2):125-135.
Wednesday, Feb. 4.
Optimization Techniques. [Kornel Laskowski]
[ps, pdf]
- Bishop, C. M. (1995) Neural Networks for Pattern
Recognition, chapter 7. Oxford University Press.
Problem set 2 due.
Problem set 3: ALVINN.
Monday, Feb. 9.
Overfitting and Early Stopping.
[ps, pdf]
- Caruana, R., Lawrence, S., and Giles, C. L. (2001) Overfitting
in neural networks trained with backpropagation, conjugate gradient,
and early stopping. In T. K. Leen, T. G. Dietterich, and V. Tresp
(eds.), Advances in Neural Information Processing Systems 13. MIT
Press.
- Weigend, A. S. (1993) On overfitting and the effective number of
hidden units. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and
A. Weigend (eds.), Proceedings of the 1993 Connectionist Models
Summer School, pp. 335-342.
Wednesday, Feb. 11.
Recurrent Backprop Networks.
- Hertz, Krogh & Palmer, section 7.3 up to the middle of p. 184.
- Elman, J. L. (1991) Distributed representations, simple recurrent networks,
and grammatical structure. Machine Learning 7(2,3):195-225.
Monday, Feb. 16.
No class (President's Day).
Wednesday, Feb. 18.
Neural Networks for Control.
[ps, pdf]
Problem set 3 due.
- Nguyen, D., and Widrow, B. (1990) The truck backer-upper: an
example of self-learning in neural networks. In W. T. Miller III,
R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for
Control, ch. 12, pp. 287-299. Cambridge, MA: MIT Press.
- Jordan, M. I., and Rumelhart, D. E. (1992) Forward models:
supervised learning with a distal teacher. Cognitive
Science 16(3):307-354.
Monday, Feb. 23.
Shared-Weight Networks.
- Lang, K., Waibel, A. and Hinton, G. E. (1990) A Time-Delay Neural Network Architecture
for Isolated Word Recognition. Neural Networks 3:23-43.
- LeCun, Y, Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,
Hubbard, W., and Jackel, L. D. (1989) Backpropagation applied to
handwritten zip code recognition. Neural Computation
1(4):541-551.
Wednesday, Feb. 25.
Time Series Prediction. [Kornel Laskowski]
[ps, pdf]
Monday, March 1.
Midterm Exam.
Wednesday, March 3.
Radial Basis Functions.
[ps, pdf]
- Moody, J.. and Darken, C. (1989) Fast learning in networks of
locally-tuned processing units. Neural Computation
1(2):281-294.
- Note: radial basis functions are also covered briefly in Hertz,
Krogh, & Palmer section 9.7.
Monday, March 8.
Spring break. No class.
Wednesday, March 10.
Spring break. No class.
Monday, March 15.
Object Recognition with Radial Basis Functions.
[ps, pdf]
Wednesday, March 17.
Competitive Learning and Kohonen Nets.
[ps, pdf]
- Optional reading: Rumelhart, D. E., and Zipser, D. (1986)
Feature discovery by competitive learning. In D. E. Rumelhart and
J. L. McClelland (eds.), Parallel Distributed Processing:
Explorations in the Microstructure of Cognition, vol. 1, chapter
5. Cambridge, MA: MIT Press.
Problem set 4: learning with a distal teacher.
Monday, March 22.
The EM (Expectation-Maximization) Algorithm.
[ps, pdf]
- Bishop, chapter 2, just section 2.6.
- Williamson, J. R. (1997) A constructive, incremental-learning network
for mixture modeling and classification.
Neural Computation 9(7):1517-1543.
Wednesday, March 24.
Hebbian Learning and Principal Components Analysis. [Kornel Laskowski]
[ps, pdf]
Monday, March 29.
Hopfield Nets and Boltzmann Machines.
[ps, pdf]
- Hertz, Krogh & Palmer, chapter 2 and section 7.1.
Problem set 4 due.
Wednesday, March 31.
Boltzmann Machines and Mean Field Approximation
- Hopfield, J. J. (1984) Neurons with graded response have
collective computational properties like those of two-state neurons.
Proceedings of the National Academy of Sciences,
81:3088-3092.
- Hopfield, J. J., and Tank, D. W. (1985) ``Neural'' computation of decisions
in optimization problems. Biological Cybernetics, 52:141-152.
Monday, April 5.
Helmholtz Machines; Minimum Description Length.
- Hinton, G. E., Dayan, P. Frey, B. J., and Neal, R. M. (1995)
The wake-sleep algorithm for
unsupervised neural networks. Science,
268:1158-1160.
- Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995)
The Helmholtz machine.
Neural Computation, 7(5):889-904.
- Brief
tutorial on information theory, by Dave Touretzky.
Wednesday, April 7.
Bayesian Networks.
[ps, pdf]
Monday, April 12.
Computational Learning Theory.
[ps, pdf]
- Kearns, M. J., and Vazirani, U. V. (1994)
An introduction to Computational Learning Theory, chapter 1.
Cambridge, MA: MIT Press.
Problem set 5: Hopfield and
Boltzmann networks.
Wednesday, April 14.
Recursive Structures 1: Backprop.
Monday, April 19.
Recursive Structures 2: Convolutional Approaches.
[ps, pdf]
- Plate, T. (1998) Analogy
retrieval and processing with distributed representations.
Technical report CS-TR-98-4, Victoria University of Wellington,
Computer Science.
- Touretzky, D. S. (in press) Connectionist and symbolic
representations. In M. A. Arbib (Ed.), Handbook of Brain Theory
and Neural Networks, 2nd edition. Cambridge, MA: MIT Press.
Problem set 5 due.
Problem set 6: digit
recognition.
Wednesday, April 21.
Attractor Bump Models.
- Seung, H. S. (1996) How the brain keeps the eyes still.
Proceedings of the National Academy of Sciences, 93:13339-13344.
Monday, April 26.
Neurophysiology for Computer Scientists.
Wednesday, April 28.
The Mammalian Visual System.
- Van Essen, D. C. and Anderson, C. H. (1990) Information processing strategies
and pathways in the primate retina and visual cortex. In S. F. Zornetzer, J. L. Davis,
and C. Lau (Eds.), An Introduction to Neural and Electronic Networks, pp. 43-72.
San Diego: Academic Press.
- Van Essen, D. C., and DeYoe, E. A. (1995) Concurrent processing in
the primate visual cortex. In M. S. Gazzaniga (Ed.), The Cognitive
Neurosciences, pp. 383-400. MIT Press.
Problem set 6 due.
Monday, May 3. Open book/open notes Final Exam.
1:00 pm to 4:00 pm in Wean Hall 7500.
Dave Touretzky