Machine Learning, 10-701 and 15-781, 2005

Tom Mitchell and Andrew W. Moore
Center for Automated Learning and Discovery
School of Computer Science, Carnegie Mellon University

Fall 2005


It is hard to imagine anything more fascinating than systems that automatically improve their own performance through experience.  Machine learning deals with computer algorithms for learning from many types of experience, ranging from robots exploring their environments, to mining pre-existing databases, to actively exploring and mining the web.  This course is designed to give PhD students a thorough grounding in the methodologies, technologies, mathematics and algorithms needed to do research in learning and data mining, or to apply learning or data mining techniques to a target problem.

The topics of the course draw from classical statistics, from machine learning, from data mining, from Bayesian statistics and from statistical algorithmics.

Students entering the class with a pre-existing working knowledge of probability, statistics and algorithms will be at an advantage, but the class has been designed so that anyone with a strong numerate background can catch up and fully participate.

IF YOU ARE ON THE WAIT LIST:  This class if now fully subscribed.  You may want to consider the following options:

Class lectures: Tuesdays & Thursdays 10:30am-11:50am, Wean Hall 7500 starting on Tuesday September 13th, 2005

Review sessions: Thursdays 5-6pm, Location NSH 1305, starting on thursday September 15.    TA's will cover material from lecture and the homeworks, and answer your questions.  These review sessions are optional (but very helpful!).  


Course secretary:

Teaching Assistants:


Late homework policy:
Homework regrade policy:
Collaboration on Homeworks:
Course project: Exams:
Homeworks:  Coming soon!

Tentative lecture schedule:


Lecture topic and readings

 Lecturer  Homeworks
 Optional warm-up Thu Sep 8  Optional lecture:  warm-up review of some basic probability concepts. Moore
 Overview and a Machine Learning algorithm

 Tu Sep 13

  Machine Learning, Function Approximation, Decision Tree learning

 Review of probability,
Maximum likelihood estimation, MAP estimation
 Th Sep 15  Fast tour of useful concepts in probability  Moore HW1
pdf ps.gz Corrections Solutions

Tu  Sep 20

 MLE and MAP estimation Moore  
Linear models Th  Sep 22
 Linear Regression and Basis Functions Moore
Naive Bayes

Tu  Sep 27

 Bayesian classifiers, Naive Bayes classifier, MLE and MAP estimates
Mitchell HW1 due
pdf train-1.txt test-1.txt plotGauss.m Solutions
Logistic regression

Discriminative and Generative Models
Th  Sep 29  Logistic regression, Generative and discriminative classifiers, maximizing conditional data likelihood,  MLE and MAP estimates.  Mitchell

 Non-linear models
Neural Networks
Oct 4
 Neural networks and gradient descent
  • Lecture slides: neural networks
  • Required reading: Machine Learning  Chapter 4
  • Optional reading: Bishop chapter 9.1, 9.2
Oct 6
 Cross-validation and instance-based learning
  • Lecture slides: overfitting instance-based
  • Readings: 
    • Machine Learning  Chapter 4. 
    • For a worked example of using cross-validation with gradient descent see the following paper (particularly the appendix):  Memory-based learning, C. G. Atkeson, Memory-Based Approaches to Approximating Continuous Functions, Proceedings, Workshop on Nonlinear Modeling and Forecasting, Santa Fe, New Mexico, September 17-21, 1990
    • For more information about locally weighted methods see Locally Weighted Learning
Moore HW2 due
 Gaussian Mixture Models Tu
Oct 11
  Cross-validation continued Moore

Oct 13
 no lecture
Midterm Exam


Oct 18 
 Covers everything up to this date.  Open book, notes. Closed computer.

Come to class by 10.30am promptly. You will then have 80 minutes to answer six mostly-short questions on material covered in the lectures and readings up to and including October 11th. We strongly advise you to practice using previous exams, so you know what to expect. try doing the previous exams first, and then look at the solutions. You will be allowed to look at your notes in class, but don't rely on this because you will run out of time unless you are sufficiently familiar with the material that you can just do the questions without needing to look up the techniques.

In addition, to help prepare, there will be a review at the recitation session at 5pm Thursday Oct 13th, and there will be another review on Monday Oct 17th, 6pm-7.30pm in NSH 1305.

Previous examinations for practice.
Project proposals due
Computational learning theory Th
Oct 20
 PAC Learning I: sample complexity, agnostic learning
Mitchell HW3 ds2.txt


Oct 25
 PAC Learning II: VC dimension, SRM, Mistake bounds
Margin based approaches Th
Oct 27
 SVMs, kernels, and optimization methods
Moore RecitationHW3
 Graphical Models Tu
Nov 1
 Bayes nets: representation, conditional independence
HW3 due
Nov 3
 Bayes nets: inference, variable elimination, etc. Moore Recitation
Nov 8
 Bayes nets: learning parameters and structure (fully observed data, and begin EM) Goldenberg
 EM and semi-supervised learning Th
Nov 10
 EM for Bayes networks and Mixtures of Gaussians
 HMMs Tu
Nov 15
 Hidden Markov Models: representation and learning Moore
 Time series models Th
Nov 17
 Graphical Models: an overview of more advanced probabistic models that fall under a category called Graphical Models. This lecture defines and talks about specifric instances, such as Kalman filters, undirected graphs and Dynamic Bayesian Networks Goldenberg Final project reports due
  Mon Nov 21 Project poster session: 4-6:30pm in the Newell-Simon Hall Atrium   Project poster session
 Dimensionality reduction Tu
Nov 22
 Dimensionality Reduction: Feature selection, PCA, SVD, ICA, Fisher discriminant Mitchell  

Nov 29
Advanced topic: Machine Learning and Text Analysis Mitchell HW4
EM notes
Inference notes Solutions
 Markov models Th
Dec 1
 Markov decision processes: Predicting the results of decisions in an uncertain world. Moore
Dec 6
 Reinforcement learning: Learning policies to maximize expected future rewards in an uncertain world. Moore
Dec 8
 Scaling: Some of Andrew's favorite data structures and algorithms for tractable statistical machine learning. Moore HW4 due
Final Exam Monday Dec 19
 December 19  8:30-11:30a.m at HH B103 and HH B131 (Hammerschlag Hall).  No rescheduling possible.
 open book, open notes, closed computer.
HMM/MDP Review

Dimension Reduction


Web pages for earlier versions of this course:  (include examples of midterms, homework questions, ...) 

Course Website (this page):

Note to people outside CMU:  Please feel free to reuse any of these course materials that you find of use in your own courses.  We ask that you retain any copyright notices, and include written notice indicating the source of any materials you use.