Tom Mitchell and Andrew W.
Moore
Center for Automated Learning and Discovery
School of Computer Science,
Carnegie Mellon University
It is hard to imagine anything more fascinating than systems that automatically improve their own performance through experience. Machine learning deals with computer algorithms for learning from many types of experience, ranging from robots exploring their environments, to mining preexisting databases, to actively exploring and mining the web. This course is designed to give PhD students a thorough grounding in the methodologies, technologies, mathematics and algorithms needed to do research in learning and data mining, or to apply learning or data mining techniques to a target problem.
The topics of the course draw from classical statistics, from machine learning, from data mining, from Bayesian statistics and from statistical algorithmics.
Students entering the class with a preexisting working knowledge of probability, statistics and algorithms will be at an advantage, but the class has been designed so that anyone with a strong numerate background can catch up and fully participate.
IF
YOU ARE ON THE WAIT LIST:
This class if now fully subscribed. You may want to consider
the
following options:
Class lectures: Tuesdays & Thursdays 10:30am11:50am, Wean Hall 7500 starting on Tuesday September 13th, 2005
Review sessions: Thursdays 56pm, Location NSH 1305, starting on thursday September 15. TA's will cover material from lecture and the homeworks, and answer your questions. These review sessions are optional (but very helpful!).
Instructors:
Textbook:
Module 
Date 
Lecture
topic and readings

Lecturer  Homeworks 
Optional warmup  Thu Sep 8  Optional
lecture: warmup review of some basic
probability
concepts.

Moore  
Overview
and a Machine Learning algorithm 
Tu Sep 13 
Machine
Learning, Function
Approximation,
Decision Tree learning

Mitchell 

Review
of probability, Maximum likelihood estimation, MAP estimation 
Th Sep 15  Fast tour of useful concepts in probability  Moore 
HW1 pdf ps.gz Corrections Solutions 
Tu Sep 20 
MLE
and MAP
estimation

Moore  
Linear models 
Th
Sep 22 
Linear
Regression
and Basis Functions

Moore  
Naive
Bayes 
Tu Sep 27 
Bayesian
classifiers, Naive
Bayes classifier, MLE and MAP estimates

Mitchell 
HW1
due HW2 pdf train1.txt test1.txt plotGauss.m Solutions 
Logistic
regression Discriminative and Generative Models 
Th Sep 29  Logistic
regression, Generative and discriminative classifiers, maximizing
conditional data likelihood, MLE and MAP estimates.

Mitchell 

Nonlinear
models Neural Networks 
Tu Oct 4 
Neural
networks and
gradient descent

Mitchell  
Th Oct 6 
Crossvalidation
and instancebased learning

Moore  HW2 due  
Gaussian Mixture Models 
Tu Oct 11 
Crossvalidation continued  Moore 


Th Oct 13 
no lecture 


Midterm Exam  Tu Oct 18 
Covers
everything up to this date. Open book,
notes. Closed computer. Come to class by 10.30am promptly. You will then have 80 minutes to answer six mostlyshort questions on material covered in the lectures and readings up to and including October 11th. We strongly advise you to practice using previous exams, so you know what to expect. try doing the previous exams first, and then look at the solutions. You will be allowed to look at your notes in class, but don't rely on this because you will run out of time unless you are sufficiently familiar with the material that you can just do the questions without needing to look up the techniques. In addition, to help prepare, there will be a review at the recitation session at 5pm Thursday Oct 13th, and there will be another review on Monday Oct 17th, 6pm7.30pm in NSH 1305. Previous examinations for practice. 
Project proposals due  
Computational learning theory 
Th Oct 20 
PAC
Learning I: sample complexity, agnostic learning

Mitchell  HW3 ds2.txt 
Tu Oct 25 
PAC Learning
II: VC dimension, SRM, Mistake bounds

Mitchell  
Margin based approaches  Th Oct 27 
SVMs,
kernels,
and optimization methods

Moore  RecitationHW3 
Graphical Models  Tu Nov 1 
Bayes nets:
representation, conditional independence

Mitchell  HW3 due 
Th Nov 3 
Bayes nets: inference, variable elimination, etc.  Moore  Recitation  
Tu Nov 8 
Bayes nets: learning parameters and structure (fully observed data, and begin EM)  Goldenberg  
EM and semisupervised learning  Th Nov 10 
EM for Bayes
networks and Mixtures of Gaussians 
Mitchell  
HMMs  Tu Nov 15 
Hidden Markov Models: representation and learning  Moore  
Time series models  Th Nov 17 
Graphical Models: an overview of more advanced probabistic models that fall under a category called Graphical Models. This lecture defines and talks about specifric instances, such as Kalman filters, undirected graphs and Dynamic Bayesian Networks  Goldenberg  Final project reports due 
Mon Nov 21  Project poster session: 46:30pm in the NewellSimon Hall Atrium  Project poster session  
Dimensionality reduction  Tu Nov 22 
Dimensionality Reduction: Feature selection, PCA, SVD, ICA, Fisher discriminant  Mitchell  
Tu Nov 29 
Advanced topic: Machine Learning and Text Analysis  Mitchell  HW4 missing.csv EM notes Inference notes Solutions 

Markov models  Th Dec 1 
Markov decision processes: Predicting the results of decisions in an uncertain world.  Moore  
Tu Dec 6 
Reinforcement
learning: Learning policies to
maximize expected future rewards in an uncertain world.

Moore  
Th Dec 8 
Scaling: Some of Andrew's favorite data structures and algorithms for tractable statistical machine learning.  Moore  HW4
due 

Final Exam 
Monday
Dec 19 
December
19 8:3011:30a.m at HH B103 and HH B131 (Hammerschlag Hall). No rescheduling possible. open book, open notes, closed computer. 
HMM/MDP Review 
Course Website (this page):
Note to people outside CMU: Please feel free to reuse any of these course materials that you find of use in your own courses. We ask that you retain any copyright notices, and include written notice indicating the source of any materials you use.