Introduction to Machine Learning

10-701, Fall 2016
School of Computer Science
Carnegie Mellon University

Important Notes

This schedule is tentative and subject to change. Please check back often.

The first lecture for 10-701 is Wednesday, September 7th, 2016. That is, we start during the second week of classes.


Date Lecture Topics Readings Announcements


Wed, 9/7/16 Lecture 1 (Eric) : Intro to probability, MLE
[Slides] [Annotated Slides] [Video]
Course introduction, Basic probability, Maximum likelihood estimate

Supervised Learning

Mon, 9/12/16 Lecture 2 (Eric) : Classification, kNN
[Slides] [Annotated Slides] [Video]
Optimal decision using Bayes rule, Types of classifiers, Effect of values of k on kNN classifiers, Probabilistic interpretation of kNN

HW 1 Out

Wed, 9/14/16 Lecture 3 (Matt) : Naive Bayes
[Slides] [Video]
Problems with estimating full joints, Advantages of Naive Bayes assumptions, Applications to discrete and continuous cases, Problems with Naive Bayes classifiers
  • Mitchell, 6.1-6.10

Mon, 9/19/16 Lecture 4 (Matt) : Linear regression
[Slides] [Video]
Basic model, Solving linear regression, Error in linear regression, Advanced regression models
  • Bishop, 3.1

Wed, 9/21/16 Lecture 5 (Matt) : Logistic regression
[Slides] [Video]
Logistic regression vs. linear regression, Sigmoid funcion, MLE via gradient ascent, Regularization, Logistic regression for multiple classes
  • Bishop 4.2-4.3

Mon, 9/26/16 Lecture 6 (Eric) : SVM
[Slides] [Annotated Slides] [Video]
Support vector machines, Primal and dual versions of SVM, Duality, KKT conditions

HW2 Out

HW1 Due

Wed, 9/28/16 Lecture 7 (Eric) : Kernels & The Kernel Trick
[Slides] [Annotated Slides] [Video]
Kernel trick, [SVM Optimization e.g. Sequential Minimal Optimization (SMO)]

Mon, 10/3/16 Lecture 8 (Eric) : Ensemble learning - Boosting, Random Forests
[Slides] [Annotated Slides] [Video]
Combing weak learners, Bagging and random forest, AdaBoost, Algorithm and generalization bounds, Gradient boosting

Proposal Due


Wed, 10/5/16 Lecture 9 (Eric) : Learning theory
[Slides] [Annotated Slides] [Video]
Realizable vs agnostic, PAC learning in finite concept class, Sample complexity

Mon, 10/10/16 Lecture 10 (Eric) : VC dimension
[Slides] [Annotated Slides] [Video]
Sample complexity for infinite concept classes, VC dimension as a complexity measure, Structural risk minimization
  • Ch3, An Introduction to Computational Learning Theory, M. Kearns and U. Vazirani

HW3 Out

HW2 Due

Wed, 10/12/16 Lecture 11 (Eric) : Evaluating classifiers, Bias-variance decomposition
[Slides] [Annotated Slides] [Video]
Bias-variance decomposition, Structural risk minimization, Ways to avoid overfitting

Supervised Learning

Mon, 10/17/16 Lecture 12 (Matt) : Perceptron, Neural networks
[Slides] [Video]
Perceptron, Multilayer Perceptron, Backpropagation

Wed, 10/19/16 Lecture 13 (Matt) : Deep Learning
[Slides] [Video]
"Deep" Learning, Convolutional Neural Networks, Layer-wise Pre-training

Unsupervised Learning

Mon, 10/24/16 Lecture 14 (Matt) : PCA and dimension reduction
[Slides] [Video]
Principal component analysis, Dimensionality reduction
  • Murphy, Ch12
  • Bishop, Ch12

HW4 Out

HW3 Due

Wed, 10/26/16 Lecture 15 (Eric) : K-means
[Slides] [Video]
Hierarchical clustering, K-means and Gaussian mixture models, Number of clusters
  • Bishop Ch 9

Mon, 10/31/16 Lecture 16 (Eric) : EM
[Slides] [Annotated Slides] [Video]
Expectation Maximization

Wed, 11/2/16 Midterm

Probabilistic Modeling

Mon, 11/7/16 Lecture 17 (Eric) : Graphical models, Bayes nets
[Slides] [Annotated Slides] [Video]

Midway Report Due

Wed, 11/9/16 Lecture 18 (Eric) : Inference and learning of graphical models
[Slides] [Annotated Slides] [Video]
Learning, Exact Inference
  • Bishop, 8.4

Mon, 11/14/16 Lecture 19 (Matt) : HMMs and CRFs
[Slides] [Video]
Directed vs. undirected, Undirected graphical models, Conditional random fields

HW5 Out

HW4 Due

Wed, 11/16/16 Lecture 20 (Matt) : HMMs and CRFs (continued)
[Slides] [Video]

Mon, 11/21/16 Lecture 21 (Matt) : Topic modeling and Approximate Inference
[Slides] [Video]
Advanced probabilistic modeling, Approximate inference, MCMC

Wed, 11/23/16 Lecture (No Class: Thanksgiving) :

Advanced Topics

Mon, 11/28/16 Lecture 22 (Eric) : Distributed ML
[Slides] [Video]

HW5 Due

Wed, 11/30/16 Lecture 23 (Matt) : MDPs, Reinforcement learning
Markov decision processes, Value iteration, Policy iteration, Q-Learning

Poster Session

Fri, 12/2/16 Lecture 25 : NSH 3305
2:30 - 5:30 pm

Project Report

Fri, 12/9/16 Project Report Due