10-702 Statistical Machine Learning

Instructors: John Lafferty and Larry Wasserman
Time: MW 3:00-4:20
Place: Wean 5409

TA: Jure Leskovec
Office hours: Thursday 4:00-6:00
Place: Wean 5123 (or Wean 4616)

Course secretary: Amelia Williams
Office: Wean 4114

Course description

The course combines machine learning methodology with theoretical foundations---it treats both the "art" of designing good learning algorithms, and the "science" of analyzing an algorithm's statistical properties and performance guarantees. Theorems are presented together with practical aspects of methodology and intuition to help students develop tools for selecting appropriate methods and approaches to problems in their own research.

The course covers both Bayesian and non-Bayesian techniques, and parametric and nonparametric methods. Theoretical aspects include topics in statistical theory that are now becoming important for researchers in machine learning, including consistency, minimax estimation, concentration of measure, empirical processes, and a theoretical treatment of semi-supervised learning. The course also presents topics in computation that are not part of standard statistics courses, including dynamic programming, elements of convex optimization, structured variational methods, randomized projection algorithms, and techniques for handling large data sets.

A special focus topic is sparsity, which is an essential concept for modern statistical methods applied to very high dimensional data. Also included are case studies of statistical machine learning applied to practical problems in text analysis, image processing, biological sequence analysis, and astronomy.

Prerequisites

Machine Learning 10-701 and Intermediate Statistics 36-705, or Probability and Statistics 36-725 and 36-726. Students who have not completed the statistics prerequisite can take the Intermediate Statistics final to demonstrate competence with the material.

The course document includes information about assignments, exams and grading.

Lecture Notes

There is no required text for the course; however, lecture notes will be regularly distributed (but not posted on the web). These are draft chapters and sections from a book in progress.
Comments, corrections, and other input on the drafts are highly encouraged.

Assignments

Assignments are due on Fridays at 5:00 p.m. You can hand in the assignment at course secretary's office (Amelia Williams) in Wean Hall 4114.

• Assignment 1: due: Friday, Feb 2, 5pm.
• Assignment 2: due: Friday, Feb 16, 5pm.
• Assignment 3: due: Friday, Mar 2, 5pm.
  Data for problem 6:
  • Training set
  • Evaluation set
  • Test set -- you have to fill-in Y values
• Assignment 4: due: Friday, Apr 6, 5pm.
  
  • tree.txt
• Assignment 5: due: Friday, Apr 20, 5pm.
  
  • xy.Rdata
  • flies.txt
• Assignment 6: due: Friday, May 4, 5pm.
  
  • homer.txt
  • Data for problem 3

Code and other files

• lasso.r
• Support Vector Machines with Applications

Outline of Topics

Topics will be chosen from the following basic outline, as announced in class.

• Background
• Probability
• Basic statistical theory
• Function spaces
• Convexity
• Matrix computations
• Numerical optimization
• Algorithms
• Parametric Methods
• Linear regression
• Model selection
• Generalized linear models
• Mixture models
• Classification
• Graphical models
• Clustering
• Dimension reduction methods
• Structured classification
• Nonparametric Methods
• Nonparametric regression
• Multivariate nonparametric regression
• Kernels
• Density estimation
• Gaussian Processes
• Nonparametric graphical models
• Manifold methods
• The bootstrap and subsampling
• Nonparametric Bayes
• Hybrid methods: Multiscale analysis and graphical models
• Nonparametric time series
• Computation
• The EM algorithm
• Simulation and Markov Chain Monte Carlo
• Particle filtering
• Variational methods
• Regularization path methods
• Recursive dyadic partitioning
• Algorithms for dimension reduction
• Graph algorithms
• Spectral methods
• Handling large data sets
• Advanced Statistical Theory
• Consistency
• Minimax theory
• Concentration of measure
• Empirical processes
• Sparsity
• Basis pursuit and the lasso
• Greedy algorithms for sparse linear estimation
• Sparsity in nonparametric regression: The rodeo
• Sparsity in graphical models
• Compressed sensing
• Other learning paradigms
• Semi-supervised learning
• Active learning and experimental design