ADVANCED MACHINE LEARNING

10-716, Spring 2019
POS 160, Tue & Thurs 1:30PM - 2:50PM

Instructor Pradeep Ravikumar (pradeepr at cs dot cmu dot edu)
Teaching Assistants Ritika Mulagalapalli (rmulagal at andrew dot cmu dot edu)
Leqi Liu (leqil at cs dot cmu dot edu)
Boxiang (Shawn) Lyu (blyu at andrew dot cmu dot edu)
Karthika Nair (knair at andrew dot cmu dot edu)
Biswajit Paria (bparia at cs dot cmu dot edu)
Yao-Hung Tsai (yaohungt at cs dot cmu dot edu)
Office Hours Pradeep Ravikumar: Thursday: 3:00pm - 4:00pm in GHC 8111
Shawn Lyu: Tuesdays 10:00am - 11:00am outside GHC 8009
Leqi Liu: Wednesdays 10:45am - 11:45am outside GHC 8009
Karthika Nair: Wednesdays 3:00pm - 4:00pm outside GHC 8009
Ritika Mulagalapalli: Mondays 11:00am - 12:00pm outside GHC 8009
Yao-Hung Tsai: Fridays 4:00pm - 5:00pm outside GHC 8009
Biswajit Paira: Thursdays 5:00pm - 6:00pm outside GHC 8009

Course Description Advanced Machine Learning is a graduate level course introducing the theoretical foundations of modern machine learning, as well as advanced methods and frameworks used in modern machine learning. The course assumes that students have taken graduate level introductory courses in machine learning (Introduction to Machine Learning, 10-701 or 10-715), as well as Statistics (Intermediate Statistics, 36-700 or 36-705). The course treats both the art of designing good learning algorithms, as well as the science of analyzing an algorithm's computational and statistical properties and performance guarantees. We will cover theoretical foundation topics such as computational and statistical convergence rates, minimax estimation, and concentration of measure. We will also cover advanced machine learning methods such as nonparametric density estimation, nonparametric regression, and Bayesian estimation, as well as advanced frameworks such as privacy, causality, and stochastic learning algorithms.

Grading 50% Homeworks, 15% Exam 1, 15% Exam 2, 20% Project

Textbooks Lectures are intended to be self-contained. The following references might be useful:
  • JB: Statistical Decision Theory and Bayesian Analysis, by James O. Berger
  • MW: High-Dimensional Statistics: A Non-Asymptotic Viewpoint, by Martin J. Wainwright
  • BL: Prediction, Learning, and Games, by Nicolo Cesa-Bianchi, Gabor Lugosi
  • W: All of Nonparametric Statistics, by Larry Wasserman
  • AB: Computational Complexity: A Modern Approach, by Sanjeev Arora, Boaz Barak
  • N: Introductory Lectures on Convex Optimization, by Yurii Nesterov


    		•
    Course details Syllabus. Piazza. Homeworks. Project. Scribe signup.

    Tentative Schedule Lecture Notes
    Date Topic Readings Notes
    Module: Statistical Decision Theory
    Jan 15 Decision Theory Principles and Paradigms
    Lecture Notes     		JB Chap 1
    Jan 17 Decision Theory Principles Contd.
    Lecture Notes     		JB Chap 4,2
    Jan 22 Bayesian Analysis
    Lecture Notes     		JB Chap 5
    Jan 24 Minimax Analysis
    Lecture Notes    		 JB Chap 5 HW1 out
    Module: Statistical Complexity
    Jan 29 Empirical Risk Minimization and Decision Theory, Tail Bounds
    Lecture Notes     		MW Chap 2
    Jan 31 No class: Polar Vortex
    Feb 5 Tail Bounds Contd.
    Lecture Notes     		MW Chap 1,2 HW2 out, HW1 due (Solutions)
    Feb 7 Tail Bounds Contd.
    Lecture Notes     		MW Chap 1,2
    Feb 12 Uniform Laws, Complexity Measures
    Lecture Notes     		MW Chap 4
    Feb 14 Uniform Laws, Complexity Measures Contd.
    Lecture Notes     		MW Chap 4
    Feb 15 HW2 due
    Feb 19 Review Session
    Feb 21 Test 1
    Feb 26 Sparse Linear Models
    Lecture Notes     		MW Chap 7 HW3 out, Project Proposal Due
    Feb 28 Sparse Linear Models Contd.
    Lecture Notes     		MW Chap 7
    Mar 5 Sparse Linear Models Contd.
    Lecture Notes     		MW Chap 7
    Mar 7 Lower bounds
    Lecture Notes    		 MW Chap 15
    Mar 8 HW3 due
    Mar 12 No Class: Spring Break
    Mar 14 No Class: Spring Break
    Mar 19 Lower Bounds
    Lecture Notes    		 MW Chap 15
    Module: Computational Complexity
    Mar 22 HW4 out
    Mar 26 Optimization and Statistical Complexity
    Lecture Notes    		 Papers: [1], [2], [3], [4]
    Mar 28 Optimization and Statistical Complexity Contd.
    Lecture Notes    		 Papers: [1], [2], [3], [4]
    Apr 2 Intro to Computational Complexity, Oracle Complexity
    Lecture Notes    		 AB Chap 1, 2

    Papers: [1], [2]
    Apr 4 Stat. vs Complexity tradeoffs
    Lecture Notes    		 Papers: [1], [2], [3] HW4 due
    Apr 8 HW5 out
    Module: Sequential Settings
    Apr 9 Prediction with expert advice
    Lecture Notes    		 BL Chap 1, 2
    Apr 11 No Class: Spring Carnival Project progress report due
    Apr 16 Prediction with expert advice contd.
    Lecture Notes    		 BL Chap 1, 2
    Apr 18 Prediction with expert advice contd.,
    Prediction with limited feedback.
    Lecture Notes    		 BL Chap 1, 2, 6
    Apr 23 Prediction with limited feedback contd.
    Lecture Notes    		 BL Chap 6
    Apr 24 HW5 due
    Apr 25 Prediction and games.
    Lecture Notes    		 BL Chap 7
    Apr 30 Review Session
    Lecture Notes
    May 2 Test 2
    May 7 Final Project Report Due

    Homeworks There will be 5 homework assignments, approximately evenly spaced throughout the semester. The assignments will be posted on the course website, and on Piazza. We will use Gradescope for submitting, and grading assignments. You will get a late day quota of 10 days, which you can distribute among the five homeworks as you wish, subject to a maximum of 3 days per homework. Homeworks submitted after your late day quota will lose all points. The homework schedule is posted right at the beginning of the semester, so please plan in advance. We expect you to use the late day quota for conference deadlines and events of the like, so we cannot provide an additional extension for such cases. In the case of an emergency (sudden sickness, family problems, etc.), we can give you a reasonable extension. But we emphasize that this is reserved for true emergencies.

    Class Project There will be a class project. You can form groups of up to 2 students. Further details can be found here.