Syllabus and (tentative) Course Schedule

Date Lecture Topics Readings and useful links
Block 1: Supervised Learning
Mon 9/8 Lecture 1: Regression: Linear and Logistic
Eric Xing: slides, annotated slides
No Reading assignment due
Bishop, PRML: Ch 4, Ch 5
Mitchell: Ch 4
Chapter (Draft) from Mitchell
On Discriminative and Generative Classifiers Ng, Jordan

Wed 9/10 Lecture 2: Linear Regression and Lasso
Eric Xing: slides annotated slides
Tutorial on Regression by Andrew Moore
Bishop, PRML: Ch 3
Mitchell: Ch 8.3
Regression Shrinkage and Selection via the Lasso by Rob Tibshirani
Model Selection and Estimation in Regression with Grouped Variables by Yuan, Lin
Large Scale Online Learning by Bottou, Le Cun
Feature Selection for High-Dimensional Genomic Microarray Data by Xing, Jordan, Karp
On Model Selection Consistency of Lasso by Peng Zhao, Bin Yu
Mon 9/15 Lecture 3: Structured sparsity with application in Computational Genomics
Eric Xing: slides
Statistical Estimation of Correlated Genome Associations to a Quantitative Trait Network by S. Kim and E. P. Xing
Tree-Guided Group Lasso for Multi-Response Regression with Structured Sparsity, with applications to eQTL Mapping by S. Kim and E. P. Xing
Smoothing Proximal Gradient Method for General Structured Sparse Regression by X. Chen, Q. Lin, S. Kim, J. Carbonell and E. P. Xing
Wed 9/17 Lecture 4: Perceptron, Deep Neural Networks
Barnabas Poczos: MultiLayerPerceptron DeepArchitectures
Learning Deep Architectures for AI by Yoshua Bengio
ImageNet Classification with Deep Convolutional Neural Networks by Krizhevsky et. al.
Multilayer Feedforward Networks are Universal Approximators by Kur Hornik
A Logical Calculus of the ideas immanent in Nervous Activity by Warren S. McCulloch, Walter Pitts
The Perceptron: A probabilistic model for information storage and organization in the brain by F. Rosenblatt
Optional: Some slides by Eric on Learning DNNs.
Homework 1 out
Block 2: Kernel Machines
Mon 9/22 Lecture 5: SVMs and Duality
Barnabas Poczos: SupportVectorMachines Duality
  • Burges, Christopher J. C.; A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998
  • Optional
  • Learning with Kernels Support Vector Machines, Regularization, Optimization, and Beyond. Bernhard Scholkopf and Alexander J. Smola, MIT Press, 2002.
  • Advances in Kernel Methods - Support Vector Learning Edited by Chris Burges, Bernhard Scholkopf and Alexander J. Smola, MIT Press, 1998.
  • Support Vector Machines and other kernel-based learning methods John Shawe-Taylor and Nello Cristianini - Cambridge University Press, 2000
Wed 9/24 Lecture 6: The Kernel Trick & RKHS Eric Xing: slides annotated slides Required
Learning with Kernels, Scholkof & Smola, Ch 2
Mon 9/29 Lecture 7: Reproducing Kernel Hilbert Space
Eric Xing
Wed 10/1 Lecture 8: Learning with Kernels
Barnabas Poczos
Homework 1 due
Homework 2 out
Block 3: Unsupervised Learning, Density estimation, Graphical Models
Mon 10/6 Lecture 9: Clustering, mixture models, the EM algorithm
Barnabas Poczos: slides
Max Welling's notes on Clustering and EM.
Wed 10/8 Lecture 10: Clustering, mixture models, the EM algorithm
Barnabas Poczos: Slides same as above
Mon 10/13 Lecture 11: Structured Models: Hidden Markov Models vs. Conditional Random Fields
Eric Xing: slides
Chap. 12 from Michael Jordan's book Chap 12
Shallow Parsing with Conditional Random Fields
Rabiner, Lawrence R. (1989). A Tutorial on Hidden Markov Model and selected Applications in Speech Recognition
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Wed 10/15 Lecture 12: Structured Models: Hidden Markov Models vs. Conditional Random Fields, Graphical Models
Eric Xing: CRF GraphicalModels
Michael Jordan's Introduction to Graphical Models
Homework 2 due
Mon 10/20 Lecture 13: Graphical Models, Markov Chain Monte Carlo and Topic Models
Eric Xing: slides
Wed 10/22 Lecture 14: Markov Chain Monte Carlo
Eric Xing: slides (cont'd)

Mon 10/27 Mid Term
Block 4: Latent Space Analysis, Eigen space analysis
Wed 10/29 Lecture 15: Principal Component Analysis
Barnabas Poczos: slides
  • A Tutorial on Principal Component Analysis, Jon Shlens. pdf
  • Kernel Principal Components Analyis, Max Welling. pdf
Homework 3 out
Mon 11/3 Lecture 16: Independent Component Analysis
Barnabas Poczos: slides
  • Independent Component Analysis, Aapo Hyvarinen, Erkki Oja. pdf
Wed 11/5 Lecture 17: Independent Component Analysis
Barnabas Poczos: slides continued from previous lecture
Block 5: Bayesian Nonparametrics
Mon 11/10 Lecture 18: Gaussian Processes
Barnabas Poczos: slides
  • Gaussian Processes for Machine Learning, Rasmussen, Williams. Chapter 2

Wed 11/12 Non-parametric Bayesian Models
Eric Xing: slides
  • Bayesian Haplotype Inference via the Dirichlet Process by Xing et al., ICML 2004 pdf
  • Hierarchical Dirichlet Processes by Teh et al., JASA 2006 pdf
  • Markov Chain Sampling Methods for Dirichlet Process Mixture Models by Radford Neal, 2000 pdf
  • A Constructive Definition of Dirichlet Priors by Sethuraman, 1994 pdf
  • Bayesian Density Estimation and Inference using Mixture models by Escobar et al., 1995 pdf

Homework 3 due
Homework 4 out (Fri 11/14)
Mon 11/17 Spectral clustering
Eric Xing: slides
  • Normalized Cuts and Image Segmentation by Shi and Malik, 2000 pdf
  • On Spectral Clustering: Analysis and an Algorithm by Ng, Jordan and Weiss, 2001 pdf

Block 6: Computational Learning theory
Wed 11/19 Risk Minimization
Barnabas Poczos: slides
  • Sections 1-3 from Introduction to Statistical Learning Theory by Bousquet et al. pdf

Homework 4 due (Fri 11/21)
Mon 11/24 VC theory
Barnabas Poczos slides

Wed 11/26 Thanks Giving Holiday
Mon 12/1 Manifold Learning
Barnabas Poczos slides
  • Spectral Methods for Dimensionality Reduction by Saul et. al pdf
  • A Global Geometric Framework for Nonlinear Dimensionality Reduction by Tenenbaum et. al pdf
  • Nonlinear dimensionality reduction by locally linear embedding by Roweis et. al pdf
  • Laplacian eigenmaps for dimensionality reduction and data representation by Belkin et. al pdf

Block 7: Ensemble methods
Wed 12/3 Boosting, random forests
Additional: If we have more time
Online Learning
Local linear embedding, and manifold learning

© 2012 Eric Xing @ School of Computer Science, Carnegie Mellon University
[validate xhtml]