10-708 – Lectures (tentative)

2020 Spring
Lecture Date Topic Slides Videos Further Reading Note Scribe
Design of GMs
01 Jan 13 Introduction to GM: (Eric)
- Association between random variables
- Marginal/partial correlation
- Conditional independence
pdf panopto,
youtube
Jordan 2004, Airoldi 2007,
Larry's notes
pdf
02 Jan 15 Undirected GMs: (Eric)
- Markov property
pdf panopto,
youtube
Koller & Friedman Ch. 4 pdf
Jan 20 MLK day
03 Jan 22 Directed GMs: (Eric)
- Markov property
pdf panopto,
youtube
Koller & Friedman Ch. 3 hw1 out pdf
Basic Inference and Learning
04 Jan 27 Exact Inference: (Eric)
- Variable elimination
- Sum-product on trees
- Belief propagation on junction trees
pdf panopto,
youtube
Jordan Ch. 3, Ch. 4,
Koller & Friedman Ch. 9, Ch. 10
pdf
05 Jan 29 Parameter Estimation: (Eric)
- Fully observed: MLE, MAP, Bayesian
- Exponential family distributions, GLMs
- Partially observed: EM algorithm
pdf panopto,
youtube
Koller & Friedman Ch. 17.1-4,
Ch. 19.1-4,
Neal & Hinton 1998
pdf
06 Feb 03 Case Studies: HMM and CRF (Eric) pdf panopto,
youtube
Koller & Friedman Ch. 6.2,
Wallach 2004,
Lafferty, McCallum, Pereira 2001
pdf
Approximate Inference
07 Feb 05 Variational Inference 1: (Eric)
- Variational methods
- LDA
pdf panopto,
youtube
Wainwright, Jordan 2008 pdf
08 Feb 10 Variational Inference 2: (Xun)
- Stochastic/Black-box VI
- VI Theory
pdf panopto,
youtube
Wainwright, Jordan 2008 pdf
09 Feb 12 Sampling 1: (Eric)
- Accept-reject sampling
- Importance sampling
- Metropolis-Hastings
- Gibbs sampling
pdf panopto,
youtube
MacKay 2003, Ch. 29.1-3 hw1 due pdf
10 Feb 17 Sampling 2: (Eric)
- Hamiltonian Monte Carlo
- Langevin dynamics
- Sequential Monte Carlo
pdf panopto,
youtube
MacKay 2003, Ch. 29.4-10 pdf
Deep Learning and Deep Generative Models
11 Feb 19 Foundations of Deep Learning: (Eric)
- Insight into DL
- Connectionss to GM
pdf panopto,
youtube
Goodfellow, Bengio, Courville 2016
Ch. 6.2-5, 20.3-4
proposal due, hw2 out pdf
12 Feb 24 Deep Generative Models 1: (Eric)
- Wake-sleep algorithm
- Variational autoencoder
- Generative adversarial networks
pdf panopto,
youtube
Goodfellow, Bengio, Courville 2016
Ch. 20.9-10
Mohamed et al. 2019
pdf
13 Feb 26 Deep Generative Models 2: (Eric)
- More GANs and variants
- Normalizing flows
- Integrating domain knowledge in DL
pdf panopto,
youtube
Arjovsky, Bottou 2017,
Papamakarios et al. 2019,
Hu et al. 2018
pdf
14 Mar 02 Deep Sequence Models: (Zhiting)
- RNN and LSTM
- CNN and Transformers
- Attention mechanisms
pdf panopto,
youtube
Pascanu, Mikolov, Bengio 2013,
Vaswani et al. 2017,
Devlin et al. 2018
pdf
15 Mar 04 Case Study: Text Generation (Zhiting)
- Encoder-decoder framework
- Machine translation as conditional generation
- Unifying MLE and RL for text generation
pdf panopto,
youtube
Ranzato et al. 2015,
Hu et al. 2017
hw2 due, hw3 out pdf
Mar 09 Spring break
Mar 11 Spring break
Structure and Causal Inference
Mar 16 No class. Stay healthy!
16 Mar 18 Structure Learning (Eric):
- Undirected GM: Gaussian GM
- Directed: Causal discovery
pdf panopto,
youtube
Meinshausen, B├╝hlmann 2006,
Kolar et al. 2010
pdf
17 Mar 23 Causality 1: (Kun Zhang) pdf panopto,
youtube
Pearl et al. 2016 pdf
18 Mar 25 Causality 2: (Kun Zhang) pdf panopto,
youtube
Spirtes et al. 1993,
Zhang et al. 2017
pdf
RL as Inference in GMs
19 Mar 30 RL as Inference 1 (Maruan) pdf panopto,
youtube
Sutton, Barto Ch. 3-4,
Lilian Weng blog post,
Levine Sec. 1-4,
Ziebart Ch. 5.1-2, 6.1-2
pdf
20 Apr 01 RL as Inference 2 (Maruan) pdf panopto,
youtube
hw3 due, hw4 out pdf
RL as Inference in GMs
21 Apr 06 Gaussian Process (Eric) pdf panopto,
youtube
pdf
22 Apr 08 Determinantal Point Process (Pengtao) panopto,
youtube
midway report due pdf
Bayesian Nonparametrics
23 Apr 13 Dirichlet Process (Eric) pdf panopto,
youtube
pdf
24 Apr 15 Indian Buffet Process (Eric) pdf panopto,
youtube
pdf
Applications and Systems
25 Apr 20 Spectral Graphical Models (Eric) pdf panopto,
youtube
26 Apr 22 Large-scale Algorithms and Systems (Qirong) pdf panopto,
youtube
27 Apr 27 Meta-Learning (Maruan) pdf panopto,
youtube
28 Apr 29 Robust Machine Learning (Haohan) pdf panopto,
youtube
final report due


Video playlists: Panopto, Youtube

Candidates for open slots:

  • Structure Learning for Markov Networks

  • Theory of Variational Inference

  • Spectral and Kernel GMs

  • Max-margin GMs

  • Regularized Bayesian Learning

  • Meta-learning and Neural Process

  • ...