Generative AI

• Sequence Modeling: Recurrent and Recursive Nets. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.1-10.5.
• A Framework for the Cooperation of Learning Algorithms. Léon Bottou, Patrick Gallinari (1991). Advances in Neural Information Processing Systems.
• PyTorch: An Imperative Style, High-Performance Deep Learning Library. Paszke et al. (2019). Advances in Neural Information Processing Systems.

• The Long Short-Term Memory and Other GatedRNNs. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.10-10.12.
• Attention Is All You Need. Vaswani et al. (2017). NeurIPS.
• The Illustrated Transformer. Alammar (2018).

HW0 out

• Generating Sequences With Recurrent Neural Networks. Graves (2014).
• Recurrent neural network based language model. Mikolov et al. (2010). Interspeech.
• [GPT-1] Improving Language Understanding by Generative Pre-Training. Radford et al. (2018).
• [GPT-2] Language Models are Unsupervised Multitask Learners. Radford et al. (2019).

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Ainslie et al. (2023). EMNLP.
• Longformer: The Long-Document Transformer. Beltagy et al. (2020).
• RoFormer: Enhanced Transformer with Rotary Position Embedding. Su et al. (2021).

HW0 due

HW1 out (L1-L4)

• Convolutional Networks. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 9.1 - 9.3.
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. (2018). NAACL.
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Dosovitskiy et al. (2021). ICLR.

Quiz 1 (in-class, L1-L4)

• Generative Adversarial Nets. Goodfellow et al. (2014). NeurIPS.
• NuerIPS 2016 Tutorial: Generative Adversarial Networks. Goodfellow (2017). NeurIPS Tutorials.

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Sohl-Dickstein et al. (2015). MLR.
• Denoising Diffusion Probabilistic Models. Ho et al. (2020). NeurIPS.

• Variational Inference: A Review for Statisticians. Blei, Kucukelbir, and McAuliffe (2018).
• High-Level Explanation of Variational Inference. Jason Eisner (2011).
• Tutorial on Variational Autoencoders. Carl Doersch (2016).

HW1 due

HW2 out (L5-L8)

Quiz 2 (in-class, L5-L8)

Project description out

HW2 due

HW3 out (L9-L11)

(Quiz 3 in-class, L9-L11)

HW3 due

HW4 out (L12-L14)

(Quiz 4 in-class, L12-L15)

HW4 due

HW623 out; Practice problems out

(Quiz 5 in-class, L16-L20)

Project proposal due

Project midway report due

HW623 due

Project poster due

Project final report due