Generative AI

Important Notes

This schedule is tentative and subject to change. Please check back often.

You can access the OneNote notebook containing all whiteboards from lecture/recitation here. The PDF version of each whiteboard is linked below.

Tentative Schedule

Date	Lecture	Readings	Announcements
Generative models of text
Mon, 25-Aug	Lecture 1 : RNN LMs / Autodiff [Slides] [Slides (Inked)]	Sequence Modeling: Recurrent and Recursive Nets. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.1-10.5. A Framework for the Cooperation of Learning Algorithms. Léon Bottou, Patrick Gallinari (1991). Advances in Neural Information Processing Systems. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Paszke et al. (2019). Advances in Neural Information Processing Systems.
Wed, 27-Aug	Lecture 2 : Transformer LMs [Slides]	The Long Short-Term Memory and Other GatedRNNs. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.10-10.12. Attention Is All You Need. Vaswani et al. (2017). NeurIPS. The Illustrated Transformer. Alammar (2018).	HW0 out
Fri, 29-Aug	Recitation: HW0 [Handout]
Mon, 1-Sep	(Labor Day - No Class)
Wed, 3-Sep	Lecture 3 : Learning LLMs / Decoding [Slides]	Generating Sequences With Recurrent Neural Networks. Graves (2014). Recurrent neural network based language model. Mikolov et al. (2010). Interspeech. [GPT-1] Improving Language Understanding by Generative Pre-Training. Radford et al. (2018). [GPT-2] Language Models are Unsupervised Multitask Learners. Radford et al. (2019).
Fri, 5-Sep	(No Recitation)
Mon, 8-Sep	Lecture 4 : Pre-training, fine-tuning / Modern Transformers [Slides] [Slides (Inked)]	GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Ainslie et al. (2023). EMNLP. Longformer: The Long-Document Transformer. Beltagy et al. (2020). RoFormer: Enhanced Transformer with Rotary Position Embedding. Su et al. (2021).	HW0 due HW1 out (L1-L4)
Generative models of images
Wed, 10-Sep	Lecture 5 : Computer Vision: CNNs / Encoder-only Transformers / Vision Transformers [Slides]	Convolutional Networks. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 9.1 - 9.3. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. (2018). NAACL. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Dosovitskiy et al. (2021). ICLR.	Quiz 1 (in-class, L1-L4)
Fri, 12-Sep	Recitation: HW1 [Handout]
Mon, 15-Sep	Lecture 6 : Generative Adversarial Networks (GANs) / PGM [Slides]	Generative Adversarial Nets. Goodfellow et al. (2014). NeurIPS. NuerIPS 2016 Tutorial: Generative Adversarial Networks. Goodfellow (2017). NeurIPS Tutorials.
Wed, 17-Sep	Lecture 7 : Diffusion models (Part I) [Slides] [Slides (Inked)]	Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Sohl-Dickstein et al. (2015). MLR. Denoising Diffusion Probabilistic Models. Ho et al. (2020). NeurIPS.
Fri, 19-Sep	(No Recitation)
Mon, 22-Sep	Lecture 8 : Diffusion models (Part II) / Variational Autoencoders (VAEs)	Variational Inference: A Review for Statisticians. Blei, Kucukelbir, and McAuliffe (2018). High-Level Explanation of Variational Inference. Jason Eisner (2011). Tutorial on Variational Autoencoders. Carl Doersch (2016).	HW1 due HW2 out (L5-L8)
Applying and adapting foundation models
Wed, 24-Sep	Lecture 9 : Variational Inference / In-context learning		Quiz 2 (in-class, L5-L8)
Fri, 26-Sep	Recitation: HW2
Mon, 29-Sep	Lecture 10 : Parameter-efficient fine tuning
Wed, 1-Oct	Lecture 11 : Prompt Engineering / Instruction Fine-tuning / Reinforcement learning with human feedback (RLHF)		Project description out
Fri, 3-Oct	Recitation: HW3
Sat, 4-Oct			HW2 due
Sun, 5-Oct			HW3 out (L9-L11)
Multimodal foundation models
Mon, 6-Oct	Lecture 12 : Direct Preference Optimization (DPO) / Text-to-image generation / Latent diffusion model
Wed, 8-Oct	Lecture 13 : Prompt-to-Prompt		Quiz 3 (in-class, L9-L11)
Fri, 10-Oct	(No Recitation)
Mon, 13-Oct	Fall break
Tue, 14-Oct
Wed, 15-Oct	Fall break
Thu, 16-Oct
Fri, 17-Oct	Fall break
Mon, 20-Oct	Lecture 14 : Vision-language models
Scaling Up
Wed, 22-Oct	Lecture 15 : Scaling Laws
Thu, 23-Oct			HW3 due HW4 out (L12-L14)
Fri, 24-Oct	Recitation: HW4		Project team formation due by 2pm
Mon, 27-Oct	Lecture 16 : Mixture of Experts		(Quiz 4 in-class, L12-L15)
Wed, 29-Oct	Lecture 17 : Distributed training
Fri, 31-Oct	(No Recitation)
Mon, 3-Nov	Lecture 18 : Flash Attention / Efficient decoding strategies		HW4 due HW623 out; Practice problems out
Tue, 4-Nov
Advanced Topics
Wed, 5-Nov	Lecture 19 : Long Context in LLM
Fri, 7-Nov	(No Recitation)
Mon, 10-Nov	Lecture 20 : Reasoning Models
Mon, 10-Nov	Exam (evening exam, details will be announced on Piazza)
Wed, 12-Nov	Lecture 21 : Real-world Issues and Considerations / What can go wrong?
Thu, 13-Nov
Fri, 14-Nov	(No Recitation)
Mon, 17-Nov	Lecture 22 : State Space Models		(Quiz 5 in-class, L16-L20)
Tue, 18-Nov			Project proposal due
Wed, 19-Nov	Lecture 23 : Code Generation / Autonomous Agents
Fri, 21-Nov	(No Recitation)
Mon, 24-Nov	Lecture 24 : Audio understanding and synthesis
Wed, 26-Nov	Thanksgiving Break - No Class
Thu, 27-Nov	Thanksgiving Break - No Class		Project midway report due
Fri, 28-Nov	Thanksgiving Break - No Class
Mon, 1-Dec	Lecture 25 : Generative Models for Videos		HW623 due; (Quiz 6 in-class, L21-L24 for 10-723 only)
Wed, 3-Dec	Lecture 26 : Topic TBD
Fri, 5-Dec	(No Recitation)
Dec-8 to Dec-14	Project Final Presentations (during Final Exam Period -- exact time/date TBD by the registrar, details will be announced on Piazza)		Project final poster/report due

Important Notes

Tentative Schedule

Generative models of text

Generative models of images

Applying and adapting foundation models

Multimodal foundation models

Scaling Up

Advanced Topics