Generative AI

Important Notes

This schedule is tentative and subject to change. Please check back often.

You can access the OneNote notebook containing all whiteboards from lecture/recitation here. The PDF version of each whiteboard is linked below.

Tentative Schedule

Date	Lecture	Readings	Announcements
Generative models of text
Mon, 12-Jan	Lecture 1 : RNN LMs / Autodiff [Slides] [Slides (Inked)]	Sequence Modeling: Recurrent and Recursive Nets. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.1-10.5. A Framework for the Cooperation of Learning Algorithms. Léon Bottou, Patrick Gallinari (1991). Advances in Neural Information Processing Systems. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Paszke et al. (2019). Advances in Neural Information Processing Systems.
Wed, 14-Jan	Lecture 2 : Transformer LMs [Slides]	The Long Short-Term Memory and Other GatedRNNs. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 10.10-10.12. Attention Is All You Need. Vaswani et al. (2017). NeurIPS. The Illustrated Transformer. Alammar (2018).	HW0 out
Fri, 16-Jan	Recitation: HW0 [Handout]
Mon, 19-Jan	(MLK Day - No Class)
Wed, 21-Jan	Lecture 3 : Learning LLMs / Decoding [Slides] [Slides (Inked)]	Generating Sequences With Recurrent Neural Networks. Graves (2014). Recurrent neural network based language model. Mikolov et al. (2010). Interspeech. [GPT-1] Improving Language Understanding by Generative Pre-Training. Radford et al. (2018). [GPT-2] Language Models are Unsupervised Multitask Learners. Radford et al. (2019).
Fri, 23-Jan	(No Recitation)
Mon, 26-Jan	Lecture 4 : Pre-training, fine-tuning / Modern Transformers [Slides] [Slides (Inked)]	GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Ainslie et al. (2023). EMNLP. Longformer: The Long-Document Transformer. Beltagy et al. (2020). RoFormer: Enhanced Transformer with Rotary Position Embedding. Su et al. (2021).	HW0 Slot A due HW1 out (L1-L4)
Generative models of images
Wed, 28-Jan	Lecture 5 : Computer Vision: CNNs / Encoder-only Transformers / Vision Transformers	Convolutional Networks. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 9.1 - 9.3. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. (2018). NAACL. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Dosovitskiy et al. (2021). ICLR.	Quiz 1 (in-class, L1-L4)
Thu, 29-Jan			HW0 Feedback (tentative)
Fri, 30-Jan	Recitation: HW1
Sun, 1-Feb			HW0 Slot B due (tentative)
Mon, 2-Feb	Lecture 6 : Generative Adversarial Networks (GANs) / PGM	Generative Adversarial Nets. Goodfellow et al. (2014). NeurIPS. NuerIPS 2016 Tutorial: Generative Adversarial Networks. Goodfellow (2017). NeurIPS Tutorials.
Wed, 4-Feb	Lecture 7 : Diffusion models (Part I)	Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Sohl-Dickstein et al. (2015). MLR. Denoising Diffusion Probabilistic Models. Ho et al. (2020). NeurIPS.
Fri, 6-Feb	(No Recitation)
Mon, 9-Feb	Lecture 8 : Diffusion models (Part II)	Variational Inference: A Review for Statisticians. Blei, Kucukelbir, and McAuliffe (2018). High-Level Explanation of Variational Inference. Jason Eisner (2011). Tutorial on Variational Autoencoders. Carl Doersch (2016).	HW1 Slot A due HW2 out (L5-L8)
Applying and adapting foundation models
Wed, 11-Feb	Lecture 9 : Variational Autoencoders (VAEs)
Fri, 13-Feb	Recitation: HW2
Sat, 14-Feb			HW1 Feedback (tentative)
Mon, 16-Feb	Lecture 10 : Parameter-efficient fine tuning		Quiz 2 (in-class, L5-L9)
Tue, 17-Feb			HW1 Slot B due (tentative)
Wed, 18-Feb	Lecture 11 : In-Context Learning / Prompt Engineering / Instruction Fine-tuning / Reinforcement learning with human feedback (RLHF)		Project description out
Fri, 20-Feb	Recitation: HW3
Sat, 21-Feb			HW2 Slot A due HW3 out (L9-L12)
Multimodal foundation models
Mon, 23-Feb	Lecture 12 : Direct Preference Optimization (DPO) / Text-to-image generation / Latent diffusion model
Wed, 25-Feb	Lecture 13 : Vision-language models		(Quiz 3 in-class, L9-L12)
Thu, 26-Feb			HW2 Feedback (tentative)
Fri, 27-Feb	Programming Test HW1/HW2		Project Team Formation form released
Sun, 1-Mar			HW2 Slot B due (tentative)
Mon, 2-Mar	Spring Break
Tue, 3-Mar
Wed, 4-Mar	Spring Break
Thu, 5-Mar
Fri, 6-Mar	Spring Break
Mon, 9-Mar	Lecture 14 : Latent diffusion model / Diffusion Transformer / Prompt-to-Prompt
Scaling Up
Wed, 11-Mar	Lecture 15 : Querying Transformer / Scaling Laws
Thu, 12-Mar			HW3 Slot A due HW4 out (L12-L14)
Fri, 13-Mar	Recitation: HW4		Project team formation due by 2pm
Mon, 16-Mar	Lecture 16 : Mixture of Experts		(Quiz 4 in-class, L12-L15)
Tue, 17-Mar			HW3 Feedback (tentative)
Wed, 18-Mar	Lecture 17 : Distributed training
Fri, 20-Mar	(No Recitation)		HW3 Slot B due (tentative)
Mon, 23-Mar	Lecture 18 : Flash Attention / Efficient decoding strategies		HW4 Slot A due HW623 out; Practice problems out
Advanced Topics
Wed, 25-Mar	Lecture 19 : Long Context in LLM
Fri, 27-Mar	Programing Test HW3/HW4
Sat, 28-Mar			HW4 Feedback (tentative)
Mon, 30-Mar	Lecture 20 : Reasoning Models
Mon, 30-Mar	Exam (evening exam, details will be announced on Piazza)		HW4 Slot B due (tentative)
Wed, 1-Apr	Lecture 21 : State Space Models / Hybrid Models
Fri, 3-Apr	(No Recitation)		Project proposal due
Mon, 6-Apr	Lecture 22 : Real-world Issues and Considerations / What can go wrong?		(Quiz 5 in-class, L16-L20)
Wed, 8-Apr	Lecture 23 : Code Generation / Autonomous Agents
Thu, 9-Apr	Spring Carnival
Fri, 10-Apr	Spring Carnival
Mon, 13-Apr	Lecture 24 : Audio understanding and synthesis		Project midway report due
Wed, 15-Apr	Lecture 25 : Generative Models for Videos
Fri, 17-Apr	(No Recitation)
Mon, 20-Apr	Lecture 26 : Interactive World Models + Science of Alignment		HW623 due; (Quiz 6 in-class, L21-L24)
Wed, 22-Apr	(No lecture)
Fri, 24-Apr	(No Recitation)
Sun, 26-Apr			Project poster due
Thu, 30-Apr			Project final report due
Apr-27 to May-04	Project Final Presentations (during Final Exam Period -- exact time/date TBD by the registrar, details will be announced on Piazza)		Project final poster/report due

Important Notes

Tentative Schedule

Generative models of text

Generative models of images

Applying and adapting foundation models

Multimodal foundation models

Scaling Up

Advanced Topics