15-779 | Schedule

Date	Plan	Readings
08/29 Fri	Week 1: Course Introduction [ slides ]	(Required) How to Read a Paper
09/03 Wed	Week 2: Introduction to ML Systems [ slides ]	(Required) TensorFlow: A System for Large-Scale Machine Learning (Required) PyTorch: An Imperative Style, High-Performance Deep Learning Library
09/05 Fri	Week 2: GPU Architecture and CUDA Programming (1) [ slides ]
09/10 Wed	Week 3: GPU Architecture and CUDA Programming (2) [ slides ]
09/12 Fri	Week 3: Case Study: Transformers, Attention, FlashAttention [ slides ]	(Required) Transformer: Attention is All you Need (Required) Fast and Memory-Efficient Exact Attention with IO-Awareness (Optional) Language Models are Unsupervised Multitask Learners
09/17 Wed	Week 4: Advanced CUDA Programming: Warp Specialization [ slides ]
09/19 Fri	Week 4: Advanced CUDA Programming: Mega Kernel [ slides ]	(Required) Designing a Low-Latency Megakernel for Llama-1B (Required) Compiling LLMs into a MegaKernel
09/24 Wed	Week 5: ML Compilers Part 1: Tile-based DSL [ slides ]	(Required) Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations (Required) TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (Optional) MLIR: A Compiler Infrastructure for the End of Moore's Law
09/26 Fri	Week 5: ML Compilation: Kernel Auto-tuning [ slides ]	(Required) Learning to Optimize Tensor Programs (Required) Ansor: Generating High-Performance Tensor Programs for Deep Learning (Optional) Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
10/01 Wed	Week 6: Advanced ML Compilation: Graph-Level Optimizations [ slides ]	(Required) TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions (Required) PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections (Optional) Equality Saturation for Tensor Graph Superoptimization
10/03 Fri	Week 6: Advanced ML Compilation: ML Superoptimization [ slides ]	(Required) Mirage: A Multi-Level Superoptimizer for Tensor Programs
10/08 Wed	Week 7: ML Parallelization: Data Parallelism [ slides ]	(Required) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Required) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training
10/10 Fri	Week 7: ML Parallelization: Zero Redundancy and PyTorch Fully Sharded Data Parallelism [ slides ]	(Required) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Required) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training
10/15 Wed	Week 8: Fall Break; No Class [ slides ]
10/17 Fri	Week 8: Fall Break; No Class [ slides ]
10/22 Wed	Week 9: ML Parallelization: Model and Pipeline Parallelism [ slides ]	(Required) PipeDream: Generalized Pipeline Parallelism for DNN Training (Required) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Optional) Memory-Efficient Pipeline-Parallel DNN Training
10/24 Fri	Week 9: Advanced ML Parallelization: Automated ML Parallelization [ slides ]	(Required) Beyond Data and Model Parallelism for Deep Neural Networks (Required) Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
10/29 Wed	Week 10: LLMs: Serving (Batching, PagedAttention, RadixAttention) [ slides ]	(Required) Efficient Memory Management for Large Language Model Serving with PagedAttention (Required) SGLang: Efficient Execution of Structured Language Model Programs
10/31 Fri	Week 10: LLMs: Serving (Speculative Decoding) [ slides ]	(Required) Fast Inference from Transformers via Speculative Decoding (Required) Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
11/05 Wed	Week 11: LLMs: Training (Parameter-Efficient Fine-Tuning) [ slides ]	(Required) LoRA: Low-Rank Adaptation of Large Language Models (Required) QLoRA: Efficient Finetuning of Quantized LLMs
11/07 Fri	Week 11: Meetings to discuss project ideas [ slides ]
11/12 Wed	Week 12: Guest Lecture: New Problems in LLM Post-training Systems [ slides ]		Banghua Zhu (UW/NVIDIA)
11/14 Fri	Week 12: Guest Lecture: Advancing the Pareto Frontier of Training Open Language Models [ slides ]		Mengzhou Xia (OpenAI/CMU)
11/19 Wed	Week 13: LLMs: Mixture-of-Experts (Architectures, Kernels, Parallelism) [ slides ]
11/21 Fri	Week 13: Working on course project [ slides ]
11/26 Wed	Week 14: Thanksgiving Break; No Class [ slides ]
11/28 Fri	Week 14: Thanksgiving Break; No Class [ slides ]
12/03 Wed	Week 15: [ slides ]
12/05 Fri	Week 15:

15-779: Advanced Topics in Machine Learning Systems (LLM Edition)

Schedule