Date	Plan	Readings
08/29 Fri	Week 1: Course Introduction [ slides ]	(Required) How to Read a Paper
09/03 Wed	Week 2: Introduction to ML Systems [ slides ]	(Required) TensorFlow: A System for Large-Scale Machine Learning (Required) PyTorch: An Imperative Style, High-Performance Deep Learning Library
09/05 Fri	Week 2: GPU Architecture and CUDA Programming (1) [ slides ]
09/10 Wed	Week 3: GPU Architecture and CUDA Programming (2) [ slides ]
09/12 Fri	Week 3: Advanced CUDA Programming: Warp Specialization [ slides ]
09/17 Wed	Week 4: Advanced CUDA Programming: Mega Kernel [ slides ]
09/19 Fri	Week 4: ML Compilation (1) [ slides ]	(Required) TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (Required) Halide: Decoupling Algorithms from Schedules for Easy Optimization of Image Processing Pipelines (Optional) MLIR: A Compiler Infrastructure for the End of Moore's Law
09/24 Wed	Week 5: ML Compilation (2) [ slides ]	(Required) Learning to Optimize Tensor Programs (Required) Ansor: Generating High-Performance Tensor Programs for Deep Learning (Optional) Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
09/26 Fri	Week 5: Advanced ML Compilation: Graph-Level Optimizations [ slides ]	(Required) TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions (Required) PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections (Optional) Equality Saturation for Tensor Graph Superoptimization
10/01 Wed	Week 6: Advanced ML Compilation: Superoptimization [ slides ]	(Required) Mirage: A Multi-Level Superoptimizer for Tensor Programs
10/03 Fri	Week 6: ML Parallelization: Data Parallelism and Zero Redundancy [ slides ]	(Required) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Required) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training
10/08 Wed	Week 7: ML Parallelization: Model and Pipeline Parallelism [ slides ]	(Required) PipeDream: Generalized Pipeline Parallelism for DNN Training (Required) GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Optional) Memory-Efficient Pipeline-Parallel DNN Training
10/10 Fri	Week 7: Advanced ML Parallelization: Automated ML Parallelization [ slides ]	(Required) Beyond Data and Model Parallelism for Deep Neural Networks (Required) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Optional) GSPMD: General and Scalable Parallelization for ML Computation Graphs (Optional) Ray: A Distributed Framework for Emerging AI Applications
10/15 Wed	Week 8: Fall Break; No Class [ slides ]
10/17 Fri	Week 8: Fall Break; No Class [ slides ]
10/22 Wed	Week 9: LLMs: Transformers + Attention [ slides ]	(Required) Transformer: Attention is All you Need (Required) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Optional) Language Models are Unsupervised Multitask Learners
10/24 Fri	Week 9: LLMs: FlashAttention, PagedAttention, and RadixAttention [ slides ]	(Required) Fast and Memory-Efficient Exact Attention with IO-Awareness (Required) Efficient Memory Management for Large Language Model Serving with PagedAttention
10/29 Wed	Week 10: LLMs: Serving (speculative decoding, continuous batching) [ slides ]	(Required) Fast Inference from Transformers via Speculative Decoding (Required) Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
10/31 Fri	Week 10: LLMs: Training (Parameter-Efficient Fine-Tuning) [ slides ]	(Required) LoRA: Low-Rank Adaptation of Large Language Models (Required) QLoRA: Efficient Finetuning of Quantized LLMs
11/05 Wed	Week 11: Meetings to discuss project ideas [ slides ]
11/07 Fri	Week 11: LLMs: retrieval-augmented generation [ slides ]
11/12 Wed	Week 12: LLMs: Mixture of Experts [ slides ]
11/14 Fri	Week 12: Guest Lecture: Disruptive Research on ML Systems [ slides ]
11/19 Wed	Week 13: Working on course project [ slides ]
11/21 Fri	Week 13: Working on course project [ slides ]
11/26 Wed	Week 14: Thanksgiving Break; No Class [ slides ]
11/28 Fri	Week 14: Thanksgiving Break; No Class [ slides ]
12/03 Wed	Week 15: Final Project Presentation [ slides ]
12/05 Fri	Week 15: Final Project Presentation