Virginia Smith

Virginia Smith

I am the Leonardo Associate Professor of Machine Learning at Carnegie Mellon University, and a courtesy faculty member in the Electrical and Computer Engineering Department. My research interests are in machine learning, optimization, and distributed systems. Recent topics include: large-scale machine learning, distributed optimization, federated and collaborative learning, multi-task learning, and privacy-preserving ML.


Recent News


PhD Students & Postdocs


Alumni


Teaching


Publications


Preprints

Jogging the Memory of Unlearned LLMs Through Targeted Relearning Attacks
S. Hu, Y. Fu, Z. S. Wu, V. Smith
LLM Unlearning Benchmarks are Weak Measures of Progress
P. Thaker, S. Hu, N. Kale, Y. Maurya, Z. S. Wu, V. Smith
Revisiting Cascaded Ensembles for Efficient Inference
S. Kolawole, D. Dennis, A. Talwalkar, V. Smith
Guardrail Baselines for Unlearning in LLMs
P. Thaker, Y. Maurya, S. Hu, Z. S. Wu, V. Smith
Federated LoRA with Sparse Communication
K. Kuo, A. Raje, K. Rajesh, V. Smith
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
L. Dery, S. Kolawole, J-F. Kagy, V. Smith, G. Neubig, A. Talwalkar
2024

No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices
Q. Pang, S. Hu, W. Zheng, V. Smith
Neural Information Processing Systems (NeurIPS), 2024
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
P. Thaker, A. Setlur, Z. S. Wu, V. Smith
Neural Information Processing Systems (NeurIPS), 2024
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
A. Setlur, S. Garg, X. Geng, N. Garg, V. Smith, A. Kumar
Neural Information Processing Systems (NeurIPS), 2024
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
A. Muhamed, O. Li, D. Woodruff, M. Diab, V. Smith
Empirical Methods in Natural Language Processing (EMNLP), 2024
Prompting is a Double-Edged Sword: Improving Worst-Group Robustness of Foundation Models
A. Setlur*, S. Garg*, V. Smith, S. Levine
International Conference on Machine Learning (ICML), 2024
Maximizing Global Model Appeal in Federated Learning
Y. J. Cho, D. Jhunjhunwala, T. Li, V. Smith, G. Joshi
Transactions on Machine Learning Research (TMLR), 2024
Fair Federated Learning via Bounded Group Loss
S. Hu, Z. S. Wu, V. Smith
IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2024
Best Paper Award at ICLR 2022 Socially Responsible ML Workshop
2023

Progressive Knowledge Distillation: Building Ensembles for Efficient Inference
D. Dennis, A. Shetty, A. Sevekari, K. Koishida, V. Smith
Neural Information Processing Systems (NeurIPS), 2023
Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies
O. Li, J. Harrison, J. Sohl-Dickstein, V. Smith, L. Metz
Neural Information Processing Systems (NeurIPS), 2023
Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift
S. Garg*, A. Setlur*, Z. Lipton, S. Balakrishnan, V. Smith, A. Raghunathan
Neural Information Processing Systems (NeurIPS), 2023
On Tilted Losses in Machine Learning: Theory and Applications
T. Li*, A. Beirami*, M. Sanjabi, V. Smith
Journal of Machine Learning Research (JMLR), 2023
Private Multi-Task Learning: Formulation and Applications to Federated Learning
S. Hu, Z. S. Wu, V. Smith
Transactions on Machine Learning Research (TMLR), 2023
On Noisy Evaluation in Federated Hyperparameter Tuning
K. Kuo, P. Thaker, M. Khodak, J. Ngyuen, D. Jiang, A. Talwalkar, V. Smith
Conference on Machine Learning and Systems (MLSys), 2023
Validating Large Language Models with ReLM
M. Kuchnik, V. Smith, G. Amvrosiadis
Conference on Machine Learning and Systems (MLSys), 2023
Outstanding Paper Award
Differentially Private Adaptive Optimization with Delayed Preconditioners
T. Li, M. Zaheer, K. Liu, S. Reddi, B. McMahan, V. Smith
International Conference on Learning Representations (ICLR), 2023
Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts
A. Setlur, D. Dennis, B. Eysenbach, A. Raghunathan, C. Finn, V. Smith, S. Levine
International Conference on Learning Representations (ICLR), 2023
2022

On Privacy and Personalization in Cross-Silo Federated Learning
Z. Liu, S. Hu, Z. S. Wu, V. Smith
Neural Information Processing Systems (NeurIPS), 2022
Adversarial Unlearning: Reducing Confidence Along Adversarial Directions
A. Setlur, B. Eysenbach, V. Smith, S. Levine
Neural Information Processing Systems (NeurIPS), 2022
Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
S. Wu, T. Li, Z. Charles, Y. Xiao, Z. Liu, Z. Xu, V. Smith
Workshop on Federated Learning at NeurIPS, 2022
Private Adaptive Optimization with Side Information
T. Li, M. Zaheer, S. Reddi, V. Smith
International Conference on Machine Learning (ICML), 2022
Label Leakage and Protection in Two-party Split Learning
O. Li, J. Sun, X. Yang, W. Gao, H. Zhang, J. Xie, V. Smith, C. Wang
International Conference on Learning Representations (ICLR), 2022
Diverse Client Selection for Federated Learning via Submodular Maximization
R. Balakrishnan, T. Li, T. Zhou, N. Himayat, V. Smith, J. Bilmes
International Conference on Learning Representations (ICLR), 2022
Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
M. Kuchnik, A. Klimovic, J. Simsa, V. Smith, G. Amvrosiadis
Conference on Machine Learning and Systems (MLSys), 2022
2021

A Field Guide to Federated Optimization
J. Wang, Z. Charles, Z. Xu, G. Joshi, H. B. McMahan, et al.
On Large-Cohort Training for Federated Learning
Z. Charles, Z. Garrett, Z. Huo, S. Shmulyian, V. Smith
Neural Information Processing Systems (NeurIPS), 2021
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
M. Khodak, R. Tu, T. Li, L. Li, M.-F. Balcan, V. Smith, A. Talwalkar
Neural Information Processing Systems (NeurIPS), 2021
Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
A. Setlur*, O. Li*, V. Smith
Neural Information Processing Systems (NeurIPS), 2021
Progressive Compressed Records: Taking a Byte out of Deep Learning Data
M. Kuchnik, G. Amvrosiadis, V. Smith
Conference on Very Large Data Bases (VLDB), 2021
Ditto: Fair and Robust Federated Learning Through Personalization
T. Li, S. Hu, A. Beirami, V. Smith
International Conference on Machine Learning (ICML), 2021
Best Paper Award at ICLR 2021 Secure ML Workshop
Heterogeneity for the Win: One-Shot Federated Clustering
D. Dennis, T. Li, V. Smith
International Conference on Machine Learning (ICML), 2021
Tilted Empirical Risk Minimization
T. Li*, A. Beirami*, M. Sanjabi, V. Smith
International Conference on Learning Representations (ICLR), 2021
2020

Federated Learning: Challenges, Methods, and Future Directions
T. Li, A. K. Sahu, A. Talwalkar, V. Smith
IEEE Signal Processing Magazine, Special Issue on Distributed Machine Learning, 2020
Fair Resource Allocation in Federated Learning
T. Li, M. Sanjabi, A. Beirami, V. Smith
International Conference on Learning Representations (ICLR), 2020
Learning Context-aware Policies from Multiple Smart Homes via Federated Multi-Task Learning
T. Yu, T. Li, Y. Sun, S. Nanda, V. Smith, V. Sekar, S. Seshan
ACM/IEEE Conference on Internet of Things Design and Implementation (IoTDI), 2020
Federated Optimization in Heterogeneous Networks
T. Li, A. K. Sahu, M. Sanjabi, M. Zaheer, A. Talwalkar, V. Smith
Conference on Machine Learning and Systems (MLSys), 2020
2019

LEAF: A Benchmark for Federated Settings
S. Caldas, P. Wu, T. Li, J. Konecny, B. McMahan, V. Smith, A. Talwalkar
Workshop on Federated Learning for Data Privacy and Confidentiality at NeurIPS, 2019
FedDANE: A Federated Newton-Type Method
T. Li, A. K. Sahu, M. Sanjabi, M. Zaheer, A. Talwalkar, V. Smith
Asilomar Conference on Signals, Systems and Computers, 2019, Invited Paper
A Kernel Theory of Modern Data Augmentation
T. Dao, A. Gu, A. Ratner, V. Smith, C. De Sa, C. Re
International Conference on Machine Learning (ICML), 2019
Efficient Augmentation via Data Subsampling
M. Kuchnik, V. Smith
International Conference on Learning Representations (ICLR), 2019
MLSys: The New Frontier of Machine Learning Systems
Technical Report, 2019
2018 & prior

CoCoA: A General Framework for Communication-Efficient Distributed Optimization
V. Smith, S. Forte, C. Ma, M. Takac, M. I. Jordan, M. Jaggi
Journal of Machine Learning Research (JMLR), 2018
One-Shot Federated Learning
N. Guha, A. Talwalkar, V. Smith
Machine Learning on Devices Workshop at NeurIPS, 2018
Federated Multi-Task Learning
V. Smith, C. Chiang, M. Sanjabi, A. Talwalkar
Neural Information Processing Systems (NeurIPS), 2017
Distributed Optimization with Arbitrary Local Solvers
C. Ma, J. Konecny, M. Jaggi, V. Smith, M. I. Jordan, P. Richtarik, M. Takac
Optimization Methods and Software, 2017
L1-Regularized Distributed Optimization: A Communication-Efficient Primal-Dual Framework
V. Smith, S. Forte, M. I. Jordan, M. Jaggi
ML Systems Workshop at ICML, 2016
Going In-Depth: Finding Longform on the Web
V. Smith, M. Connor, I. Stanton
Conference on Knowledge Discovery and Data Mining (KDD), 2015
Adding vs. Averaging in Distributed Primal-Dual Optimization
C. Ma*, V. Smith*, M. Jaggi, M. I. Jordan, P. Richtarik, M. Takac
International Conference on Machine Learning (ICML), 2015
Communication-Efficient Distributed Dual Coordinate Ascent
M. Jaggi*, V. Smith*, M. Takac, J. Terhorst, S. Krishnan, T. Hofmann, M. I. Jordan
Neural Information Processing Systems (NeurIPS), 2014
MLI: An API for User-friendly Distribued Machine Learning
E. Sparks, A. Talwalkar, V. Smith, X. Pan, J. Gonzalez, T. Kraska, M. I. Jordan, and M. J. Franklin
IEEE International Conference on Data Mining (ICDM), 2013
A Comparative Study of High Renewables Penetration Electricity Grids
J. Taneja, V. Smith, D. Culler, and C. Rosenberg
IEEE International Conference on Smart Grid Communications (SmartGridComm), 2013
Classification of Sidewalks in Street View Images
V. Smith, J. Malik, and D. Culler
WiP Workshop at International Green Computing Conference (IGCC), 2013
MLbase: A Distributed Machine Learning Wrapper
A. Talwalkar, T. Kraska, R. Griffith, J. Duchi, J. Gonzalez, D. Britz, X. Pan, V. Smith, E. Sparks, A. Wibisono, M. J. Franklin, and M. I. Jordan
Big Learning Workshop at NeurIPS, 2012
Identifying Models of HVAC Systems Using Semiparametric Regression
A. Aswani, N. Master, J. Taneja, V. Smith, A. Krioukov, D. Culler, and C. Tomlin
Proceedings of the American Control Conference (ACC), 2012
Modeling Building Thermal Response to HVAC Zoning
V. Smith, T. Sookoor, and K. Whitehouse
ACM SIGBED Review, 2012