Paul Liang, CMU
Paul Pu Liang
Email: pliang(at)cs.cmu.edu
Office: Gates and Hillman Center 8011
5000 Forbes Avenue, Pittsburgh, PA 15213
Machine Learning Department and Language Technologies Institute, School of Computer Science, Carnegie Mellon University
[CV]
[google scholar]
[linkedin]
[github]
[twitter]
[instagram]
[Publications] [Honors] [Teaching] [Talks] [Activities]
I am a final-year Ph.D. student in the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. I also collaborate with Manuel Blum, Lenore Blum, Jack Hessel, and Yejin Choi at Berkeley and UW/AI2.
I am on the academic job market this year.
[CV] [research statement] [teaching statement] [DEI statement]
My research lies in the foundations of multimodal machine learning with applications in socially intelligent AI, natural language processing, healthcare, and education. As steps towards this goal, I work on:
Foundations of multimodal machine learning: modality heterogeneity, connections, and interactions
[foundations,
interactions,
disagreement,
FactorCL,
MultiViz,
factorized,
translation].
Representation learning and foundation models for multisensory and temporal data
[HighMMT,
MultiBench,
T2FN,
RMFN,
MulT].
Multimodal applications in socially intelligent AI, NLP, healthcare, education, and robotics
[CMU-MOSEI,
Social-IQ,
mental health,
pathology,
education].
Real-world fair, robust, interpretable, and efficient representation learning
[fairness in LMs,
fairness in BERT,
federated learning,
efficiency].
My research is generously supported by a Siebel Scholars Award, Waibel Presidential Fellowship, Facebook PhD Fellowship, and Center for Machine Learning and Health Fellowship, and has been recognized by 3 best paper/honorable mention awards at NeurIPS workshops and ICMI 2017. I love teaching and advising, and was honored to receive the Alan J. Perlis Graduate Student Teaching Award for co-instructing courses (CMU 11-877, CMU 11-866, CMU 11-777), and organizing workshops (ICCV, NAACL, NeurIPS, ACL) and tutorials (ICML, CVPR, NAACL) on multimodal machine learning. Previously, I received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science and Neural Computation from CMU, where I am grateful for the mentorship of Louis-Philippe Morency, Ruslan Salakhutdinov, Tai Sing Lee, Roni Rosenfeld, and Ryan Tibshirani. I have also been fortunate to spend time at DeepMind, Facebook AI Research, Nvidia AI, Google Research, and RIKEN Artificial Intelligence Project.
Research opportunities: I am happy to collaborate and answer questions about my research and CMU academic programs. If you are interested, please send me an email. I especially encourage students from underrepresented groups to reach out.
News
- 2023: Excited to release some recent work formalizing and quantifying multimodal interactions from statistical (NeurIPS 2023) and human (ICMI 2023) perspectives, with applications in visualizing and interpreting multimodal models (ICLR 2023), contrastive learning of unique information (NeurIPS 2023), and guarantees for multimodal semi-supervised learning (arXiv 2023).
- 2023: Co-teaching 11-777 Multimodal Machine Learning, Fall 2023, course content will be updated on the website.
- 2023: Tutorials on multimodal machine learning at ICML 2023, ICMI 2023, CVPR 2022 and NAACL 2022, teaching at CIFAR DLRL summer school and African Masters of Machine Intelligence (day1, day2, day3, day4): check out our survey paper, slides, and videos.
- 2023: Nothing has excited me more than collaborating with and advising great students. I've learned so much from them and I'm excited to watch them embark on their new research agendas - follow their work for more exciting new ideas! Yun Cheng -> PhD at Princeton, Rulin Shao -> PhD at UW, Xiang Fan -> PhD at UW, Jivat Neet -> PhD at Berkeley, Yiwei Lyu -> PhD at Michigan, Yuxin Xiao -> PhD at MIT, Peter Wu -> PhD at Berkeley, Dong Won Lee -> PhD at MIT, Terrance Liu -> PhD at CMU.
- 2023: LP and I are teaching 2 new graduate seminar courses: 11-866 Artificial Social Intelligence and 11-877 Advanced Topics in Multimodal Machine Learning.
- 2022: Check out course content for 11-777 Multimodal Machine Learning, Fall 2022, where LP and I have completely revamped the course content. Also check out fully recorded lecture videos and course content for 11-777 in Fall 2020.
- 2022: Are you working on multimodal tasks and can't decide on a model? Check out HighMMT (TMLR 2022), our attempt at a single multimodal model that can predict sentiment, emotion, humor, disease, robot pose, and more, as well as MultiBench (NeurIPS 2021) and MultiZoo (JMLR 2022), a large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
- 2021: Extremely honored to have received a Facebook PhD Fellowship and a Center for Machine Learning and Health Fellowship to support my research in socially intelligent AI! For students applying for graduate fellowships, I uploaded my statement from the 2020 application.
- 2020: Check out the CMU Machine Learning Blog - new research and educational content every few weeks on ML research going on at CMU!
Education
Selected Publications
(* denotes joint first-authors, see full list of publications here)
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, Ruslan Salakhutdinov
[arXiv] [code]
- Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2023
[arXiv] [code]
- Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Paul Pu Liang*, Zihao Deng*, Martin Ma*, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov
NeurIPS 2023
[arXiv] [code]
- Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
Paul Pu Liang, Yun Cheng, Ruslan Salakhutdinov, Louis-Philippe Morency
ICMI 2023
[arXiv] [code]
- MultiViz: Towards Visualizing and Understanding Multimodal Models
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2023, CHI 2023 Late Breaking Work
[arXiv] [code]
- Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
Tutorials at ICML 2023, ICMI 2023, CVPR 2022, NAACL 2022, Under revision at ACM Computing Surveys 2023
[arXiv] [tutorial website] [tutorial videos]
- High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov
TMLR 2022
[arXiv] [code]
- MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Arav Agarwal, Yun Cheng, Louis-Philippe Morency, Ruslan Salakhutdinov
JMLR Open Source Software 2022
[arXiv] [website] [code]
- Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
Paul Pu Liang
Association for the Scientific Study of Consciousness 2022, Models of Consciousness 2022 (oral)
[arXiv]
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2021
[arXiv] [website] [code]
- Towards Understanding and Mitigating Social Biases in Language Models
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov
ICML 2021
[arXiv] [code]
- Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Paul Pu Liang*, Terrance Liu*, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2021 (oral)
[arXiv]
- Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
Paul Pu Liang*, Peter Wu*, Liu Ziyin, Louis-Philippe Morency, Ruslan Salakhutdinov
ACM Multimedia 2021 (oral)
[arXiv] [code]
- Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2020
[arXiv] [code]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
[arXiv] [code]
- Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Paul Pu Liang*, Zhun Liu*, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2019
[arXiv] [poster]
- Learning Factorized Multimodal Representations
Yao-Hung Hubert Tsai*, Paul Pu Liang*, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2019
[arXiv] [code] [poster]
- Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Hai Pham*, Paul Pu Liang*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
AAAI 2019
[arXiv] [code] [slides] [poster]
- Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (first runner-up award)
[paper] [slides] [poster]
- Multimodal Language Analysis with Recurrent Multistage Fusion
Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
EMNLP 2018 (oral)
[arXiv] [slides] [poster]
- Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
ICMI 2017 (oral, honorable mention award)
[arXiv] [code] [slides]
Honors
Teaching
- Co-Lecturer: 11-777 Multimodal Machine Learning, Fall 2023, CMU with Louis-Philippe Morency
- Instructor: Multimodal Artificial Intelligence (day1, day2, day3, day4), African Masters Of Machine Intelligence, Summer 2023
- Co-Instructor: Tutorials on Multimodal ML at ICML 2023, ICMI 2023, CVPR 2022 and NAACL 2022 with Louis-Philippe Morency
- Co-Instructor: 11-866 Artificial Social Intelligence, Spring 2023, CMU with Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2023, CMU with Louis-Philippe Morency
- Co-Lecturer: 11-777 Multimodal Machine Learning, Fall 2022, CMU with Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2022, CMU with Louis-Philippe Morency and Amir Zadeh
- Guest Lecturer: 10-707 Deep Learning, 05-618 Human AI Interaction, 17-728 Machine Learning and Sensing, 2021-2023, CMU.
Lectures on multimodal machine learning [slides] [video]
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2020, CMU. Instructor: Louis-Philippe Morency
4 lectures on multimodal tasks [slides] [video], deep generative models [slides] [video], reinforcement learning [slides] [video], and multimodal RL [slides] [video].
Public videos on YouTube have amassed more than 10000 views.
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2019, CMU. Instructor: Louis-Philippe Morency
2 lectures on reinforcement learning [slides] and multimodal RL [slides]
- TA: 10-708 Probabilistic Graphical Models, Spring 2019, CMU. Instructor: Eric Xing
- TA: 10-715 Advanced Introduction to Machine Learning, Fall 2018, CMU. Instructor: Maria-Florina Balcan
- TA: 10-601 Introduction to Machine Learning, Fall 2016, CMU. Instructor: Roni Rosenfeld
- TA: 15-213/18-213/15-513 Introduction to Computer Systems, Summer 2016, CMU. Instructor: Brian Railing
Student Advising
Some amazing students I've had the pleasure of advising:
- Rohan Pandey, now at Reworkd AI (YC S23) (Best Senior Thesis Award)
- Samuel Yu (CRA Finalist)
- Yun Cheng, now PhD student at Princeton
- Rulin Shao, now PhD student at University of Washington
- Xiang Fan, now PhD student at University of Washington (CRA Honorable Mention)
- Jivat Neet, then research fellow at Microsoft Research, now PhD student at UC Berkeley
- Yiwei Lyu, now PhD student at University of Michigan (CRA Honorable Mention)
- Yuxin Xiao, now PhD student at MIT
- Peter Wu, now PhD student at UC Berkeley
- Dong Won Lee, now PhD student at MIT
- Xiangru Tang, now PhD student at Yale
- Terrance Liu, now PhD student at CMU
- Seong Hyeon Park, now PhD student at KAIST
- Chengfeng Mao, now PhD student at MIT
- Ziyin Liu, now PhD student at University of Tokyo
- Irene Li, now at SoundHound (CRA Honorable Mention)
Academic Talks
- Foundations of Multimodal Machine Learning: Principles, Challenges, and Open Questions
ICMI Tutorial, October 2023
African Masters of Machine Intelligence, July 2023
CIFAR DLRL Summer School, July 2023
ICML Tutorial, July 2023
ICLR Workshop on Multimodal Representation Learning, April 2023
IBM Zurich, March 2023
Harvard Medical School, Oct 2022
Heidelberg Laureate Forum, Sept 2022
UC Berkeley Speech Group, Sept 2022
Stanford University MedAI Group, Sept 2022
National University of Singapore, Aug 2022
Amazon AI, Aug 2022
NAACL Tutorial, July 2022
CVPR Tutorial, June 2022
Allen Institute of AI & University of Washington, June 2022
Shandong University, June 2022
Carnegie Mellon University, May 2022
[slides1] [slides2]
- Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
Peking University, March 2023
Models of Consciousness Conference, Sept 2022
International Joint Conference on Theoretical Computer Science, Aug 2022
- Introduction to Multimodal Machine Learning
Guest lectures at CMU 10-707 Deep Learning, Peking University, University of Florida, 2023
Guest lectures at CMU 10-707 Deep Learning, 05-618 Human AI Interaction, 17-728 Machine Learning and Sensing, 2022
CMU 11-777 Multimodal Machine Learning 8 lectures, Fall 2022
CMU 11-777 Multimodal Machine Learning 4 lectures, Fall 2020
CMU 11-777 Multimodal Machine Learning 2 lectures, Fall 2019
[slides]
- Towards Real-World Social AI
Facebook Fellowship Summit, Sept 2021
DeepMind Multimodal Team, Sept 2021
IJCAI Workshop on Multimodal Analytics, Aug 2021
Big Data and AI Conference, July 2021
Agency for Science, Technology and Research Singapore, June 2021
Adobe Research, Jan 2021
Carnegie Mellon University, Oct 2020
[slides]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Agency for Science, Technology and Research Singapore, June 2021
NeurIPS 2019 Workshop on Federated Learning, Dec 2019
[slides]
- Computational Modeling of Human Multimodal Language
Google Research, July 2019
RIKEN Artificial Intelligence Project Tokyo Machine Learning Seminar, Jan 2019
RIKEN Artificial Intelligence Project Kyoto Machine Learning Seminar, Dec 2018
ACL 2018, July 2018
CMU Machine Learning Department Data Analysis Project Presentation, Apr 2018
[slides]
Professional Activities
- Co-organizer: Tutorial on Multimodal Machine Learning at ICML 2023, ICMI 2023, CVPR 2022, NAACL 2022
- Co-organizer: Workshop on Machine Learning for Cognitive and Mental Health at AAAI 2024
- Co-organizer: Artificial Social Intelligence Workshop and Challenge at ICCV 2023
- Co-organizer: Workshop on Foundation Models in Vision and Language at ICDM 2022
- Co-organizer: Workshop on Multimodal Artificial Intelligence at NAACL 2022, NAACL 2021
- Co-organizer: Workshop on Tensor Networks in Machine Learning at NeurIPS 2020
- Co-organizer: Grand Challenge and Workshop on Human Multimodal Language at ACL 2020, ACL 2018
- Workflow Chair: ICML 2019
- Session Chair: AAAI 2019
- Conference Program Committee: NeurIPS, ICML, ICLR, ACL, EMNLP, NAACL, EACL, COLING, IJCNLP, AACL, ACL Rolling Review, CVPR, ICCV, ECCV, WACV, ACCV, AAAI, IJCAI, AISTATS, UAI, CHI, ICMI, FG, ACML, ML4H, CHIL, ACM Multimedia, ACM Multimedia Asia, Interspeech
- Workshop Program Committee: NeurIPS workshop on Meta-Learning, NeurIPS workshop on Machine Learning for Health, ICLR workshop on Embodied Multimodal Learning, ICLR workshop on Never-ending RL, ICLR workshop on Enormous Language Models, ACL workshop on Multimodal Language, EMNLP workshop on NLP Open Source Software, EMNLP workshop on NLP Beyond Text, NAACL workshop on Trustworthy NLP, NAACL workshop on Multimodal AI, IJCAI workshop on Federated Learning, WWW workshop on NLP Beyond Text, ICRA workshop on Social Intelligence in Humans and Robots, NeurIPS Workshop on Human Evaluation of Generative Models, NeurIPS Workshop on Self-Supervised Learning, EMNLP Workshop on Generation, Evaluation & Metrics
- Journal Reviewer: IEEE Transactions on Affective Computing, IEEE Transactions on Audio, Speech and Language Processing, IEEE Transactions on Multimedia, IEEE Transactions on Cybernetics, IEEE Computational Intelligence Magazine, IEEE Signal Processing Letters, Elsevier Information Fusion, Elsevier Computer Speech and Language, Machine Learning, Transactions on Machine Learning Research, Neural Networks, Journal of Artificial Intelligence Research, Journal of Machine Learning Research, Medical Image Analysis
- CMU Machine Learning Blog Editorial Board: 2019, 2020, 2021, 2022 (chief editor)
- CMU AI Undergraduate Research Mentor: 2018, 2019, 2020, 2021
- CMU Graduate Applicant Support Program Organizer: 2020
- CMU Machine Learning Department PhD Admissions Committee: 2018, 2019, 2020, 2021, 2022
- CMU Machine Learning Department Masters Admissions Committee: 2017, 2018
- CMU Singapore Students Association Co-President: 2015
I have an Erdős number of 3 (Paul Erdős → Giuseppe Melfi → Erik Cambria → Paul Pu Liang).
This page has been accessed at least
times since Feb 8, 2018.