Paul Liang, CMU
Paul Pu Liang
Email: pliang(at)cs.cmu.edu
Office: Gates and Hillman Center 8011
5000 Forbes Avenue, Pittsburgh, PA 15213
Machine Learning Department and Language Technologies Institute, School of Computer Science, Carnegie Mellon University
[CV]
@pliang279
@pliang279
@lpwinniethepu
I am a fifth-year Ph.D. student in the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. I also collaborate closely with Manuel Blum, Lenore Blum, and Daniel Rubin at Berkeley and Stanford. My research lies in the foundations of multimodal machine learning with applications in socially intelligent AI, understanding human and machine intelligence, natural language processing, healthcare, and education. As steps towards this goal, I work on:
Foundations of multimodal machine learning: representation, translation, fusion, and alignment of heterogeneous data
[foundations,
interactions,
MultiViz,
HighMMT,
Brainish,
MultiBench,
factorized,
translation,
alignment].
Social intelligence: AI that can perceive human behaviors and engage in multimodal interactions in embodied environments
[CMU-MOSEI,
Social-IQ,
sentiment,
emotions].
Human-centered AI applications in language, vision, speech, robotics, healthcare, and education
[mobile health,
education].
Real-world representation learning: learning fair, robust, interpretable, efficient, and generalizable representations
[fairness in language models,
fairness in sentence representations,
federated learning,
robustness,
efficiency].
My research is generously supported by a Waibel Presidential Fellowship, Facebook PhD Fellowship, and Center for Machine Learning and Health Fellowship, and has been recognized by 3 best-paper awards at NeurIPS workshops and ICMI 2017. I love teaching and advising, and was honored to receive the Alan J. Perlis Graduate Student Teaching Award for co-instructing courses (CMU 11-877, CMU 11-866, CMU 11-777), and organizing workshops (ICCV, NAACL, NeurIPS, ACL) and tutorials (ICML, CVPR, NAACL) on multimodal machine learning. Previously, I received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science and Neural Computation from CMU, where I am grateful for the mentorship of Louis-Philippe Morency, Ruslan Salakhutdinov, Tai Sing Lee, Roni Rosenfeld, and Ryan Tibshirani. I have also been fortunate to spend time at DeepMind, Facebook AI Research, Nvidia AI, Google Research, and RIKEN Artificial Intelligence Project.
Research opportunities: I am happy to collaborate and answer questions about my research and CMU academic programs. If you are interested, please send me an email. I especially encourage students from underrepresented groups to reach out.
[News] [Education] [Publications] [Honors] [Teaching] [Talks] [Activities]
News
- 01/2023: LP and I are teaching 2 new graduate seminar courses: 11-866 Artificial Social Intelligence and 11-877 Advanced Topics in Multimodal Machine Learning.
- 09/2022: Survey paper covering all the core definitions, technical challenges, and open questions in multimodal ML.
- 08/2022: Check out course content for 11-777 Multimodal Machine Learning, Fall 2022, where LP and I have completely revamped the course content, with new lectures on multimodal reasoning, generation, transfer, and quantification.
- 05/2022: I gave several talks recently on my work in multimodal representation learning, social intelligence, and healthcare, slides can be found here: [multimodal ML] [brainish] [short tutorial]
- 05/2022: We're organizing tutorials on multimodal machine learning at CVPR 2022 and NAACL 2022: check out our survey paper, slides, and video.
- 04/2022: Are you working on multimodal tasks and can't decide on a model? Check out HighMMT - our attempt at a single multimodal model with shared parameters for sentiment, emotion, humor, disease, robot pose prediction & more!
- 04/2022: Nothing has excited me more than collaborating with and advising great students. I've learned so much from them and I'm hugely excited to watch them embark on their new research agendas as incoming PhD students - follow their work for more exciting new ideas! Yiwei Lyu -> PhD at University of Michigan, Yuxin Xiao -> PhD at MIT, Peter Wu -> PhD at UC Berkeley, Dong Won Lee -> PhD at MIT, Terrance Liu -> PhD at CMU.
- 01/2022: We're organizing a new course 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2022 @ CMU. It will primarily be reading and discussion-based. We plan to post discussion probes, relevant papers, and summarized discussion highlights every week on the website.
- 06/2021: Excited to release MultiBench, a large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas, at NeurIPS 2021.
- 04/2021: Extremely honored to have received a Facebook PhD Fellowship and a Center for Machine Learning and Health Fellowship to support my research in socially intelligent AI! For students applying for graduate fellowships, I uploaded my statement from the 2020 application.
- 12/2020: Fully recorded lecture videos and course content for 11-777 Multimodal Machine Learning, Fall 2020.
- 10/2020: Check out the CMU Machine Learning Blog - new research and educational content every few weeks on ML research going on at CMU!
- 12/2019: We are organizing the Second Grand Challenge and Workshop on Human Multimodal Language at ACL 2020! Consider submitting your work!
- 06/2019: I am compiling a reading list for multimodal ML containing papers, software, workshops, tutorials, and courses! It spans various modalities (language, vision, speech, video, touch) and applications (QA, dialog, RL, reasoning, grounding, navigation, affective computing, healthcare, robotics). Links to code and data are included. I'll be updating regularly, if there's anything I missed, please let me know.
- 01/2019: I will be a TA for 10-708 Probabilistic Graphical Models in Spring 2019, with new content on deep generative models, RL, and probabilistic programming!
- 01/2019: Excited to be a workflow chair for ICML 2019!
- 05/2018: I completed my Master's Data Analysis Project research and received the 1st runner-up award.
- 01/2018: We are organizing the First Grand Challenge and Workshop on Human Multimodal Language at ACL 2018.
- 11/2017: Our paper on gated multimodal fusion won the honorable mention award at ICMI 2017!
Education
- Carnegie Mellon University, Pittsburgh, PA, USA. 2018 - Present
Ph.D. in Machine Learning
Advisors: Louis-Philippe Morency and Ruslan Salakhutdinov
- Carnegie Mellon University, Pittsburgh, PA, USA. 2017 - 2018
M.S. in Machine Learning
Advisors: Louis-Philippe Morency and Ruslan Salakhutdinov
Thesis: Computational Modeling of Human Multimodal Language
- Carnegie Mellon University, Pittsburgh, PA, USA. 2014 - 2017
B.S. with University Honors in Computer Science and Neural Computation
Selected Publications
(* denotes joint first-authors, see full list here)
- Quantifying & Modeling Feature Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
[arXiv] [code]
- MultiViz: Towards Visualizing and Understanding Multimodal Models
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2023, CHI 2023 Late Breaking Work
[arXiv] [code]
- Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
Tutorials at ICML 2023, CVPR 2022, NAACL 2022, Under revision at ACM Computing Surveys 2023
[arXiv] [tutorial website] [tutorial videos]
- High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov
TMLR 2022
[arXiv] [code]
- Fundamentals of Multimodal Representation Learning: Towards Generalization and Quantification
Paul Pu Liang
PhD Thesis Proposal 2022. Committee: Louis-Philippe Morency, Ruslan Salakhutdinov, Manuel Blum, Lenore Blum, Trevor Darrell
[document]
- Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
Paul Pu Liang
Association for the Scientific Study of Consciousness 2022, Models of Consciousness 2022 (oral)
[arXiv]
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2021, JMLR Open Source Software 2022
[arXiv] [website] [code]
- Towards Understanding and Mitigating Social Biases in Language Models
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov
ICML 2021
[arXiv] [code]
- Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Paul Pu Liang*, Terrance Liu*, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2021 (oral)
[arXiv]
- Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
Paul Pu Liang*, Peter Wu*, Liu Ziyin, Louis-Philippe Morency, Ruslan Salakhutdinov
ACM Multimedia 2021 (oral)
[arXiv] [code]
- Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2020
[arXiv] [code]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
[arXiv] [code]
- Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Paul Pu Liang*, Zhun Liu*, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2019
[arXiv] [poster]
- Learning Factorized Multimodal Representations
Yao-Hung Hubert Tsai*, Paul Pu Liang*, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2019
[arXiv] [code] [poster]
- Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Hai Pham*, Paul Pu Liang*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
AAAI 2019, NeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech and Language (oral)
[arXiv] [code] [slides] [poster]
- Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (first runner-up award)
[paper] [slides] [poster]
- Multimodal Language Analysis with Recurrent Multistage Fusion
Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
EMNLP 2018 (oral), NeurIPS 2018 Workshop on Modeling and Decision-making in the Spatiotemporal Domain (oral)
[arXiv] [slides] [poster]
- Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
ICMI 2017 (oral, honorable mention award)
[arXiv] [code] [slides]
Honors
Teaching
- Co-Instructor: 11-866 Artificial Social Intelligence, Spring 2023, CMU. With Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2023, CMU. With Louis-Philippe Morency
- Co-Lecturer: 11-777 Multimodal Machine Learning, Fall 2022, CMU. With Louis-Philippe Morency
- Co-Instructor: 11-877 Advanced Topics in Multimodal Machine Learning, Spring 2022, CMU. With Louis-Philippe Morency and Amir Zadeh
- Guest Lecturer: 10-707 Deep Learning, 05-618 Human AI Interaction, 17-728 Machine Learning and Sensing, 2021-2023, CMU.
Lectures on multimodal machine learning [slides] [video]
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2020, CMU. Instructor: Louis-Philippe Morency
4 lectures on multimodal tasks [slides] [video], deep generative models [slides] [video], reinforcement learning [slides] [video], and multimodal RL [slides] [video].
Public videos on YouTube have amassed more than 10000 views.
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2019, CMU. Instructor: Louis-Philippe Morency
2 lectures on reinforcement learning [slides] and multimodal RL [slides]
- TA: 10-708 Probabilistic Graphical Models, Spring 2019, CMU. Instructor: Eric Xing
- TA: 10-715 Advanced Introduction to Machine Learning, Fall 2018, CMU. Instructor: Maria-Florina Balcan
- TA: 10-601 Introduction to Machine Learning, Fall 2016, CMU. Instructor: Roni Rosenfeld
- TA: 15-213/18-213/15-513 Introduction to Computer Systems, Summer 2016, CMU. Instructor: Brian Railing
Student Advising
Some amazing students I've had the pleasure of advising:
- Rohan Pandey, now at Microsoft (Best Senior Thesis Award)
- Samuel Yu (CRA Finalist)
- Yun Cheng, now PhD student at Princeton
- Rulin Shao, now PhD student at University of Washington
- Xiang Fan, now PhD student at University of Washington (CRA Honorable Mention)
- Jivat Neet, then research fellow at Microsoft Research, now PhD student at UC Berkeley
- Yiwei Lyu, now PhD student at University of Michigan (CRA Honorable Mention)
- Yuxin Xiao, now PhD student at MIT
- Peter Wu, now PhD student at UC Berkeley
- Dong Won Lee, now PhD student at MIT
- Xiangru Tang, now PhD student at Yale
- Terrance Liu, now PhD student at CMU
- Seong Hyeon Park, now PhD student at KAIST
- Chengfeng Mao, now PhD student at MIT
- Ziyin Liu, now PhD student at University of Tokyo
- Irene Li, now at SoundHound (CRA Honorable Mention)
Academic Talks
- Foundations of Multimodal Machine Learning: Principles, Challenges, and Open Questions
ICML Tutorial, July 2023
ICLR Workshop on Multimodal Representation Learning, April 2023
IBM Zurich, March 2023
Harvard Medical School, Oct 2022
Heidelberg Laureate Forum, Sept 2022
UC Berkeley Speech Group, Sept 2022
Stanford University MedAI Group, Sept 2022
Models of Consciousness Conference, Sept 2022
National University of Singapore, Aug 2022
Amazon AI, Aug 2022
International Joint Conference on Theoretical Computer Science, Aug 2022
NAACL Tutorial, July 2022
CVPR Tutorial, June 2022
Allen Institute of AI & University of Washington, June 2022
Shandong University, June 2022
Carnegie Mellon University, May 2022
[slides1] [slides2]
- Introduction to Multimodal Machine Learning
Guest lectures at CMU 10-707 Deep Learning, Peking University, University of Florida, 2023
Guest lectures at CMU 10-707 Deep Learning, 05-618 Human AI Interaction, 17-728 Machine Learning and Sensing, 2022
CMU 11-777 Multimodal Machine Learning 8 lectures, Fall 2022
CMU 11-777 Multimodal Machine Learning 4 lectures, Fall 2020
CMU 11-777 Multimodal Machine Learning 2 lectures, Fall 2019
[slides]
- Towards Real-World Social AI
Facebook Fellowship Summit, Sept 2021
DeepMind Multimodal Team, Sept 2021
IJCAI Workshop on Multimodal Analytics, Aug 2021
Big Data and AI Conference, July 2021
Agency for Science, Technology and Research Singapore, June 2021
Adobe Research, Jan 2021
Carnegie Mellon University, Oct 2020
[slides]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Agency for Science, Technology and Research Singapore, June 2021
NeurIPS 2019 Workshop on Federated Learning, Dec 2019
[slides]
- Computational Modeling of Human Multimodal Language
Google Research, July 2019
RIKEN Artificial Intelligence Project Tokyo Machine Learning Seminar, Jan 2019
RIKEN Artificial Intelligence Project Kyoto Machine Learning Seminar, Dec 2018
ACL 2018, July 2018
CMU Machine Learning Department Data Analysis Project Presentation, Apr 2018
[slides]
Professional Activities
- Co-organizer: ICML 2023 Tutorial on Multimodal Machine Learning
- Co-organizer: ICCV 2023 Artificial Social Intelligence Workshop and Challenge
- Co-organizer: CVPR 2022 Tutorial on Multimodal Machine Learning
- Co-organizer: NAACL 2022 Tutorial on Multimodal Machine Learning
- Co-organizer: ICDM 2022 Workshop on Foundation Models in Vision and Language
- Co-organizer: NAACL 2022 Workshop on Multimodal Artificial Intelligence
- Co-organizer: NAACL 2021 Workshop on Multimodal Artificial Intelligence
- Co-organizer: NeurIPS 2020 Workshop on Tensor Networks in Machine Learning
- Co-organizer: ACL 2020 Grand Challenge and Workshop on Human Multimodal Language
- Co-organizer: ACL 2018 Grand Challenge and Workshop on Human Multimodal Language
- Senior PC Member: IJCAI
- Workflow Chair: ICML 2019
- Session Chair: AAAI 2019
- Conference Program Committee: NeurIPS, ICML, ICLR, ACL, EMNLP, NAACL, EACL, COLING, IJCNLP, AACL, ACL Rolling Review, CVPR, ICCV, ECCV, WACV, ACCV, AAAI, IJCAI, AISTATS, UAI, CHI, ICMI, FG, ACML, ML4H, CHIL, ACM Multimedia, ACM Multimedia Asia, Interspeech
- Workshop Program Committee: NeurIPS workshop on Meta-Learning, NeurIPS workshop on Machine Learning for Health, ICLR workshop on Embodied Multimodal Learning, ICLR workshop on Never-ending RL, ICLR workshop on Enormous Language Models, ACL workshop on Multimodal Language, EMNLP workshop on NLP Open Source Software, EMNLP workshop on NLP Beyond Text, NAACL workshop on Trustworthy NLP, NAACL workshop on Multimodal AI, IJCAI workshop on Federated Learning, WWW workshop on NLP Beyond Text, ICRA workshop on Social Intelligence in Humans and Robots, NeurIPS Workshop on Human Evaluation of Generative Models, NeurIPS Workshop on Self-Supervised Learning, EMNLP Workshop on Generation, Evaluation & Metrics
- Journal Reviewer: IEEE Transactions on Affective Computing, IEEE Transactions on Audio, Speech and Language Processing, IEEE Transactions on Multimedia, IEEE Transactions on Cybernetics, IEEE Computational Intelligence Magazine, IEEE Signal Processing Letters, Elsevier Information Fusion, Elsevier Computer Speech and Language, Machine Learning, Transactions on Machine Learning Research, Neural Networks, Journal of Artificial Intelligence Research, Journal of Machine Learning Research, Medical Image Analysis
- CMU Machine Learning Blog Editorial Board: 2019, 2020, 2021, 2022 (chief editor)
- CMU AI Undergraduate Research Mentor: 2018, 2019, 2020, 2021
- CMU Graduate Applicant Support Program Organizer: 2020
- CMU Machine Learning Department PhD Admissions Committee: 2018, 2019, 2020, 2021, 2022
- CMU Machine Learning Department Masters Admissions Committee: 2017, 2018
- CMU Singapore Students Association Co-President: 2015
I have an Erdős number of 3 (Paul Erdős → Giuseppe Melfi → Erik Cambria → Paul Pu Liang).
This page has been accessed at least 
times since Feb 8, 2018.