Paul Liang, CMU
Paul Pu Liang
Email: pliang(at)cs.cmu.edu
Office: Gates and Hillman Center 8011
5000 Forbes Avenue, Pittsburgh, PA 15213
Multicomp Lab, Language Technologies Institute, School of Computer Science, Carnegie Mellon University
[CV]
@pliang279
@pliang279
@lpwinniethepu
I am a third-year Ph.D. student in the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. My long-term research goal is to build socially intelligent embodied agents with the ability to perceive and engage in multimodal human communication. As steps towards this goal, my research focuses on:
Social intelligence: agents that can perceive human behaviors and engage in multimodal interactions in embodied environments
[1,
2,
3,
4].
Fundamentals of multimodal machine learning: representation, translation, fusion, and alignment of heterogeneous data
[1,
2].
Human-centered AI applications in language, vision, speech, robotics, healthcare, and education
[1,
2,
3,
4].
Real-world representation learning: learning fair, robust, interpretable, efficient, and generalizable representations
[1,
2,
3,
4].
My research has been recognized by paper awards at the NeurIPS 2019 workshop on federated learning and ICMI 2017. I regularly organize the workshop on multimodal learning (NAACL 2021, ACL 2020, and ACL 2018) and have also participated as an organizer for the workshop on tensor networks at NeurIPS 2020 and a workflow chair for ICML 2019. Previously, I received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science from CMU, where I am grateful for the mentorship of Louis-Philippe Morency, Ruslan Salakhutdinov, Tai Sing Lee, Ryan Tibshirani, and Roni Rosenfeld. I have also been fortunate to spend time at Facebook AI Research, Nvidia AI, Google Research, and RIKEN Artificial Intelligence Project.
Research opportunities: I am happy to collaborate and/or answer questions about my research and CMU academic programs. If you are interested, please send me an email. I especially encourage students from underrepresented groups to reach out.
[News] [Education] [Experience] [Publications] [Honors] [Teaching] [Students] [Talks] [Activities]
News
- 02/2021: Check out our new ICLR paper on learning efficient sparse embeddings for text, users, and recommendation sytems!
- 12/2020: Fully recorded lecture videos and course content for 11-777 Advanced Multimodal Machine Learning, Fall 2020.
- 10/2020: Check out the CMU Machine Learning Blog - new research and educational content every few weeks on ML research going on at CMU!
- 08/2020: I will be the head TA for 11-777 Advanced Multimodal Machine Learning in Fall 2020. Follow the class here!
- 06/2020: Check out our ACL paper on mitigating bias in sentence representations and the proceedings from our ACL workshop on multimodal language.
- 02/2020: New preprints on sparse representation learning, robust learning from noisy labels, and diverse trajectory forecasting.
- 12/2019: We are organizing the Second Grand Challenge and Workshop on Human Multimodal Language at ACL 2020! Consider submitting your work!
- 09/2019: Learning to abstain under uncertainty with portfolio theory accepted to NeurIPS 2019.
- 08/2019: I will be the head TA for 11-777 Advanced Multimodal Machine Learning in Fall 2019, with new content on multimodal RL, alignment, language grounding, and interpretable learning!
- 06/2019: I am compiling a reading list for multimodal ML containing papers, software, workshops, tutorials, and courses! It spans various modalities (language, vision, speech, video, touch) and applications (QA, dialog, RL, reasoning, grounding, navigation, affective computing, healthcare, robotics). Links to code and data are included. I'll be updating regularly, if there's anything I missed, please let me know.
- 06/2019: New preprints on fair federated learning via local models, learning to abstain under uncertainty, and generative models for incomplete data.
- 05/2019: 2 papers accepted at ACL 2019 on learning from imperfect and unaligned multimodal time series data.
- 02/2019: We will present 2 papers at NAACL and CVPR 2019: strong baselines for multimodal learning and a new QA dataset on comprehending social interactions.
- 01/2019: I will be a TA for 10-708 Probabilistic Graphical Models in Spring 2019, with new content on deep generative models, RL, and probabilistic programming!
- 01/2019: Excited to be a workflow chair for ICML 2019!
- 12/2018: "Learning Factorized Multimodal Representations" to appear at ICLR 2019.
- 08/2018: I will be a TA for 10-715 Advanced Introduction to Machine Learning in Fall 2018.
- 05/2018: I completed my Master's Data Analysis Project research and received the 1st runner-up award.
- 04/2018: 3 papers presented at ACL 2018 main conference and workshops: CMU-MOSEI dataset, low-rank fusion, and seq2seq2sentiment.
- 01/2018: We are organizing the First Grand Challenge and Workshop on Human Multimodal Language at ACL 2018.
- 11/2017: Our paper on gated multimodal fusion won the honorable mention award at ICMI 2017!
Education
- Carnegie Mellon University, Pittsburgh, PA, USA. Aug 2018 - Present
Ph.D. in Machine Learning (GPA: 4.23/4.00)
Advisors: Louis-Philippe Morency and Ruslan Salakhutdinov
- Carnegie Mellon University, Pittsburgh, PA, USA. Aug 2017 - May 2018
M.S. in Machine Learning (GPA: 4.24/4.00)
Advisors: Louis-Philippe Morency and Ruslan Salakhutdinov
Thesis: Computational Modeling of Human Multimodal Language
- Carnegie Mellon University, Pittsburgh, PA, USA. Aug 2014 - May 2017
B.S. with University Honors in Computer Science (GPA: 3.87/4.00)
Minor in Neural Computation
Experience
- Google DeepMind, London, UK. June 2021 - Oct 2021
Research Intern
Advisors: Dani Yogatama, Aida Nematzadeh, and Phil Blunsom
- Facebook AI Research, New York, NY, USA. May 2020 - Aug 2020
Research Intern
Advisors: Brandon Amos, Tim Rocktäschel, and Ed Grefenstette
- Nvidia AI, Santa Clara, CA, USA and Toronto, Canada. Feb 2020 - May 2020
Research Intern
Advisors: Yuke Zhu, Anima Anandkumar, and Sanja Fidler
- Google Research, Mountain View, CA, USA and Pittsburgh, PA, USA. May 2019 - Nov 2019
Research Intern
Advisors: Manzil Zaheer, Yuan Wang, and Amr Ahmed
- RIKEN Artificial Intelligence Project, Tokyo, Japan and Kyoto, Japan. Dec 2018 - Jan 2019
Visiting Researcher
Advisors: Makoto Yamada, Qibin Zhao, and Masashi Sugiyama
Publications
(* denotes joint first-authors)
2021
- StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Yiwei Lyu*, Paul Pu Liang*, Hai Pham*, Eduard Hovy, Barnabás Póczos, Ruslan Salakhutdinov, Louis-Philippe Morency
NAACL 2021
[arXiv] [code]
- Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration
Jivat Neet, Yiding Jiang, Paul Pu Liang
ICLR 2021 Workshop on Embodied Multimodal Learning
[arXiv] [code]
- Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
Paul Pu Liang, Manzil Zaheer, Yuan Wang, Amr Ahmed
ICLR 2021
[arXiv] [code]
- Understanding the Tradeoffs in Client-Side Privacy for Speech Recognition
Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
preprint
[arXiv]
- An Investigation of how Label Smoothing Affects Generalization
Blair Chen, Liu Ziyin, Zihao Wang, Paul Pu Liang
preprint
[arXiv]
2020
- Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
Paul Pu Liang, Peter Wu, Liu Ziyin, Louis-Philippe Morency, Ruslan Salakhutdinov
NeurIPS 2020 Workshop on Meta Learning
[arXiv] [code]
- Multimodal Privacy-preserving Mood Prediction from Mobile Data: A Preliminary Study
Terrance Liu*, Paul Pu Liang*, Michal Muszynski, Ryo Ishii, David Brent, Randy Auerbach, Nicholas Allen, Louis-Philippe Morency
NeurIPS 2020 Workshop on Machine Learning for Mobile Health
[arXiv]
- MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Amir Zadeh, Yansheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, Louis-Philippe Morency
EMNLP 2020
[paper]
- Diverse and Admissible Trajectory Prediction through Multimodal Context Understanding
Seong Hyeon Park, Gyubok Lee, Manoj Bhat, Jimin Seo, Minseok Kang, Jonathan Francis, Ashwin R. Jadhav, Paul Pu Liang, Louis-Philippe Morency
ECCV 2020
CVPR 2020 Argoverse competition (honorable mention award)
[arXiv] [code]
- Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2020
[arXiv] [code]
- On Emergent Communication in Competitive Multi-Agent Teams
Paul Pu Liang, Jeffrey Chen, Ruslan Salakhutdinov, Louis-Philippe Morency, Satwik Kottur
AAMAS 2020 (oral)
NeurIPS 2019 Workshop on Emergent Communication
[arXiv] [code] [slides]
- Empirical and Theoretical Studies of Multimodal Co-learning
Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency
Elsevier Information Fusion 2020
[arXiv]
2019
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
[arXiv] [code]
- Deep Gamblers: Learning to Abstain with Portfolio Theory
Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, Masahito Ueda
NeurIPS 2019
[arXiv] [code] [poster]
- Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Paul Pu Liang*, Zhun Liu*, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2019
[arXiv] [poster]
- Multimodal Transformer for Unaligned Multimodal Language Sequences
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
ACL 2019
[arXiv] [code]
- Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, Louis-Philippe Morency
CVPR 2019 (oral)
[paper] [code] [poster]
- Strong and Simple Baselines for Multimodal Utterance Embeddings
Paul Pu Liang*, Yao Chong Lim*, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency
NAACL 2019 (oral)
[arXiv] [code] [slides]
- Learning Factorized Multimodal Representations
Yao-Hung Hubert Tsai*, Paul Pu Liang*, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2019
NeurIPS 2018 Workshop on Bayesian Deep Learning
[arXiv] [code] [poster]
- Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Hai Pham*, Paul Pu Liang*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
AAAI 2019
NeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech and Language (oral)
[arXiv] [code] [slides] [poster]
- Words can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
AAAI 2019
[arXiv] [code] [slides] [poster]
2018
- Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (first runner-up award)
[paper] [slides] [poster]
- Multimodal Language Analysis with Recurrent Multistage Fusion
Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
EMNLP 2018 (oral)
NeurIPS 2018 Workshop on Modeling and Decision-making in the Spatiotemporal Domain (oral)
[arXiv] [slides] [poster]
- Multimodal Local-Global Ranking Fusion for Emotion Recognition
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ICMI 2018
[arXiv] [poster]
- An Empirical Evaluation of Sketched SVD and its Application to Leverage Score Ordering
Hui Han Chin, Paul Pu Liang
ACML 2018
[arXiv] [slides] [poster]
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Amir Zadeh, Paul Pu Liang, Jonathan Vanbriesen, Soujanya Poria, Edmund Tong, Erik Cambria, Minghai Chen, Louis-Philippe Morency
ACL 2018 (oral)
[arXiv] [code] [slides]
- Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Zhun Liu, Ying Shen, Varun Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ACL 2018 (oral)
[arXiv] [code] [slides]
- Multi-attention Recurrent Network for Human Communication Comprehension
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, Louis-Philippe Morency
AAAI 2018 (oral)
[arXiv] [code] [slides]
- Memory Fusion Network for Multi-view Sequential Learning
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
AAAI 2018 (oral)
[arXiv] [code] [slides]
2017
- Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
ICMI 2017 (oral, honorable mention award)
[arXiv] [code] [slides]
Organized Workshop Proceedings
- Proceedings of the Second Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
ACL 2020 Workshop Proceedings
[proceedings] [website]
- Proceedings of the First Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
ACL 2018 Workshop Proceedings
[proceedings] [website] [introduction] [datasets] [results]
Honors
- Facebook PhD Fellowship: 2021-2023
- Center for Machine Learning and Health Fellowship (declined): 2021
- CMU Graduate Research Fellowship: 2018-2023
- MIT Sandbox Innovation Program Funding Recipient: 2021
- CVPR 2020 Argoverse Competition Honorable Mention: 2020
- NeurIPS 2019 Workshop on Federated Learning Distinguished Student Paper: 2019
- NeurIPS 2019 Travel Award and Reviewer Award: 2019
- CMU Teaching Assistant Award: 2019
- Princeton University Gordon Wu Fellowship (declined): 2018
- NSF Graduate Research Fellowship Honorable Mention: 2018
- CMU Machine Learning Department Data Analysis Project First Runner-up: 2018
- ICMI 2017 Best Paper Honorable Mention (top 1.3% of submissions): 2017
- CMU School of Computer Science University Honors: 2017
- CMU School of Computer Science Dean's List: 2015, 2016, 2017
- Harvard University Hacking Eating Tracking Hackathon Judge's Choice Award: 2015
Teaching
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2020, CMU. Instructor: Louis-Philippe Morency
- Head TA & Lecturer: 11-777 Multimodal Machine Learning, Fall 2019, CMU. Instructor: Louis-Philippe Morency
- TA: 10-708 Probabilistic Graphical Models, Spring 2019, CMU. Instructor: Eric Xing
- TA: 10-715 Advanced Introduction to Machine Learning, Fall 2018, CMU. Instructor: Maria-Florina Balcan
- TA: 10-601 Introduction to Machine Learning, Fall 2016, CMU. Instructor: Roni Rosenfeld
- TA: 15-213/18-213/15-513 Introduction to Computer Systems, Summer 2016, CMU. Instructor: Brian Railing
Academic Talks
- Towards Real-World Social AI
Adobe Research, Online, Jan 2021
CMU, Online, Oct 2020
[slides]
- Towards Debiasing Sentence Representations
ACL 2020, Online, July 2020
[slides]
- On Emergent Communication in Competitive Multi-Agent Teams
AAMAS 2020, Online, May 2020
[slides]
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
NeurIPS 2019 Workshop on Federated Learning, Vancouver, Canada, Dec 2019
[slides]
- Learning Sparse Representations of Discrete Objects
ICLR 2021, Online, May 2021
Google Research, Mountain View, CA, USA, Aug 2019
[slides]
- Computational Modeling of Human Multimodal Language
Google Research, Mountain View, CA, USA, July 2019
RIKEN Artificial Intelligence Project Machine Learning Seminar, Tokyo, Japan, Jan 2019
RIKEN Artificial Intelligence Project Machine Learning Seminar, Kyoto, Japan, Dec 2018
ACL 2018, Melbourne, Australia, July 2018
CMU Machine Learning Department Data Analysis Project Presentation, Pittsburgh, PA, USA, Apr 2018
[slides]
- Learning Robust Joint Representations by Cyclic Translations Between Modalities
AAAI 2019, Honolulu, HI, USA, Feb 2019
NeurIPS 2018 Workshop on Interpretability and Robustness, Montreal, Canada, Dec 2018
[slides]
- Multimodal Language Analysis with Recurrent Multistage Fusion
NeurIPS 2018 Workshop on Spatiotemporal Modeling, Montreal, Canada, Dec 2018
EMNLP 2018, Brussels, Belgium, Nov 2018
[slides]
- Advances in Multimodal Datasets
First Workshop on Human Multimodal Language at ACL 2018, Melbourne, Australia, July 2018
[slides]
- Multi-attention Recurrent Network for Human Communication Comprehension
AAAI 2018, New Orleans, LA, USA, Feb 2018
[slides]
Professional Activities
- Workflow Chair: ICML 2019
- Session Chair: AAAI 2019
- General Chair: NAACL 2021 Third Workshop on Multimodal Artificial Intelligence
- General Chair: NeurIPS 2020 First Workshop on Quantum Tensor Networks in Machine Learning
- General Chair: ACL 2020 Second Grand Challenge and Workshop on Human Multimodal Language
- General Chair: ACL 2018 First Grand Challenge and Workshop on Human Multimodal Language
- Senior PC Member: IJCAI
- Conference Program Committee: NeurIPS, ICML, ICLR, ACL, EMNLP, NAACL, EACL, COLING, IJCNLP, AACL, CVPR, ICCV, ECCV, WACV, ACCV, AAAI, IJCAI, AISTATS, UAI, CHI, ICMI, FG, ACML, ML4H, CHIL, ACM Multimedia, Interspeech
- Workshop Program Committee: NeurIPS workshop on Meta Learning, NeurIPS workshop on Machine Learning for Health, ICLR workshop on Embodied Multimodal Learning, ICLR workshop on Never-ending RL, ICLR workshop on Enormous Language Models, ACL workshop on Multimodal Language, EMNLP workshop on NLP Open Source Software, EMNLP workshop on NLP Beyond Text, NAACL workshop on Trustworthy NLP, NAACL workshop on Multimodal AI, IJCAI workshop on Federated Learning, WWW workshop on NLP Beyond Text
- Journal Reviewer: IEEE Transactions on Affective Computing, IEEE Transactions on Multimedia, IEEE Transactions on Cybernetics, IEEE Computational Intelligence Magazine, IEEE Signal Processing Letters, Elsevier Information Fusion, Elsevier Computer Speech and Language, Machine Learning
- CMU AI+ Committee: 2019, 2020
- CMU Machine Learning Blog Editorial Board: 2019, 2020
- CMU AI Undergraduate Research Mentor: 2018, 2019, 2020
- CMU Machine Learning Department PhD Admissions Committee: 2018, 2019, 2020
- CMU Machine Learning Department Masters Admissions Committee: 2017, 2018
- CMU Singapore Students Association Co-President: 2015
I have an Erdős number of 3 (Paul Erdős → Giuseppe Melfi → Erik Cambria → Paul Pu Liang).
This page has been accessed at least
times since Feb 8, 2018.