Paul Pu Liang

@pliang279

@lpwinniethepu

News

• 06/2019: I am compiling a reading list for multimodal ML containing papers, software, workshops, tutorials, and courses! It spans various modalities (language, vision, speech, video, touch) and applications (QA, dialog, RL, reasoning, grounding, navigation, affective computing, healthcare, robotics). Links to code and data are included. I'll be updating regularly, if there's anything I missed, please let me know.
• 06/2019: New preprints on fair federated learning via local models, learning to abstain under uncertainty, and generative models for incomplete data.
• 05/2019: 2 papers accepted at ACL 2019 on learning from imperfect and unaligned multimodal time series data.
• 04/2019: I will be the head TA for 11-777 Advanced Multimodal Machine Learning in Fall 2019, with new content on multimodal RL, unsupervised alignment, knowledge graphs, and interpretable learning!
• 02/2019: We will present 2 papers at NAACL and CVPR 2019: strong baselines for multimodal learning and a new QA dataset on comprehending social interactions.
• 01/2019: I will be a TA for 10-708 Probabilistic Graphical Models in Spring 2019, with new content on deep generative models, RL, and probabilistic programming!
• 12/2018: Excited to be a workflow chair for ICML 2019!
• 12/2018: "Learning Factorized Multimodal Representations" to appear at ICLR 2019.
• 11/2018: 2 papers accepted at AAAI 2019: multimodal cyclic translations and dynamic word representations.
• 10/2018: 2 papers on multimodal NLP [1,2] presented at EMNLP 2018 and ICMI 2018.
• 08/2018: I will be a TA for 10-715 Advanced Introduction to Machine Learning in Fall 2018.
• 05/2018: I completed my Master's Data Analysis Project research and received the best presentation 1st runner-up award.
• 04/2018: 3 papers presented at ACL 2018 main conference and workshops: CMU-MOSEI dataset, low-rank fusion, and seq2seq2sentiment.
• 01/2018: I will be co-organizing the First Grand Challenge and Workshop on Human Multimodal Language at ACL 2018.
• 11/2017: 2 papers on neural networks [1,2] for multimodal communication accepted at AAAI 2018.
• 11/2017: Our paper on gated multimodal fusion won the honorable mention award at ICMI 2017!

Education

• Carnegie Mellon University, Aug 2018 - Present
Doctor of Philosophy in Machine Learning
Advisors: Louis-Philippe Morency and Ruslan Salakhutdinov
GPA: 4.23/4.00
• Carnegie Mellon University, Aug 2017 - May 2018
Master of Science in Machine Learning
Advisor: Louis-Philippe Morency
Thesis: Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
GPA: 4.24/4.00
• Carnegie Mellon University, Aug 2014 - May 2017
Bachelor of Science with University Honors in Computer Science
Minor in Neural Computation
GPA: 3.87/4.00

Experience

• Google Research, May 2019 - Present
Research Intern
Advisors: Manzil Zaheer, Yuan Wang, and Amr Ahmed
• RIKEN Artificial Intelligence Project, Dec 2018 - Jan 2019
Visiting Researcher
Advisors: Makoto Yamada, Qibin Zhao, and Masashi Sugiyama

Publications

2019

24. On Emergent Communication in Competitive Multi-Agent Games
Paul Pu Liang, Jeffrey Chen, Satwik Kottur, Ruslan Salakhutdinov, Louis-Philippe Morency
arXiv 2019
[paper] [arXiv]
25. Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
arXiv 2019
[paper] [arXiv]
26. Revisiting Disentanglement in Unsupervised Text Style Transfer
Paul Pu Liang*, Hai Pham*, Ramon Sanabria*, Harvey Zhang, Ruslan Salakhutdinov, Louis-Philippe Morency, Florian Metze, Barnabás Poczós
arXiv 2019
[paper] [arXiv]
27. Think Locally, Act Globally: Federated Learning with Local and Global Representations
Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Ruslan Salakhutdinov, Louis-Philippe Morency
arXiv 2019
[paper] [arXiv]
28. Deep Gamblers: Learning to Abstain with Portfolio Theory
Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, Masahito Ueda
arXiv 2019
[paper] [arXiv]
29. Variational Auto-Decoder
Amir Zadeh, Yao Chong Lim, Paul Pu Liang, Louis-Philippe Morency
arXiv 2019
[paper] [arXiv] [code]
30. Hierarchical Seq2Seq: Learning Representations of Online Spoken Speech
Irene Li, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
arXiv 2019
[paper] [arXiv] [code]
31. Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Paul Pu Liang*, Zhun Liu*, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
ACL 2019
[paper] [arXiv]
32. Multimodal Transformer for Unaligned Multimodal Language Sequences
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
ACL 2019
[paper] [arXiv] [code]
33. Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, Louis-Philippe Morency
CVPR 2019 (oral)
[paper] [arXiv] [code] [slides] [poster]
34. Strong and Simple Baselines for Multimodal Utterance Embeddings
Paul Pu Liang*, Yao Chong Lim*, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency
NAACL 2019 (oral)
[paper] [arXiv] [code] [slides]
35. Learning Factorized Multimodal Representations
Paul Pu Liang*, Yao-Hung Hubert Tsai*, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2019
NeurIPS 2018 Workshop on Bayesian Deep Learning
[paper] [arXiv] [code] [poster]
36. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Paul Pu Liang*, Hai Pham*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
AAAI 2019
NeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech and Language (oral)
[paper] [arXiv] [code] [slides] [poster]
37. Words can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
AAAI 2019
[paper] [arXiv] [code] [slides] [poster]

2018

10. Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (best presentation runner-up)
[paper] [slides] [poster]
11. Multimodal Language Analysis with Recurrent Multistage Fusion
Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
EMNLP 2018 (oral)
NeurIPS 2018 Workshop on Modeling and Decision-making in the Spatiotemporal Domain (oral)
[paper] [arXiv] [slides] [poster]
12. Multimodal Local-Global Ranking Fusion for Emotion Recognition
Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ICMI 2018
[paper] [arXiv] [poster]
13. An Empirical Evaluation of Sketched SVD and its Application to Leverage Score Ordering
Hui Han Chin, Paul Pu Liang
ACML 2018
[paper] [arXiv] [slides] [poster]
14. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Amir Zadeh, Paul Pu Liang, Jonathan Vanbriesen, Soujanya Poria, Edmund Tong, Erik Cambria, Minghai Chen, Louis-Philippe Morency
ACL 2018 (oral)
[paper] [arXiv] [slides] [code]
15. Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Zhun Liu, Ying Shen, Varun Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
ACL 2018 (oral)
[paper] [arXiv] [slides] [code]
16. Proceedings of the First Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency, Soujanya Poria, Erik Cambria, Stefan Scherer
ACL 2018 Workshop Proceedings
[proceedings] [website] [introduction] [datasets] [results]
17. Multi-attention Recurrent Network for Human Communication Comprehension
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, Louis-Philippe Morency
AAAI 2018 (oral)
[paper] [arXiv] [slides] [code]
18. Memory Fusion Network for Multi-view Sequential Learning
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
AAAI 2018 (oral)
[paper] [arXiv] [slides] [code]

2017

1. Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
Paul Pu Liang*, Minghai Chen*, Sen Wang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
ICMI 2017 (oral, honorable mention award)
[paper] [arXiv] [slides]

Honors

• CMU Graduate Research Fellowship, 2018-2023
• CMU Puzzlehunt 1st Place, 2019
• CMU Machine Learning Department Data Analysis Project Best Presentation 1st Runner-up, 2018
• NSF Graduate Research Fellowship Program Honorable Mention Award, 2018
• ACM ICMI 2017 Best Paper Honorable Mention Award, 2017
• CMU School of Computer Science University Honors, 2017
• CMU School of Computer Science Dean's List, 2015-2017
• Harvard University Hacking Eating Tracking Hackathon Judge's Choice Award, 2015

Teaching

• Head TA: 11-777 Advanced Multimodal Machine Learning, Fall 2019, CMU. Instructor: Louis-Philippe Morency
• TA: 10-708 Probabilistic Graphical Models, Spring 2019, CMU. Instructor: Eric Xing
• TA: 10-715 Advanced Introduction to Machine Learning, Fall 2018, CMU. Instructor: Maria-Florina Balcan
• TA: 10-601 Introduction to Machine Learning, Fall 2016, CMU. Instructor: Roni Rosenfeld
• TA: 15-213/18-213/15-513 Introduction to Computer Systems, Summer 2016, CMU. Instructor: Brian Railing

Student Advising

• Terrance Liu, M.S. in Machine Learning, CMU
Federated representation learning, adversarial learning

• Irene Li, M.S. in Computer Science, CMU
Commonsense reasoning, knowledge graphs, debiasing sentence representations, speech representations

• Zhun Liu, M.S. in Intelligent Information Systems, CMU
Deep generative models, multimodal learning, tensor representations

• Ying Shen, M.S. in Intelligent Information Systems, CMU
Interpretable learning, multimodal learning, tensor representations, bayesian ranking algorithms

• Peter Wu, B.S. in Computer Science, CMU
Deep reinforcement learning, meta learning

• Jeffrey Chen, B.S. in Computer Science, CMU
Emergent communication

• Senyu Tong, B.S. in Mathematics and Computer Science, CMU
Sequence models

• Emily Zheng, B.S. in Artificial Intelligence, CMU
Debiasing sentence representations

Academic Talks

• Learning Sparse Representations of Discrete Objects
Google Research, Mountain View, CA, USA, Aug 2019

• Computational Modeling of Human Multimodal Language
Google Research, Mountain View, CA, USA, July 2019
RIKEN Artificial Intelligence Project Machine Learning Seminar, Tokyo, Japan, Jan 2019
RIKEN Artificial Intelligence Project Machine Learning Seminar, Kyoto, Japan, Dec 2018
University of Electronic Science and Technology of China, Chengdu, China, Aug 2018
ACL 2018, Melbourne, Australia, July 2018
Jilin University, Changchun, China, May 2018
CMU Machine Learning Department Data Analysis Project Presentation, Pittsburgh, PA, USA, Apr 2018
[slides]
• Learning Robust Joint Representations by Cyclic Translations Between Modalities
NeurIPS 2018 Workshop on Interpretability and Robustness, Montreal, Canada, Dec 2018
[slides]
• Multimodal Language Analysis with Recurrent Multistage Fusion
NeurIPS 2018 Workshop on Spatiotemporal Modeling, Montreal, Canada, Dec 2018
EMNLP 2018, Brussels, Belgium, Nov 2018
[slides]
• Advances in Multimodal Datasets
First Workshop on Human Multimodal Language at ACL 2018, Melbourne, Australia, July 2018
[slides]
• Multi-attention Recurrent Network for Human Communication Comprehension
AAAI 2018, New Orleans, LA, USA, Feb 2018
[slides]
• Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
University of Electronic Science and Technology of China, Chengdu, China, Dec 2017
[slides]

Professional Activities

• Workflow Chair: ICML 2019
• Session Chair: AAAI 2019
• General Chair: ACL 2018 First Grand Challenge and Workshop on Human Multimodal Language
• Conference PC Member or Reviewer: NeurIPS, ICML, ICLR, ACL, EMNLP, NAACL, CVPR, ICCV, AAAI, ICMI, FG
• Journal Reviewer: IEEE Transactions on Affective Computing, IEEE Computational Intelligence Magazine, Elsevier Information Fusion, Elsevier Computer Speech and Language
• CMU Machine Learning Department PhD Admissions Committee
• CMU Machine Learning Department Masters Admissions Committee
• CMU AI Undergraduate Research Mentor