Paul Liang, CMU

Paul Pu Liang

Email: pliang(at)cs.cmu.edu
Office: Gates and Hillman Center 8011
5000 Forbes Avenue, Pittsburgh, PA 15213
Machine Learning Department and Language Technologies Institute, School of Computer Science, Carnegie Mellon University

[CV] @pliang279 @pliang279 @lpwinniethepu
I am a fourth-year Ph.D. student in the Machine Learning Department at Carnegie Mellon University, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. I also collaborate closely with Manuel Blum, Lenore Blum, and Daniel Rubin at Berkeley and Stanford. My research lies in the foundations of multimodal machine learning with applications in socially intelligent AI, understanding human and machine intelligence, natural language processing, healthcare, and education. As steps towards this goal, I work on:

  • Foundations of multimodal machine learning: representation, translation, fusion, and alignment of heterogeneous data [HighMMT, Brainish multimodal language, MultiBench, factorized representations, translation, alignment].
  • Social intelligence: AI that can perceive human behaviors and engage in multimodal interactions in embodied environments [CMU-MOSEI, Social-IQ, sentiment analysis, emotion recognition].
  • Human-centered AI applications in language, vision, speech, robotics, healthcare, and education [controllable text generation, learning from mobile health data].
  • Real-world representation learning: learning fair, robust, interpretable, efficient, and generalizable representations [fairness in language models, fairness in sentence representations, federated learning, robustness, sparse embeddings].

    • My research is generously supported by a Facebook PhD Fellowship and a Center for Machine Learning and Health Fellowship, and has been recognized by awards at the NeurIPS 2019 workshop on federated learning and ICMI 2017. I regularly organize courses (CMU 11-877, CMU 11-777), workshops (NAACL 2022, NAACL 2021, ACL 2020, and ACL 2018), and tutorials (CVPR 2022, NAACL 2022) on multimodal machine learning and was a workflow chair for ICML 2019. Previously, I received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science and Neural Computation from CMU, where I am grateful for the mentorship of Louis-Philippe Morency, Ruslan Salakhutdinov, Tai Sing Lee, Roni Rosenfeld, and Ryan Tibshirani. I have also been fortunate to spend time at DeepMind, Facebook AI Research, Nvidia AI, Google Research, and RIKEN Artificial Intelligence Project.

    Research opportunities: I am happy to collaborate and answer questions about my research and CMU academic programs. If you are interested, please send me an email. I especially encourage students from underrepresented groups to reach out.

    [News] [Education] [Publications] [Honors] [Teaching] [Talks] [Activities]

    News

    Education

    Publications

    (* denotes joint first-authors)

    2022

    1. MultiViz: An Analysis Benchmark for Visualizing and Understanding Multimodal Models
      2. Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, Ruslan Salakhutdinov
      [arXiv] [code]
    2. Fundamentals of Multimodal Representation Learning: Towards Generalization and Quantification
      3. Paul Pu Liang
      PhD Thesis Proposal. Committee: Louis-Philippe Morency, Ruslan Salakhutdinov, Manuel Blum, Lenore Blum, Trevor Darrell
      [document]
    3. Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
      4. Paul Pu Liang
      Annual Meeting of the Association for the Scientific Study of Consciousness 2022
      [arXiv]
    4. HighMMT: Towards Modality and Task Generalization for High-Modality Representation Learning
      5. Paul Pu Liang*, Yiwei Lyu*, Xiang Fan, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov
      [arXiv] [code]
    5. Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
      6. Dong Won Lee, Chaitanya Ahuja, Paul Pu Liang, Sanika Natu, Louis-Philippe Morency
      [arXiv coming soon] [code]
    6. Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
      7. Martin Q. Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, Louis-Philippe Morency
      [arXiv]
    7. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models
      8. 442 authors including Paul Pu Liang
      [arXiv] [code]
    8. GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
      9. 77 authors including Paul Pu Liang
      [arXiv] [code]
    9. PACS: A Dataset for Physical Audiovisual Commonsense Reasoning
      10. Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
      ECCV 2022
      [arXiv] [code]
    10. DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
      11. Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, Louis-Philippe Morency
      AIES 2022
      [arXiv] [code]
    11. Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
      12. Liangqiong Qu*, Yuyin Zhou*, Paul Pu Liang*, Yingda Xia, Feifei Wang, Li Fei-Fei, Ehsan Adeli, Daniel Rubin
      CVPR 2022
      [arXiv] [code]
    12. Tutorial on Multimodal Machine Learning
      13. Louis-Philippe Morency, Paul Pu Liang, Amir Zadeh
      CVPR 2022 Tutorial, NAACL 2022 Tutorial
      [website]

    2021

    1. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
      2. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency
      NeurIPS 2021
      [arXiv] [website] [code]
    2. Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
      3. Peter Wu, Paul Pu Liang, Jiatong Shi, Ruslan Salakhutdinov, Shinji Watanabe, Louis-Philippe Morency
      Asia Pacific Signal and Information Processing Association Annual Summit and Conference 2021
      [arXiv] [code]
    3. Towards Understanding and Mitigating Social Biases in Language Models
      4. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov
      ICML 2021
      [arXiv] [code]
    4. Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
      5. Paul Pu Liang*, Terrance Liu*, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
      ACL 2021 (oral)
      [arXiv]
    5. Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
      6. Paul Pu Liang*, Peter Wu*, Liu Ziyin, Louis-Philippe Morency, Ruslan Salakhutdinov
      ACM Multimedia 2021 (oral), NeurIPS 2020 Workshop on Meta Learning
      [arXiv] [code]
    6. StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
      7. Yiwei Lyu*, Paul Pu Liang*, Hai Pham*, Eduard Hovy, Barnabás Póczos, Ruslan Salakhutdinov, Louis-Philippe Morency
      NAACL 2021
      [arXiv] [code]
    7. Proceedings of the Third Workshop on Multimodal Artificial Intelligence
      8. Amir Zadeh, Louis-Philippe Morency, Paul Pu Liang, Candace Ross, Ruslan Salakhutdinov, Soujanya Poria, Erik Cambria, Kelly Shi
      NAACL 2021 Workshop Proceedings
      [proceedings] [website]
    8. Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration
      9. Jivat Neet, Yiding Jiang, Paul Pu Liang
      ICLR 2021 Workshop on Embodied Multimodal Learning
      [arXiv]
    9. Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
      10. Paul Pu Liang, Manzil Zaheer, Yuan Wang, Amr Ahmed
      ICLR 2021
      [arXiv] [code]

    2020

    1. Multimodal Privacy-preserving Mood Prediction from Mobile Data: A Preliminary Study
      2. Terrance Liu*, Paul Pu Liang*, Michal Muszynski, Ryo Ishii, David Brent, Randy Auerbach, Nicholas Allen, Louis-Philippe Morency
      NeurIPS 2020 Workshop on Machine Learning for Mobile Health
      [arXiv]
    2. MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
      3. Amir Zadeh, Yansheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, Louis-Philippe Morency
      EMNLP 2020
      [paper]
    3. Diverse and Admissible Trajectory Prediction through Multimodal Context Understanding
      4. Seong Hyeon Park, Gyubok Lee, Manoj Bhat, Jimin Seo, Minseok Kang, Jonathan Francis, Ashwin R. Jadhav, Paul Pu Liang, Louis-Philippe Morency
      ECCV 2020, CVPR 2020 Argoverse competition (honorable mention award)
      [arXiv] [code]
    4. Towards Debiasing Sentence Representations
      5. Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
      ACL 2020
      [arXiv] [code]
    5. Proceedings of the Second Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
      6. Amir Zadeh, Louis-Philippe Morency, Paul Pu Liang, Soujanya Poria
      ACL 2020 Workshop Proceedings
      [proceedings] [website]
    6. On Emergent Communication in Competitive Multi-Agent Teams
      7. Paul Pu Liang, Jeffrey Chen, Ruslan Salakhutdinov, Louis-Philippe Morency, Satwik Kottur
      AAMAS 2020 (oral), NeurIPS 2019 Workshop on Emergent Communication
      [arXiv] [code] [slides]
    7. Empirical and Theoretical Studies of Multimodal Co-learning
      8. Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency
      Elsevier Information Fusion 2020
      [arXiv]

    2019

    1. Think Locally, Act Globally: Federated Learning with Local and Global Representations
      2. Paul Pu Liang*, Terrance Liu*, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
      NeurIPS 2019 Workshop on Federated Learning (oral, distinguished student paper award)
      [arXiv] [code]
    2. Deep Gamblers: Learning to Abstain with Portfolio Theory
      3. Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, Masahito Ueda
      NeurIPS 2019
      [arXiv] [code] [poster]
    3. Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
      4. Paul Pu Liang*, Zhun Liu*, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
      ACL 2019
      [arXiv] [poster]
    4. Multimodal Transformer for Unaligned Multimodal Language Sequences
      5. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
      ACL 2019
      [arXiv] [code]
    5. Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
      6. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, Louis-Philippe Morency
      CVPR 2019 (oral)
      [paper] [code] [poster]
    6. Strong and Simple Baselines for Multimodal Utterance Embeddings
      7. Paul Pu Liang*, Yao Chong Lim*, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency
      NAACL 2019 (oral)
      [arXiv] [code] [slides]
    7. Learning Factorized Multimodal Representations
      8. Yao-Hung Hubert Tsai*, Paul Pu Liang*, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov
      ICLR 2019, NeurIPS 2018 Workshop on Bayesian Deep Learning
      [arXiv] [code] [poster]
    8. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
      9. Hai Pham*, Paul Pu Liang*, Thomas Manzini, Louis-Philippe Morency, Barnabás Póczos
      AAAI 2019, NeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech and Language (oral)
      [arXiv] [code] [slides] [poster]
    9. Words can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
      10. Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
      AAAI 2019
      [arXiv] [code] [slides] [poster]

    2018

    1. Computational Modeling of Human Multimodal Language: The MOSEI Dataset and Interpretable Dynamic Fusion
      2. Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
      Master's Thesis, CMU Machine Learning Data Analysis Project 2018 (first runner-up award)
      [paper] [slides] [poster]
    2. Multimodal Language Analysis with Recurrent Multistage Fusion
      3. Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency
      EMNLP 2018 (oral), NeurIPS 2018 Workshop on Modeling and Decision-making in the Spatiotemporal Domain (oral)
      [arXiv] [slides] [poster]
    3. Multimodal Local-Global Ranking Fusion for Emotion Recognition
      4. Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
      ICMI 2018
      [arXiv] [poster]
    4. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
      5. Amir Zadeh, Paul Pu Liang, Jonathan Vanbriesen, Soujanya Poria, Edmund Tong, Erik Cambria, Minghai Chen, Louis-Philippe Morency
      ACL 2018 (oral)
      [arXiv] [code] [slides]
    5. Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
      6. Zhun Liu, Ying Shen, Varun Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
      ACL 2018 (oral)
      [arXiv] [code] [slides]
    6. Proceedings of the First Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
      7. Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency, Soujanya Poria, Erik Cambria, Stefan Scherer
      ACL 2018 Workshop Proceedings
      [proceedings] [website] [introduction] [datasets] [results]
    7. An Empirical Evaluation of Sketched SVD and its Application to Leverage Score Ordering
      8. Hui Han Chin, Paul Pu Liang
      ACML 2018
      [arXiv] [slides] [poster]
    8. Multi-attention Recurrent Network for Human Communication Comprehension
      9. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, Louis-Philippe Morency
      AAAI 2018 (oral)
      [arXiv] [code] [slides]
    9. Memory Fusion Network for Multi-view Sequential Learning
      10. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency
      AAAI 2018 (oral)
      [arXiv] [code] [slides]

    2017

    1. Multimodal Sentiment Analysis with Word-level Fusion and Reinforcement Learning
      2. Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrušaitis, Amir Zadeh, Louis-Philippe Morency
      ICMI 2017 (oral, honorable mention award)
      [arXiv] [code] [slides]

    Honors

    Teaching

    Student Advising

    Some amazing students I've had the pleasure of advising:

    Academic Talks

    Professional Activities


    I have an Erdős number of 3 (Paul Erdős → Giuseppe Melfi → Erik Cambria → Paul Pu Liang).
    This page has been accessed at least several times since Feb 8, 2018.