Louis-Philippe Morency

Assistant Professor, Language Technology Institute,

School of Computer Science, Carnegie Mellon University

Director, MultiComp Lab

Gates-Hillman Center (GHC) Office 5411,

5000 Forbes Avenue, Pittsburgh, PA 15213

Email: morency@cs.cmu.edu

Phone: (412) 268-5508


I am tenure-track Faculty at CMU Language Technology Institute where I lead the Multimodal Communication and Machine Learning Laboratory (MultiComp Lab). I was previously Research Faculty at USC Computer Science Department. I received my Ph.D. in Computer Science from MIT Computer Science and Artificial Intelligence Laboratory.

My research focuses on building the computational foundations to enable computers with the abilities to analyze, recognize and predict subtle human communicative behaviors during social interactions. I have formalized this new research endeavor with the Human Communication Dynamics framework, addressing four key computational challenges: behavioral dynamic, multimodal dynamic, interpersonal dynamic and societal dynamic. This multi-disciplinary research topic overlaps the fields of multimodal interaction, social psychology, computer vision, machine learning and artificial intelligence, and has many applications in areas as diverse as medicine, robotics and education.

Human Communication Dynamics

·       Analyze, recognize and predict subtle human communicative behaviors during social interactions.

Multimodal Machine Learning

·       Probabilistic modeling of acoustic, visual and verbal modalities

·       Learning the temporal contingency between modalities

Health Behavior Informatics

·       Technologies to support clinical practice during diagnosis and treatment of mental health disorders


-          Tutorial on Multimodal Machine Learning at ACL 2017, ICMI 2016 and CVPR 2016 (slides and website)

-          After more than 2 years, we published the survey on multimodal machine learning (arxiv)

-          Program chair for IEEE Conference on Automatic Face and Gesture Recognition (FG 2018)

-          3 long papers presented at Annual Meeting of the Association for Computational Linguistics (ACL 2017)

-          Congratulations to Erroll for the Emerging Investigator Award at ACM Siggraph Symposium on Eye Tracking Research & Applications, 2016

-          Congratulations Sunghyun Park for his successful PhD thesis defense!!

-          Best paper award, ACM International Conference on Multimodal Interaction, Seattle, WA, 2015

-          Best demo paper award at International Conference on Affective Computing and Intelligent Interaction (ACII), 2015

-          NetExplo 2014 Award, in partnership with UNESCO, as one of the year’s top ten most promising digital initiatives

Selected Publications (see full list in my CV, MultiComp Lab or Google Scholar)

§  S. Stratou and L.-P. Morency, MultiSense - Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case, IEEE Transactions on Affective Computing, 2016

§  E. Tong, A. Zadeh, C. Jones and L.-P. Morency, Combating Human Trafficking with Very Deep Multimodal Models, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Pouria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh and L.-P. Morency, Context-Dependent Sentiment Analysis in User-Generated Videos, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Ghosh, M. Chollet, E. Laksana, L.-P. Morency and S. Scherer, Affect-LM: A Neural Language Model for Customizable Affective Text Generation, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  W. Pei, T. Baltrušaitis, D. Tax and L.-P. Morency. Temporal Attention-Gated Model for Robust Sequence Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

§  A. Zadeh, M. Chen, S. Poria, E. Cambria and L.-P. Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017

§  H. Wang, A. Meghawat, L.-P. Morency and E. Xing. Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis. In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), 2017

§  S. Rajagopalan, L.-P. Morency, T. Baltrus̆aitis and R. Goecke, Extending Long Short-Term Memory for Multi-View Structured Learning, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  E. Wood, T. Baltrušaitis, L.-P. Morency, P. Robinson and Andreas Bulling, A 3D Morphable Eye Region Model for Gaze Estimation, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  H. Yu, S. Zhang and L.-P. Morency, Unsupervised Text Recap Extraction for TV Series, In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

§  M. Chatterjee, S. Park, S. Scherer and L.-P. Morency. Combining Two Perspectives on Classifying Multimodal Data for Recognizing Speaker Traits. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), 2015 *** Best Paper Award ***

§  K. Bousmalis, S. Zafeiriou, L.-P. Morency, M. Pantic and Z. Ghahramani, Variational Infinite Hidden Conditional Random Fields, IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI), Volume 37, Issue 9, September 2015, Pages 1917-1929

§  T. Wortwein, M. Chollet, A. Shapiro, S. Scherer, and L.-P. Morency. Exploring Feedback Learning Strategies to Improve Public Speaking: An Interactive Virtual Audience Framework. ACM International Joint Conference on Pervasive and Ubiquitous Computing (Ubicomp), 2015

§  T. Baltrusaitis, P. Robinson and L.-P. Morency, Continuous Conditional Neural Fields for Structured Regression, ECCV 2014

§  S. Park, P. Shoemark, and L.-P. Morency.“Toward Crowdsourcing Micro-Level Behavior Annotations: The Challenges of Interface, Training, and Generalization.” IUI 2014, ***Best Paper Award finalist ***

§  Y Song, L.-P. Morency and R. Davis. Action Recognition by Hierarchical Sequence Summarization. CVPR 2013

§  V. P. Rosas, R. Mihalcea and L.-P. Morency. Utterance-Level Multimodal Sentiment Analysis. ACL 2013

§  S. Scherer, G. Stratou, J. Boberg, J. Gratch, A. Rizzo and L.-P. Morency. Automatic Behavior Descriptors for Psychological Disorder Analysis. FG 2013, *** Best paper award ***

Graduate Students Advising

Amir Ali Bagherzade, Ph.D. program (CMU)

Chaitanya Ahuja, Ph.D. program (CMU)

Volkan Cirik, Ph.D. program (CMU)

Alexandria Vail, Ph.D. program (CMU)

Torsten Wörtwein, M.S. program (CMU)

Sayan Ghosh, Ph.D. program (USC, with Prof. Stefan Scherer)

KangGeon Kim, Ph.D. program (USC, with Prof. Gerard Medioni)


       CMU-11777: Advanced Multimodal Machine Learning, Spring 2016, Spring 2017, Fall 2017

Fundamental mathematical concepts related to multimodal machine learning including multimodal alignment and fusion, heterogeneous representation learning and multi-stream temporal modeling.

·       Webpage of Spring 2017 semester which includes syllabus and lecture slides


       CMU-11776: Multimodal Affective Computing, Fall 2015, Fall 2016

Recent advances in machine learning, pattern recognition and signal processing to analyze, recognize and predict human communication behaviors during social interactions.

·       Webpage of Fall 2016 semester which includes syllabus and lecture slides


       USC-CS599: Human Communication and Machine Learning, Fall 2010, 2012, 2014

Selected Press Coverage