Louis-Philippe Morency

Finmeccanica Associate Professor, Language Technology Institute,

School of Computer Science, Carnegie Mellon University

Director, MultiComp Lab

Gates-Hillman Center (GHC) Office 5411,

5000 Forbes Avenue, Pittsburgh, PA 15213

Email: morency@cs.cmu.edu

Phone: (412) 268-5508

 

I am tenure-track Faculty at CMU Language Technology Institute where I lead the Multimodal Communication and Machine Learning Laboratory (MultiComp Lab). I was previously Research Faculty at USC Computer Science Department. I received my Ph.D. in Computer Science from MIT Computer Science and Artificial Intelligence Laboratory.

My research focuses on building the computational foundations to enable computers with the abilities to analyze, recognize and predict subtle human communicative behaviors during social interactions. Central to this research effort is the technical challenge of multimodal machine learning: mathematical foundation to study heterogeneous multimodal data and the contingency often found between modalities. This multi-disciplinary research topic overlaps the fields of multimodal interaction, social psychology, computer vision, machine learning and artificial intelligence, and has many applications in areas as diverse as medicine, robotics and education.

Multimodal Machine Learning

·        Probabilistic modeling of acoustic, visual and verbal modalities

·        Learning the temporal contingency between modalities

Human Communication Dynamics

·        Analyze, recognize and predict subtle human communicative behaviors during social interactions.

Health Behavior Informatics

·        Technologies to support clinical practice during diagnosis and treatment of mental health disorders

News

-         Congratulations KangGeon Kim for his successful PhD thesis defense at USC!

-         4 papers presented at the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)

-         Tutorial on Multimodal Machine Learning at ACL 2017, ICMI 2016 and CVPR 2016 (slides and website)

-         New survey on multimodal machine learning!! (arxiv)

-         Program chair for IEEE Conference on Automatic Face and Gesture Recognition (FG 2018)

-         3 long papers presented at Annual Meeting of the Association for Computational Linguistics (ACL 2017)

-         Congratulations to Erroll for the Emerging Investigator Award at ACM Siggraph Symposium on Eye Tracking Research & Applications, 2016

-         Congratulations Sunghyun Park for his successful PhD thesis defense!!

-         Best paper award, ACM International Conference on Multimodal Interaction, Seattle, WA, 2015

-         Best demo paper award at International Conference on Affective Computing and Intelligent Interaction (ACII), 2015

-         NetExplo 2014 Award, in partnership with UNESCO, as one of the year’s top ten most promising digital initiatives

Selected Publications (see full list in my CV, MultiComp Lab or Google Scholar)

§  C. Ahuja and L.-P. Morency, Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  V. Cirik, T. Berg-Kirkpatrick and L.-P. Morency, Using Syntax to Ground Referring Expressions in Natural Images, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  A. Zadeh, P. Liang, S. Poria, P. Vij, E. Cambria, and L.-P. Morency, Multi-attention Recurrent Network for Human Communication Comprehension, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  A. Zadeh, P. Liang, N. Mazumder, S. Poria, E. Cambria and L.-P. Morency, Memory Fusion Network for Multi-view Sequential Learning, In Proceedings of the Thirty-Second AAAI Annual Conference on Artificial Intelligence (AAAI), 2018

§  E. Tong, A. Zadeh, C. Jones and L.-P. Morency, Combating Human Trafficking with Very Deep Multimodal Models, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Pouria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh and L.-P. Morency, Context-Dependent Sentiment Analysis in User-Generated Videos, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  S. Ghosh, M. Chollet, E. Laksana, L.-P. Morency and S. Scherer, Affect-LM: A Neural Language Model for Customizable Affective Text Generation, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

§  W. Pei, T. Baltrušaitis, D. Tax and L.-P. Morency. Temporal Attention-Gated Model for Robust Sequence Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

§  A. Zadeh, M. Chen, S. Poria, E. Cambria and L.-P. Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017

§  S. Rajagopalan, L.-P. Morency, T. Baltrus̆aitis and R. Goecke, Extending Long Short-Term Memory for Multi-View Structured Learning, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  E. Wood, T. Baltrušaitis, L.-P. Morency, P. Robinson and Andreas Bulling, A 3D Morphable Eye Region Model for Gaze Estimation, In Proceedings of the European Conference on Computer Vision (ECCV), 2016

§  H. Yu, S. Zhang and L.-P. Morency, Unsupervised Text Recap Extraction for TV Series, In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

§  S. Stratou and L.-P. Morency, MultiSense - Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case, IEEE Transactions on Affective Computing, 2016

§  M. Chatterjee, S. Park, S. Scherer and L.-P. Morency. Combining Two Perspectives on Classifying Multimodal Data for Recognizing Speaker Traits. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), 2015 *** Best Paper Award ***

§  K. Bousmalis, S. Zafeiriou, L.-P. Morency, M. Pantic and Z. Ghahramani, Variational Infinite Hidden Conditional Random Fields, IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI), Volume 37, Issue 9, September 2015, Pages 1917-1929

§  T. Baltrusaitis, P. Robinson and L.-P. Morency, Continuous Conditional Neural Fields for Structured Regression, ECCV 2014

§  S. Park, P. Shoemark, and L.-P. Morency.“Toward Crowdsourcing Micro-Level Behavior Annotations: The Challenges of Interface, Training, and Generalization.” IUI 2014, ***Best Paper Award finalist ***

§  Y Song, L.-P. Morency and R. Davis. Action Recognition by Hierarchical Sequence Summarization. CVPR 2013

§  V. P. Rosas, R. Mihalcea and L.-P. Morency. Utterance-Level Multimodal Sentiment Analysis. ACL 2013

§  S. Scherer, G. Stratou, J. Boberg, J. Gratch, A. Rizzo and L.-P. Morency. Automatic Behavior Descriptors for Psychological Disorder Analysis. FG 2013, *** Best paper award ***

Graduate Students Advising

Amir Ali Bagherzade, Ph.D. program (CMU)

Chaitanya Ahuja, Ph.D. program (CMU)

Volkan Cirik, Ph.D. program (CMU)

Alexandria Vail, Ph.D. program (CMU)

Torsten Wörtwein, M.S. program (CMU)

Sayan Ghosh, Ph.D. program (USC, with Prof. Stefan Scherer)

Teaching

       CMU-11777: Advanced Multimodal Machine Learning, Spring 2016, Spring 2017, Fall 2017

Fundamental mathematical concepts related to multimodal machine learning including multimodal alignment and fusion, heterogeneous representation learning and multi-stream temporal modeling.

·        Webpage of Spring 2017 semester which includes syllabus and lecture slides

https://piazza.com/cmu/spring2017/11777/home

       CMU-11776: Multimodal Affective Computing, Fall 2015, Fall 2016

Recent advances in machine learning, pattern recognition and signal processing to analyze, recognize and predict human communication behaviors during social interactions.

·        Webpage of Fall 2016 semester which includes syllabus and lecture slides

https://piazza.com/cmu/fall2016/11776/home

       USC-CS599: Human Communication and Machine Learning, Fall 2010, 2012, 2014

Selected Press Coverage