Zhou Yu   俞舟

PhD Student

Language Technology Institute, Carnegie Mellon University

Address:

GHC 6605, 5000 Forbes Ave, Pittsburgh, PA 15213

zhouyu@cs.cmu.edu


Welcome

I am a 5th year PhD student at the Language Technology Institute under School of Computer Science, Carnegie Mellon University, working with Prof. Alan W Black and Prof. Alex I. Rudnicky in LTI. 2015 Summer, I interned with Prof. David Suendermann-Oeft in ETS San Francisco Office on cloud based mulitmodal dialog systems. 2014 Fall, I interned with Dan Bohus and Eric Horvitz in Microsoft Research on situated multimodal dialogue systems.

Prior to CMU, I received a B.S. in Computer Science and a B.A. in English Language with Linguistics focus from Zhejiang University in 2011. I worked with Prof. Xiaofei He and Prof. Deng Cai there on Machine Learning and Computer Vision. I also worked with Prof. Yunhua Qu on Machine Translation for my English Langauge degree.


Research Interests

My research aims to leverage automatic obtainable multimodal information with machine learning methods to make conversations more nature and effective. The dynamics of both verbal and nonverbal behaviors of the conversational parties contribute to the process and outcome of the conversation. In order to understand human-human and human-system interactions and improve the underlying model of the system, I design methods to predict conversation partners' attention and engagement in real time using both verbal and nonverbal behaviors, such as gaze and smiles. Then I leverage these signals to change system's conversatioanl strategies on the fly to accomendate users.


News

We just published our code and the collected database for our chatbot: TickTock. Here is the webpage just for TickTock. You can interact with it in a web service to get the first hand experience of a Chatbot. I am also co-organizing a share task on Chatbot, you can also participate in the share task here: webpage

Our papers "Multimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog Framework" and "Speed vs. Accuracy: Designing an Optimal ASR System for Spontaneous Non-Native Speech in a Real-Time Application" got accepted in IWSDS 2016.

Nov 10th-15th: I will be in Seattle for ICMI 2016. Here is a copy of my doctorial consortian paper in ICMI.[Pdf]

Nov 8th-10th: I will be in MIT EECS for the RisingStars workshop [Link]

Nov 6fh: I will give a talk in Tim Bickmore's group in Northeastern.

Nov 5th: I will give a talk about my thesis topic in Jim Glass's group in MIT EECS

Oct 9th: We will demo Video-HALEF in ASRU 2015 this year! Our demo paper "VIDEO-HALEF: A Modular Open-Source Standard-Compliant Dialog Ssystem Framework With Video Support" got accepted to ASRU 2015

Oct 7th: Will be in Montreal for NIPS this Dec. My extended abstract "Multimodal Sequence Feature Fusion for Interactive Conversation Modeling" is accepted to NIPS WiML workshop.

Sep 16th 10:30am EST, CMU GHC 6605. I proposed my thesis topic "Engagement in Multimodal Interactive Conversational Systems". [Slides]

Sep 13th: Our paper "Using Bidirectional LSTM Recurrent Neural Networks to Learn High-Level Abstractions of Sequential Features for Automated Scoring of Non-Native Spontaneous Speech" got accepted to ASRU 2015.

Sep 8th: The YRRSDS organizers published all the discussion notes on the website [WebPage Link]. It might be useful to people in the dialog system community.

Aug 31st: I gave a tutorial on HALEF, a multimodal dialog system in YRRSDS in Prague, see the tutorial announcement in the program: [Program Link] [Tutorial Document]

Aug 6th: I gave a talk in ICSI Berkeley, the talk announcement: https://www.icsi.berkeley.edu/icsi/events/2015/08/yu-dialog-systems

Aug 4th: I gave a talk in SRI on Attention and Engagement Aware Dialog Systems.[slides]

July: Our paper "Incremental Coordination: Attention-Centric Speech Production in a Physically Situated Conversational Agent" got accepted in SIGDIAL 2015.

July: Our paper "Incremental Coordination: Attention-Centric Speech Production in a Physically Situated Conversational Agent" got accepted in SIGDIAL 2015.

June: I am co-organizing the Workshop on Spoken Dialogue Systems for PhDs, PostDocs & New Researchers (YRRSDS) this year. It is a satellite workshop of SIGDIAL. Please consider to submit papers. https://sites.google.com/site/yrrsdsmmxv/. The deadline is July 3rd 2015.


Demo Videos

TickTock: an engagement aware multimodal dialog system
- below is a demo of a participant chatting with TickTock.

Direction-giving Robot: an attention aware direction-giving humanoid robot
- below is a demo and some real user cases of people interacting with the robot.

HALEF: a distributed web-based multimodal dialog system
- below is a demo of Zhou applying for a pizza delivery job. Users can access it through
a web browser. It live streams videos from usrs' local webcam to the server.


Selected Publications

-Zhou Yu, Vikram Ramanarayanan, Robert Mundkowsky, Patrick Lange, Alan Black, Alexei Ivanov, David Suendermann-Oeft, Multimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog FrameworkMultimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog Framework, to appear in IWSDS 2016.[pdf]

-Alexei Ivanov, Patrick Lange, David Suendermann-Oeft, Vikram Ramanarayanan, Yao Qian, Zhou Yu and Jidong Tao, Speed vs. Accuracy: Designing an Optimal ASR System for Spontaneous Non-Native Speech in a Real-Time Application, to appear in IWSDS 2016. [pdf]

-Zhou Yu,Vikram Ramanarayanan, David Suendermann-Oeft, Xinhao Wang, Klaus Zechner, Lei Chen, Jidong Tao and Yao Qian, Using Bidirectional LSTM Recurrent Neural Networks to Learn High-Level Abstractions of Sequential Features for Automated Scoring of Non-Native Spontaneous Speech, to appear in ASRU 2015.[pdf]

-Zhou Yu, Dan Bohus and Eric Horvitz, Incremental Coordination: Attention-Centric Speech Production in a Physically Situated Conversational Agent, SIGDIAL 2015. [pdf]

- Zhou Yu, Alexandros Papangelis, Alex Rudnicky, TickTock: Engagement Awareness in a non-Goal-Oriented Multimodal Dialogue System, AAAI Spring Symposium on Turn-taking and Coordination in Human-Machine Interaction 2015.[pdf][slides]

- Zhou Yu, Stefan Scherer, David Devault, Jonathan Gratch, Giota Stratou, Louis-Philippe Morency and Justine Cassell, Multimodal Prediction of Psychological Disorder: Learning Verbal and Nonverbal Commonality in Adjacency Pairs, SEMDIAL 2013. [pdf] [slides]

- Zhou Yu, Deng Cai, Xiaofei He, Error-correcting Output Hashing in Fast Similar Search, Best paper in The Second International Conference on Internet Multimedia Computing and Service, ICIMCS Harbin, China,Dec.2010. [pdf]


More about Zhou

- CV [pdf]