VASC Seminar Announcement
            =========================

Date: Monday, 5/1/00
Time: 3:45-4:45
Place: NSH 3002

Speaker:  Rajiv Sharma
          Penn State, Dept. CSE
          http://www.cse.psu.edu/~rsharma/

Title: Understanding Gestures in Multimodal Human-Computer Interaction

Abstract: 

Hand gestures and speech comprise the most important modalities of
human to human interaction.  Motivated by this, there has been a
considerable interest in incorporating these modalities for "natural"
human-computer interaction (HCI). To accommodate for the naturalness
of the interface it is imperative not to impose any predefined fixed
gestures on the user. We discuss how this causes a "chicken-and-egg"
problem of gathering valid data for research in speech/gesture
recognition without having an existing multimodal interface. We then
describe a domain analogous to HCI, that of a TV newsperson narrating
the weather with the help of a weather map. The gestures embedded in
weather narration, provide us with abundant data for statistical
studies on the relation between speech and gesture in the context of a
display.  We present a framework for continuous gesture recognition
using Hidden Markov Models.  It uses a reliable tracking algorithm
using adaptive color segmentation and Kalman filtering. We demonstrate
how gesture/speech co-occurrence analysis helps in improving
continuous gesture recognition.

We then describe an experimental system that has evolved out of the
study of the weather domain, where a user can interact in a fairly
unconstrained manner with the map of an urban area.  This interactive
map system (iMAP) is now being used as a test-bed for research in
multimodal HCI. Tasks are defined on the map that elicit natural
deictic hand gestures and speech. The task constraints then make it
feasible to study the critical components of the multimodal
interpretation problem and to define an architecture for natural
multimodal dialog.