My research interests lie in the area of conversational agents and spoken language interfaces. I believe these systems provide an excellent platform for investigating a number of very interesting problems related to spoken language processing, interaction, learning, and, more generally speaking, artificial intelligence. Currently, my work is focused on issues of robustness, error handling and implicit learning in interactive systems. Other interests include dialog management, multi-modal systems, automatic knowledge acquisition, spoken language understanding, etc.

robustness and error handling (dissertation project) [more details]

One of the most important and persistent problems in the development of spoken language interfaces is their lack of robustness when faced with unreliable inputs. My dissertation addresses this issue by (1) improving error awareness, (2) developing and evaluating a rich repertoire of error handling strategies and (3) developing a scalable and adaptive approach for making error handling decisions.

Some of the questions under scrutiny are: how does a system “know that it doesn’t know”? How do we develop systems that can monitor and accurately update their beliefs? What set of strategies can be used to set a conversation back on track, and what are the typical user behaviors in response to these strategies? What techniques can be used to learn the optimal behavior on-line, from detected error segments, and how do we make these systems adapt and improve their performance over time?

implicit learning in spoken language interfaces

Current learning paradigms (i.e. supervised learning) favor an off-line, or “batch” approach to learning that encourages first developing a corpus then training on it. Ideally, interactive systems should be able to learn online, and without explicit human supervision; systems should be able to inspect their past experiences, learn from them and continuously improve their capabilities over time.

I am interested in developing a new approach, implicit learning, that can address current shortcomings by building on a key property of dialog systems, their interactivity. The central idea of the proposed approach is to acquire knowledge online, by leveraging natural patterns that occur in conversation as a by-product of the collaboration between the user and the system. In the context of spoken language interfaces, many such patterns exist (e.g. error detection, error correction and error recovery). I believe that significant autonomous learning can be enabled through the discovery and harnessing of such patterns. For instance, in recent (but yet to be published) work , I have shown that implicit learning with error detection patterns can be used to learn confidence annotation models, to within 75% of the performance achieved using fully supervised learning. A second result, based on error correction patterns, shows that implicit learning supports automatic correction of recognition errors and can form new semantic associations between novel inputs and existing system concepts.

My work focuses on three questions: (1) how can systems make effective use of knowledge extracted from various interaction patterns? (2) how can systems actively trigger these patterns to create learning opportunities but without a significant negative impact on the interaction? and (3) how can systems automatically identify novel knowledge-producing patterns in interaction and thereby increase the range of learning opportunities?

dialog management / RavenClaw [more details]

As infrastructure for my work, I developed RavenClaw, a freely-available dialog management framework for task-oriented domains. RavenClaw enforces a clear separation between the domain-specific and the domain-independent aspects of the dialog control logic. System developers focus exclusively on specifying the domain-specific aspects. In parallel, the RavenClaw Dialog Engine automatically ensures a rich set of conversational behaviors, such as error-handling, timing and turn-taking, in addition to support for universal dialog mechanisms such as repeat, help, suspend, where are we, what can I say, start over, etc.

To date, RavenClaw has already been used to build and successfully deploy a number of spoken dialog systems spanning various domains and interaction types. For instance RoomLine assists students and faculty on campus in making conference room reservations. The Let's Go! Public spoken dialog system provides schedules and route information for the city of Pittsburgh's Port Authority Transit (PAT) buses. More recently, the Conquest system provided technical schedule information during the Interspeech-2006 conference. Furthermore, RavenClaw provides a robust platform for research on various dialog management and spoken language interface issues, such as timing and turn-taking, multi-participant dialog, dynamic dialog task construction, etc.