Current project

Iwaki interaction manager [GitHub]

Dialogue and interaction manager that enables soft real-time execution, multi-modal and multi-party interaction. It is used in the new version of Hala and in the Scrabble-playing gamebot Victor.

Collaborator: Reid Simmons

 

Past projects

Tank, the Roboceptionist; Hala, the Qatar robot receptionist; Victor, a Scrabble-playing gamebot

Dialogue management that supports non-verbal interactions, user adaptive dialogue, discourse modeling [papers].

Collaborators: Reid Simmons, Brett Browning, Majd Sakr.

Cross-cultural multimodal corpus of receptionist encounters

We collected and annotated videos of dyadic role-plays in an actual receptionist environment. The annotations are available online at GitHub. There is a paper that introduces the corpus.

Collaborators: Reid Simmons and Majd Sakr.

Glowing sleeve

I played with the idea of wearable lights that respond to body movement. Check out the project page and video.

Culture-specific dialogue management

In the summer of 2010 I worked at Alelo on a dialogue manager that takes into account such aspects of Afghan culture as roles of multi-stage greetings, fairness, and homophily in establishing levels of commitment during negotiations.

I had a pleasure to work closely with Alicia Sagae, Suzanne Wertheim, Mike Agar, Jerry Hobbs and Lewis W. Johnson.

Learning for spatial scan statistics

Spatial scan statistics is a popular method for anomalous spatio-temporal pattern detection, normally does not allow for learning from existing data. We incorporated learning of shape of earlier anomalous patterns into the spatial scan framework [paper].

Collaborators: Daniel Neill and Auton Lab

Extending spatial scan statistics to computer network graphs

Spatial scan statistics is a popular method for anomalous spatio-temporal pattern detection, applicable for spaces with a distance metric. We extended the spatial scan framework to graph distances and applied it to network worm detection.

Collaborators: Denver Dash (Intel Research, Pittsburgh).

Analysis of collaborative meeting instant-messaging transcripts

Analyzed predictive power of various syntactic and discourse-level features for dialogue act prediction.

Collaborators: Nik Nailah Binti Abdullah, Shinichi Honiden (NII, Japan).

BlindAid: indoor navigation assistant for the blind [project site]

Developed and tested a PDA-based indoor navigation device that localized in an RFID-instrumented environment and communicated with users in spoken language [report].

Collaborators: Sandra Mau, Nik Melchior, Aaron Steinfeld (a V-Unit project).

Tutoring scientific explanations via natural language dialogue [project site]

Developed and implemented deep natural language understanding techniques for student utterance and essay evaluation. A production system generates a logical closure of the correct and buggy domain facts. The closure graph provides a structure of a Bayesian network and its parameters are learned from a manually annotated dialogue corpus. An incoming student's utterance is mapped into the nodes of the Bayesian network that serve as an evidence for student's knowledge of the domain facts corresponding to the rest of the network's nodes. This estimate of the student's knowledge state is used to further guide the tutorial dialogue [paper].

Collaborators: Pamela Jordan, Kurt VanLehn, Umarani Pappuswamy.

Why2000, Why2-Atlas intelligent tutoring systems using natural language dialogue [project site]

Developed and implemented deep natural language understanding techniques for student utterance and essay evaluation. These include (a) performing a weighted abductive reasoning on-the-fly and (b) generating a logical closure of the domain facts offline and using it for an ATMS-style inference on-the-fly. An estimate of the student's knowledge state obtained via one of these methods is used to further guide the tutorial dialogue [papers].

Collaborators: Pamela Jordan, Kurt VanLehn, Umarani Pappuswamy, Michael Böttner.

 

Way back projects

YesterdaySushi, a reinforcement learning-based Texas Hold'em pocker player for ICCM'04 PokerBot tournament [ICCM'04]

RoboML, a markup language for human-robot interaction software agents [project site][paper]

Control and navigation for an autonomous guided vehicle

[papers]

Collaborators: Sherman Y.T. Lang, S.K. Tso, John J. McPhee (City University of Hong Kong)

Machine vision for a six-legged walking robot

Collaborators: Institute of Mechanics, Moscow State University