Gita Sukthankar's Thesis Research

Dissertation: Activity Recognition for Agent Teams (July 2007)

My thesis research is in the area of team behavior recognition: how to analyze the coordinated activity of multiple agents from cues such as spatial relations, temporal patterns, and agent groupings. I am particularly interested in applying team behavior recognition to games such as Dungeons and Dragons and Unreal Tournament; additionally I have done work on behavior recognition for battlefield analysis.

Policy Recognition in Multi-Player Games

This project addresses the problem of recognizing policies given logs of battle scenarios from multi-player table-top games. We explore two promising approaches for policy recognition: (1) a model-based system for combining evidence from observed events using Dempster-Shafer theory, and (2) a data-driven discriminative classifier using support vector machines (SVMs). These techniques were evaluated on logs of real and simulated games played using Open Gaming Foundation d20, the rule system used by many popular tabletop games, including Dungeons and Dragons.

Described in G. Sukthankar and K. Sycara, Policy Recognition for Multi-Player Tactical Scenarios, Proceedings of Sixth International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), May 2007

Recognizing Team Behaviors from Spatio-Temporal Traces

This project was to develop a framework for robustly recognizing physical team behaviors by exploiting spatio-temporal patterns. We demonstrate how complex team relationships that are not easily expressed by region-based heuristics can be modeled from data and domain knowledge in a way that is robust to noise and spatial variation. To represent team behaviors in our domain of MOUT (Military Operations in Urban Terrain) planning, we employ two classes of spatial models: (1) team templates that encode static relationships between team members and external landmarks; and (2) spatially-invariant Hidden Markov Models (HMMs) to represent evolving agent team configurations over time. These two classes of models can be combined to improve recognition accuracy, particularly for behaviors that appear similar in static snapshots. We evaluate our modeling techniques on large urban maps and position traces of two-person human teams performing MOUT behaviors in a customized version of Unreal Tournament (a commercially available first-person shooter game).

Described in G. Sukthankar and K. Sycara, Robust Recognition of Physical Team Behaviors using Spatio-Temporal Models, Proceedings of Fifth International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), May 2006

Movie: sequence of MOUT team behaviors executed in Unreal Tournament

STABR: Simultaneous Team Assignment and Behavior Recognition

Without a priori knowledge of current team assignments, the behavior recognition problem is challenging since behaviors are characterized by the aggregate motion of the entire team and cannot generally be determined by observing the movements of a single agent in isolation. To handle this problem, we introduce a new algorithm, Simultaneous Team Assignment and Behavior Recognition (STABR), that generates behavior annotations from spatio-temporal agent traces. STABR completely annotates agent traces with (1) the correct sequence of low-level actions performed by each agent and (2) an assignment of agents to teams over time. Our algorithm employs a randomized search strategy, RANSAC, to efficiently identify candidate team assignments at selected timesteps; these hypotheses are evaluated using dynamic programming to derive a parsimonious explanation for the entire observed spatio-temporal sequence. The proposed approach is able to perform accurate team behavior recognition without an exhaustive search over the combinatorial space of potential team assignments. Experiments on simulated military maneuvers demonstrate that STABR outperforms spatial clustering, both in assignment and recognition accuracy.

Described in G. Sukthankar and K. Sycara, Simultaneous Team Assignment and Behavior Recognition from Spatio-temporal Agent Traces, Proceedings of Twenty-First National Conference on Artificial Intelligence (AAAI-06), July 2006.

Movie: simple example of movement with dynamic team assignments

Using Motion Capture Input for Behavior Recognition

This project addresses single-person human behavior recognition from motion capture data. Using full body motion capture data acquired from human subjects, our system recognizes the behaviors that a human subject is performing from a set of military maneuvers. Low-level motion classification is performed using a support vector machine (SVM); output from the classifier is used as an input feature for the behavior recognizer. Given the dynamic and highly reactive nature of the domain, our system must handle behavior sequences that are frequently interrupted and often interleaved. To recognize such behavior sequences, we employ dynamic programming in conjunction with a maneuver transition cost function to efficiently select the most parsimonious explanation for the human's actions. We demonstrate that our system can handle deviations by the human subject from the expected set of actions and behaviors. Our approach is well suited for incorporation into synthetic agents that cooperate or compete against human subjects in virtual reality training environments.

Described in G. Sukthankar and K. Sycara, A Cost Minimization Approach to Human Behavior Recognition, Proceedings of Fourth International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), May 2005. Winner of Best Student Paper Award

Movie: movement sequence from human subject performing sneaking behavior

Updated by Gita Sukthankar ( on Jan 10, 2007