We have developed a gesture recognition system, based on Hidden Markov Models (HMMs), which can interactively recognize gestures and perform online learning of new gestures. In addition, it is able to update its model of a gesture iteratively with each example it recognizes. This system has demonstrated reliable recognition of 14 different gestures after only one or two examples of each. The system is currently interfaced to a Cyberglove for use in recognition of gestures from the sign language alphabet. The system is being implemented as part of an interactive interface for robot teleoperation and programming by example.
Figure 1: Final position of hand for gesture "C".
The motivations behind this kind of interaction include increasing the effectiveness of teleoperation, enabling people to interactively teach robots new tasks or refine their skills, and allowing people to more effectively control systems such as semi-autonomous airplanes or systems for automated monitoring of industrial plants.
As people interact with machines which are autonomous and highly complex, they must be allowed to focus their attention on the content of their interaction rather than the mechanisms and protocol through which the interaction occurs. This is best accomplished by making the style of interaction more closely resemble that to which they are most accustomed: interaction with other people. We believe that online recognition of gestures and online training for gesture recognition are important components of this kind of interaction.
Our goal is to make a system which can not only interact with a user by accurately recognizing gestures, but which can learn new gestures and update its understanding of gestures it already knows in an online, interactive manner. Our approach is automated generation and iterative training of a set of Hidden Markov models which represent human gestures. Using this approach, we have built and tested a system which recognizes letters from the sign language alphabet using a Virtual Technologies `Cyberglove.'
Figure 2: Interface for using gesture recognition demonstration system.
In our current system, each kind of gesture is represented by an HMM, a list of example observation sequences, and an optional action to be performed upon recognition of the gesture. Our concept of interactive training is currently based on the following general procedure:
We have found that in our implementation, recognition of the gesture and automatic update of the HMM through the Baum-Welch algorithm is fast enough not to be noticeable during normal use of the system. This provides the system with a truly interactive character.