Smart Sight

==========

At the Interactive Systems Laboratory of Carnegie Mellon University, we are developing systems that can help tourists to retrieve information, navigate unfamiliar environment, and overcome language barriers. The systems are equipped with a unique combination of sensors and software. The hardware includes computers, GPS receivers, lapel microphones and earphones, video cameras and head-mounted displays. This combination enables a multimodal interface to take advantage of speech and gesture inputs to provide assistance for tourists. The software supports natural language processing, speech recognition, machine translation, handwriting recognition and multimodal fusion. A vision module is trained to locate and read written language, is able to adapt to new environments, and is able to interpret intentions offered by the user, such as a spoken clarification or pointing gesture.

Related Publications:

  • J. Yang, J. Gao, Y. Zhang, X. Chen, and A. Waibel, "An automatic Sign Recognition and Translation System," Workshop on Perceptive User Interfaces (PUI 2001), Orlando, FL, USA.
  • J. Gao and J. Yang, "An Adaptive Algorithm for Text Detection from Natural Scenes," Proceedings of Computer Vision and Pattern Recognition (CVPR 2001).
  • J. Yang, J. Gao, Y. Zhang, and A. Waibel, "Toward Automatic Sign Translation," Proceedings of the Human Language Technology Conference (HLT-2001), San Diego, March 2001.
  • X. Zhu, J. Yang, A. Waibel, "Segmenting Hands of Arbitrary Color," Proceedings of Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG'2000).
  • J. Yang, W. Yang, M. Denecke, A. Waibel, "Smart Sight: a Tourist Assistant System," ISWC'99 .
  • J. Yang, W. Holtz, W. Yang, and M.T. Vo, "An adaptive multimodal interface for wireless applications," Proceedings of International Symposium on Wearable Computers , Pittsburgh, PA, Oct. 19-20, 1998.

    Back to the previous page

    Last updated Sept. 24, 2001

    ----------