Supporting Remote Collaboration

A collaborative physical task is the task in which two or more people interact with real objects in the 3D world. Prior studies of physical collaboration suggest that people’s speech and actions in this context are inherently multimodal, intricately related to the position and dynamics of objects, other people, and ongoing activities in the environment. In particular, communication during physical tasks combines both speech and gesture. The objective of this research is to develop technologies to support communication through speech and gesture during collaborative physical tasks. In the current work, we aim to develop an inexpensive multimodal system that can be easily incorporated into existing video conferencing systems. Our goal is to allow remote collaborators to communicate about their physical world through speech and gesture with the same ease as they can do so when co-located. As a good example, we have developed DOVE (Drawing Over Video Environment),  a system to support multimodal/multimedia communication during collaborative physical tasks. We are working on evaluation and improving the system.

Related Publications:

• J. Ou, L. Oh, S. Fussell, T. Blum, J. Yang, Predicting Visual Focus of Attention from Intention in Remote Collaborative Tasks, IEEE Transaction on Multimedia , Vol.10, No.6, pp.1034-1045, 2008. .
• J. Wong, L. Oh, J. Ou, C. Ros? J. Yang, and S. Fussell, Sharing a Single Expert among Multiple Partners, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 261 ?270, 2007.

• J. Ou, Y. Shi, J. Wong, S. Fussell, J. Yang, Combining Audio and Video to Predict Helpers' Focus of Attention in Multiparty Remote Collaboration on Physical Tasks, Proceeding of the 8th international conference on Multimodal interfaces (ICMI),pp. 217-224, 2006.
• W. Yang, J. Ou, Y. Rui, J. Yang, WEBDOVE: A Web-Based Collaboration System for Physical Tasks, Proceedings of IEEE International Conference on Multimedia & Expo 2006 (ICME 2006), pp. 1137-1140, 2006.
• J. Ou, L. M. Oh, S. R. Fussell, T. Blum, J. Yang, Analyzing and predicting focus of attention in remote collaborative tasks, Proceedings of the 7th international conference on Multimodal interfaces, pp. 116 - 123, 2005.
• J. Ou, L. M. Oh, J. Yang, and S. R. Fussell, Effects of Task Properties, Partner Actions, and Message Content on Eye Gaze Patterns in a Collaborative Task, Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 231 - 240, 2005.
• S. Fussell, L. Setlock, J. Yang, J. Ou, E. Parker, Gestures over Video Streams to Support Remote Collaboration on Physical Tasks. Human-Computer Interaction, Volume 19, pp. 273-309, 2004.
• J. Ou, S. Fussell, X. Chen, J. Yang, Gestural Communication over Video Stream: Supporting Multimodal Interaction for Remote Collaborative Physical Tasks, Proceedings of the 5th International Conference on Multimodal Interface, pp. 242 - 249, 2003.
• J. Ou, X. Chen, S. Fussell, J. Yang, DOVE: Drawing over Video Environment, Proceedings of the ACM Multimedia Conference, pp. 100 - 101, 2003.
• S. Fussell, L. Setlock, E. Parker, J. Yang, Assessing the value of a cursor pointing device for remote collaboration on physical tasks, CHI '03 extended abstracts on Human factors in computer systems, pp. 788 - 789, 2003.