CMAssist


	CMAssist : Human/Robot Interaction

About
We are interested in understanding how robots can best work around, interact with, and assist people in natural human environments. Our specific research goals include: Recognition and detection of human presence and activities. Using detected human activities to help the robot learn more about its environment. Actively learning tasks by engaging nearby humans as instructors and using spoken language dialog to facilitate the learning process. We have also participated in the annual RoboCup@Home human/robot interaction contest held as part of the international RoboCup competition.

News
July 20, 2006:	CMAssist received a technical innovation award at the 2007 AAAI Mobile Robot Competition and Exhibition. The award was for Robust Hardware and Software Design.
June 18, 2006:	CMAssist placed 2nd in the finals at the first ever RoboCup @ Home competition. Five teams made it to the finals from the technical round. We ranked 1st place overall in the technical round out of eleven teams participating in total.

People
Current Contributors Dr. Paul E. Rybski, Systems Scientist, The Robotics Institute (team leader) Prof. Manuela Veloso, Professor, Computer Science Department Prof. Jodi Forlizzi, Assistant Professor, Human-Computer Interaction Institute and School of Design Prof. Ric Crabbe, Visiting Professor, The Robotics Institute, Associate Professor, United States Naval Academy. Past Contributors Chris Broglie, Undergraduate, Computer Science Department Se Yeong Byeon, Undergraduate, Computer Science Department Jonas P. Cleveland, Undergraduate, Electrical and Computer Engineering Department Alex Grubb, Undergraduate, Computer Science Department Nick Pozek, Masters, Creative Enterprise Management Yongjun Jeon, Undergraduate, Computer Science Department Aubrey Shick, Research Associate, Human Computer Interaction Institute Scott A. Smith, Undergraduate, Human Computer Interaction Institute Stephen Spencer, Undergraduate, Humanities and Arts (Industrial Design and Psychology) Jeremy Stolarz, Masters, The Robotics Institute Kevin Yoon, Masters, The Robotics Institute

Collaborators and Sponsors
Naval Research Laboratory

Publications
P. E. Rybski, J. Stolarz, K. Yoon, M. Veloso, "Using Dialog and Human Observations to Dictate Tasks to a Learning Robot Assistant," in Journal of Intelligent Service Robots (accepted in 2007, to appear in 2008). K. Yoon, P. E. Rybski, "Teaching procedural flow through dialog and demonstration," in Proceedings of IROS'07, the 2007 International Conference on Intelligent Robots and Systems, pp. 807-814, San Diego, CA, Oct-Nov, 2007. K. Yoon, P. E. Rybski, "Teaching Task Flow Through Dialog and Observation," Masters Project, Technical Report TR-07-18, The Robotics institute, Carnegie Mellon University, Pittsburgh, PA, May 2007. J. Stolarz, P. E. Rybski, "An Architecture for the Rapid Development of Robot Behaviors," Masters Project, Technical Report TR-07-16, The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, May 2007. P. E. Rybski, K. Yoon, J. Stolarz, M. Veloso, "Interactive Robot Task Training through Dialog and Demonstration," in Proceedings of the 2007 Conference on Human-Robot Interaction, Washington DC, March, 2007. P. E. Rybski, K. Yoon, J. Stolarz, M. Veloso, "CMAssist'06: A RoboCup@Home Team," Technical Report TR-06-47, The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, October, 2006. P E. Rybski, K. Yoon, J. Stolarz, M. Veloso, CMAssist: A RoboCup@Home Team, In Proceedings of the AAAI 2006 Mobile Robot Competition and Exhibition Workshop, National Conference on Artificial Intelligence, Boston, MA, USA, July, 2006.

Robots
Construction Our platform was based originally on an Evolution ER1 mobile robot. We replaced the stepper motors with DC motors and our own motor controllers. We also replaced all of the plastic connecting pieces from the original ER1 kit with aluminum parts from 80/20. Additional x-bar aluminum for the robot's skeletons as purchased through 80/20.

Sensors Mounted on top of the robot is a custom-built high-resolution omni-directional vision system called CAMEO that consists of 4 separate firewire cameras in a daisy-chain configuration. The CAMEO is used for person detection and tracking as well as for object recognition and robot localization. Lower down on the sensor mast is a stereo camera rig from Videre Design. The stereo camera is used for obstacle detection and person tracking. Around the base of the robot are close-range IR proximity sensors. These are used solely for obstacle detection. One of our robots has a Hokuyo URG-04LX laser that is used for very accurate localization and map construction. The sensor layout of the robot is shown below:

Computation Computation is provided by two laptops mounted on top of the robot's base. Additional computation can be provided from external computers connecting to the robot over wireless Ethernet.

Speech Recognition and Synthesis Speech recognition is accomplished through the use of IBM ViaVoice and the Naval Research Lab's NAUTILUS natural language processing system. A text-to-speech system called Cepstral gives our robots the ability to speak back to the humans they are operating with.

Software Due to the heavy computational demands of speech and vision processing, the software architecture operates in a distributed fashion where sensors and actuators run as separate processes (potentially on different computers). These processes essentially are like network ``services'' which accept TCP/IP connections from clients that need to communicate with them. Behaviors, like the sensors and actuators, also run as a separate process. In order to provide a single consistent interface to the myriad of sensor and actuator services, a single abstraction, called the Featureset, is used. The Featureset is a multi-threaded container module that provides a single unified interface to all sensors and actuators for any behavior that may need to use them. As new hardware services are added to the robot, the Featureset is expanded with new interfaces. A block diagram of our software architecture is shown below:

Photos
CMAssist'06 -- 2nd Place Finalists in RobotCup'06 Here are our robots: Carmela (in red) and Erwin (in blue). This is the configuration of the robots after they returned from RoboCup 2006. Modifications made on-site included mounting the robot's speakers as well as providing a permanent place for the joysticks. We placed 2nd in the finals and 1st in the preliminary technical round in the 2006 RoboCup@Home league.

Robot Design Before Leaving for Robocup'06 Carmela before we packed her up to ship to RoboCup 2006 in Bremen, Germany. Erwin traveled there as well but was not used directly for the competition but rather served as a backup in case of equipment failure on Carmela.

Robot Design Revision 2 Here is the robot prototype from early May 2006. We enlarged the bottom area to house more electronics and added the emergency kill switch.

Robot Design Original Here is the original robot design as developed in January 2006.

Video Demonstrations
Tracking a Person's Contour -- 09/15/2007 A short demonstration of an algorithm that reduces an image to a set of contours and tracks a set of the contours as they move around the image. Note how in this video the tracked person rotates around in a complete circle as they move.

Patrol the Lab -- 05/31/2007 These videos show some more advanced task training features. These demonstrate verbal constructs such as while loops, contingencies (interrupts), as well as conditionals. The task training architecture supports on-the-fly recursive training of subtasks so that subtasks do not all have to be trained before the main task. Additionally, assistive phrases are allowed to let the robot student remind the teacher where he is at any point of the training process.

Multi-Sensor Person Tracking -- 05/31/2007 A video showing how the CMAssist robots can use video (clothing color) and laser to track multiple people. In this work, a multi-hypothesis, multiple target particle filter tracking model is used to allow the robot to successfully track a single person even though there may be more than one person that "looks" identical. If the robot does become lost, the human can speak that information to the robot. The robot then communicates back to the human and resets its tracking model. This video shows a picture-in-picture view of both the video of the robots and people but also a larger image of the particle filter tracking model. In the model, the robot is represented in green at the center of the screen and as it moves around, the positions of the tracked objects scroll around it. Lines extending from the robot to a tracked objects specifies which ones the robot believes are actually people. The other objects are hypotheses that may be people but the robot needs more information to verify them.

Giving the Lab Tour -- 03/20/2007 A demonstration of the task-training capabilities of CMAssist. In the task training video, Carmela is taught to do a task by following the human around the room and listening to his speech. The task is to give a tour of the lab to someone. Carmela queries the human for additional information regarding the task when he fails to specify what to do if someone is not present in the room. In this sequence, a subtask is trained first and then is referred to in a second, "higher-level" task. Two task execution videos are also shown. The first demonstrates a successful execution of the task and the second demonstrates the fault tolerant performance of the robot when the people it is supposed to locate are not present as well as if it cannot reach its intended goal. (Note that the robot is not programmed to plan a path around the obstacle in this instance as this video was meant to show the execution of the task training algorithm only.)

Getting the Family for Dinner -- 2/28/2007 A demonstration of the task-training capabilities of CMAssist. In the task training video, Carmela is taught to do a task by following the human around the room and listening to his speech. Carmela queries the human for additional information regarding the task when he fails to specify what to do if someone is not present in the room. Two task execution videos are also shown. The first demonstrates a successful execution of the task and the second demonstrates the fault tolerant performance of the robot when the people it is supposed to locate are not present as well as if it cannot reach its intended goal.

CMAssist'06 Highlights from RoboCup -- 09/01/2006 Here is a highlight video from RoboCup 2006 in Bremen, Germany. This shows the robot doing the follow person, navigate, and open challenge tasks.