Paul E. Rybski Back to Home Page

Research Interests
I am interested in the scientific challenges involved with solving complex tasks with heterogeneous distributed autonomous systems that can be comprised of both stationary sensors as well as mobile robots. Additionally, I am interested in the research questions involved with the interactions between humans and these autonomous systems, particularly when the human and the robot must actively work together as partners in a task. In order to operate effectively in the real world, such systems must observe, model, and infer the internal state of the world (including other robotic and human agents) around them. To succeed with such inference tasks, I believe that autonomous systems must take an active role in intelligently exploring their environment to explicitly search for the information that will further their goals. For more info see my publications list.
Research at Carnegie Mellon University
In July of 2005, I joined the faculty of The Robotics Institute as a System Scientist.
Autonomous Driving Collaborative Laboratory (AD-CRL) Following on Tartanracing's successful entry into the DARPA Urban Challenge, General Motors created a new Collaborative Research Laboratory which focuses on technologies for Autonomous Driving research. This new lab, called the AD-CRL, has four core research areas including: architectures, behaviors, percep- tion, and planning. I was brought on as the perception area leader. The project in this area that I am currently running is called "Context Sensitive Bicycle and Pedestrian Detection and Tracking." We are interested in methods of fusing sensor data from automotive sensors such as cameras and radar with context from the surrounding environment to allow an autonomous vehicle to more robustly and reliably detect the presence of people and bicycles in its vicinity. This problem is is made challenging by the complexity of general urban environments, the wide variety of possible poses that a person or a cyclist can take with respect to a vehicle, and the need to take the costs of sensing and computational technologies into account when searching for solutions. To see more, visit the project home page.	Detecting and tracking a bicycle.
Tartanracing In 2007, I joined Tartan Racing as the perception lead of CMU's entry into the DARPA Urban Challenge that was held in November of 2007. The Urban Challenge was a contest to promote the research and devel- opment of autonomous cars that operate safely in on-road (urban) settings. The perception challenges to be addressed for this problem included: the detection and tracking of moving obstacles (e.g. other vehicles), the detection and classification of static obstacles, identification of lanes and road edges, and reasoning about sensor fields of view and visibility. In addition to leading the group's efforts, I was actively involved with the development of the robot's moving obstacle fuser, a multi-target/multi-hypothesis probabilistic state es- timation system (using Kalman Filters) which took inputs from the vehicle's wide range of sensor modalities (multiple lasers and radars) to produce a single consistent set of tracked objects. I was also very involved in the development of the robot's road fusion module which estimated the shape of the road in front of the robot using six different laser systems that identified geometric road features such as curbs and obstacles. Tartan Racing took 1st place in the 2007 DARPA Urban Challenge. Some of contributions to this work have been published in the IEEE Transactions on Intelligent Transportation Systems to apper in 2009, Journal of Field Robotics in 2008, the International Conference on Intelligent Robotics and Systems (IROS) in 2008, in the IEEE Intelligent Vehicles Symposium (IVS in 2008.) To see more, visit the Tartanracing home page.	The Tartanracing logo.
CMAssist/Snackbot The CMAssist/Snackbot project seeks to understand how robots can effectively interact with humans and operate alongside people in indoor environments. This effort focusing on developing methods for robots to perceive humans, understand their activities, and observe how they interact with their environment. The CMAssist project was formed in 2006 to develop a robot to compete in a new RoboCup league called RoboCup@Home. This league is a departure from the more traditional RoboCup domains which focus their research in a multi-agent Soccer domain. Instead, RoboCup@Home focuses on socially-relevant human/robot interaction in a home environment. My students and I built two robots, named Erwin and Carmela, that were capable of visually tracking and following people, navigating safely through an environment, under- standing spoken commands, and learning simple tasks by observing and interacting with people. We placed 2nd in the first competition in Breman, Germany, 2006. Another focus of the CMAssist effort is to find ways for humans to impart procedural task knowledge to robots using natural communications mechanisms such as speech and physical demonstration. In 2007, I started collaborating with Sara Kiesler and Jodi Forlizzi in the Human Computer Interaction Institute HCII to extend the CMAssist technologies to other domains such as the development of the Snackbot. The Snackbot is a social delivery robot designed to navigate between offices at CMU and deliver food to subscribers of its services. Our research in interactive task learning from demonstration and dialog appears in the 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI) in 2007, the International Conference on Intelligent Robots and Systems (IROS) in 2007, and in the Journal of Intelligent Service Robotics in 2008. The design and interaction goals of the Snackbot has been published in the 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI) in 2009. To see more, visit the CMAssist page and the Snackbot page.	Erwin and Carmela, the CMAssist robots. The Snackbot
Postdoctoral Research at Carnegie Mellon University
I joined the Robotics Institute at Carnegie Mellon University as a postdoctoral fellow in August of 2003. My postdoctoral advisor was Prof. Manuela Veloso who runs the CORAL lab in the Computer Science Department.
CAMEO (CALO-PA) In order to effectively interact with humans in a natural fashion, an intelligent agent must have a robust physical awareness system with which it can sense humans as they perform tasks in the real world. The Camera Assisted Meeting Event Observer (CAMEO), is an omnidirectional sensory system designed to provide an intelligent computer agent with physical awareness of the real world. As a stationary camera, CAMEO has the advantages of only needing to model and track objects around it rather than having to estimate its own motion as well. However, very few assumptions can be made about the exact structure of the environment, or CAMEO's absolute position within it. I have developed an action recognition algorithm which uses the tracked relative motions of human faces as input. This algorithm makes a number of assumptions about the basic models of human behavior, but has the advantage of being able to operate within many different environments. This technique is now being used to learn specific objects in the environment based on people's activities. Through this functional object recognition algorithm, we are able to model objects associated with the tracked activities, e.g. chairs by identifying the objects where people sit. The CAMEO project is part of a much larger effort called CALO (Cognitive Agent that Learns and Organizes) To see more, visit the CAMEO page.	The CAMEO sensor.
Segway Soccer The SegwayRMP project investigates the coordination of dynamically formed, mixed human-robot teams within the realm of a team task that requires real-time decision making and response. We are working towards the realization of Segway Soccer, which is a game of soccer between two teams consisting of Segway riding humans and Segway RMP-based robots. The SegwayRMP's sensors consist of a servo-mounted color camera and has a pneumatic kicker with an actuated ball manipulation device. I have developed an Extended Kalman Filter SLAM implementation for the Segway which allows it to stay within the boundaries of the field as well was helping to develop the initial software infrastructure for the Segway's perceptual systems. I am interested in extending my work in human activity recognition into this domain to allow humans and robots to cooperatively play soccer. To see more, visit the Segway page.	Segway soccer.
AIBO Soccer We are investigating the complexities of dynamic real-time adversarial environments with an AIBO RoboCup soccer team. In the RoboCup legged league, teams of four AIBOs compete against one another in two ten-minute halves. Because the AIBOs are quadrupedal robots, their motions are extremely difficult to model correctly, particularly when they are being jostled by other players. The low viewing angle and narrow field of view of their cameras make it difficult to get an accurate world view of all objects around them. I am primarily interested in addressing the problems of using communication to create an accurate shared world model in the face of very noisy sensors and actuators. An accurate shared world model is extremely important for effective teamwork as the robots can model the effects of their teammate's actions and plan accordingly. As part of my work with the Robot Soccer group in Fall 2003, I assisted with a course being taught with the Sony AIBO. The course materials from 2003 can be downloaded here. The 2004 materials will be available early in January 2005.	CMPack AIBOs.
Graduate Research at the University of Minnesota
I did my graduate work, both MS and Ph.D, in the Department of Computer Science and Engineering at the University of Minnesota. I worked in the AIRVL - AI, Robotics, and Vision Laboratory for my advisor, Dr. Maria Gini.
Building Topological Maps with Minimalistic Sensor Models I defended my Ph.D dissertation on May 23, 2003. My thesis advisor was Dr. Maria Gini. My committee members were Dr. Nikolaos Papanikolopoulos, Dr. Stergios Roumeliotis, Dr. Richard Voyles, and Dr. Dan Kersten. My dissertation addresses the problem of simultaneous localization and mapping for miniature robots that have extremely poor odometry and sensing capabilities. Existing robotic mapping algorithms generally assume that the robots have good odometric estimates and have sensors that can return the range or bearing to landmarks in the environment. This work focuses on solutions to this problem for robots where the above assumptions do not hold. A novel method is presented for a sensor poor mobile robot to create a topological estimate of its path through an environment by using the notion of a virtual sensor that equates ``place signatures'' with physical locations in space. The method is applicable in the presence of extremely poor odometry and does not require sensors that return spatial (range or bearing) information about the environment. Without sensor updates, the robot's path estimate will degrade due to the odometric errors in its position estimates. When the robot re-visits a location, the geometry of the map can be constrained such that it corrects for the odometric error and better matches the true path. Several maximum likelihood estimators are derived using this virtual sensor methodology. The first estimator uses a physics-inspired mass and spring model to represent the uncertainties in the robot's position and motion. Errors are corrected by relaxing the spring model through numerical simulation to the state of least potential energy. The second method finds the maximum likelihood solution by linearizing a Chi-squared error function. This method has the advantage of explicitly dealing with dependencies between the robot's linear and rotational errors. Finally, the third method employs the iterated form of the Extended Kalman Filter. This method has the advantage of providing a real-time update of the robot's position where the others process all the data at once. Finally, a method is presented for dealing with multiple locations that cannot be disambiguated because their signatures appear to be identical. In order to decide which sensor readings are associated with what positions in space, the robot's sensor readings and motion history are used to calculate a discrete probability distribution over all possible robot positions. Feel free to download and read my PhD Dissertation (10M pdf).	Appearance-based mapping.
Distributed Robotics In 1999, I became involved with a project to develop a distributed team of robots for surveillance and reconnaissance applications. This robotic team consists of two kinds of robots, the scouts and the rangers. The scouts are tiny cylindrical robots 11cm in length and 4cm wide. They can roll along on the floor with two wheels and can hop over obstacles using a leaf-spring foot. They carry sensor payloads (usually a camera) to sense their environments with. The scout is a completely original robot designed (and patented) by researchers at the University of Minnesota, MTS Systems Corp, Architecture Technology Corp. and Honeywell. Scouts are carried and deployed by the larger rangers, the second kind of robot in the team. Rangers are commercially-available off-road platforms from iRobot that we have modified and added our own accessories to. Each ranger carries a custom scout launcher (also designed and built at the University of Minnesota) which it uses to deploy scouts into an environment. I developed an automated resource scheduling system that allowed multiple control processes to access the hardware necessary to control the miniature Scout robots. This system keeps track of current resource allocations and limits and schedules access to resources that are overextended. This resource allocation system was used to manage the robots in the automated placement of a sensor network. We analyzed and modeled the performance of this system from a rigorous experimental standpoint in an attempt to try to answer the question of how many robots would be needed to do this task given a simple description of the world. Finally, I developed the control electronics for a ballistic Scout deployment launcher and a microphone sensor for the Scout robots. For more info, see the Distributed Robotics home page.	Here is a scout shown next to a quarter for scale. Here is a ranger carrying a Scout Launcher.
Minnesota Distributed Autonomous Robot Team (MinDART) From 1997 to the present, our research group has been interested in simple multi-agent foraging techniques. We constructed this group of LEGO-based robots in order to examine the kinds of problems that the environment, intelligence of the robot and distribution of target objects in the environment affect the performance of the group. This team has been used to analyze the kinds of engineering decisions that typically go into creating a distributed robotic system to solve a particular task. We ran more than 100 individual experiments that tested the performance of the robotic team under different environmental conditions, communication strategies, and control strategies. Some of the results from these experiments were that for this kind of simple task, the effects of randomness in the behaviors of the robots typically outweigh any attempts to boost performance due to communication and control strategy. Instead of a statistically-significant global performance boost, we observed that additional knowledge in the system can have a significant effect on the variance of the system. These robots are programmed with the MIT Handyboard. Earlier versions of the robots used simple microswitches, IR detectors and photoresistors to sense their environment, while the current versions make use of the CMUCam and can signal each other via the use of light beacons. For more info, larger pictures, and movies, see the mindart home page.	A closeup photo of a single robot. The group of five robots with their targets. A closeup of the Infrared-transmitting target.
Pioneer Robots Our group owns a group of five ActivMedia Pioneer 1 robots as well as several Pioneer 2ATs. All of the robots Pioneer 1s are equipped with grippers and two of the Pioneer 1s can make use of the Newton Labs Cognachrome Vision System. We have upgraded our vision system to 1 Meg of RAM, which allows us to track more than 3 colors (although only three can be tracked simultaneously). One of our Pioneer 1s has a custom pan/tilt unit for its camera that is run by the gripper's servo controller. We have several SICK laser range finders and can mount these on any of the robots. One of our Pioneer 2s is equipped with a custom panoramic vision system (an omnicamera). In 1999, I did a project with these robots in human/robot interaction and task training. This project focused on exploring the how a human being can interact with a simple mobile robot and teach it how to accomplish a simple mobile manipulation task. A Human Robot Interface (HRI) was developed in order to allow a human to show a robot how to solve a particular task in real time. The robot used a set of Hidden Markov Models (HMMs) to visually classify the actions carried out by the human ``teacher''. Because this task was performed on a mobile robot, it was able to follow the human teacher around the environment and observe how the teacher interacted with objects in that environment. By coupling observation and mobility, the robotic was able to implicitly couple actions with specific locations in the environment. For more info, see my gesture-based programming page.	A Pioneer 1 with SICK laser, USB camera and Pentium III laptop running Linux. Four of our Pioneer 1s equipped the same way as above. A Pioneer 2 AT with a custom omnicamera.
Trailer Backer During the 1996-1997 school year, I built a robotic cart and trailer setup to run a neural network learning system designed by Dean Hougen. One of the most difficult things to do when driving a car with a trailer attached to it is to back it up a driveway or down a boat launch. Therefore, we thought we'd try to implement a learning system capable of doing this very task. The trailer has a tracking turret which keeps track of where a light bulb (the target) is. After about 8 or 9 tries, the robot is usually able to figure out how to hit the target under a few different circumstances. After about 50-60 tries, the robot can back up to the light from any configuration. For more info see the Trailer Backer page.	The 2nd generation Trailer Backer. The 3rd generation Trailer Backer. 3rd generation with 2 trailers.
Loon In the summer of 1996, I worked on a team of people which included Brian Schmalz, Dirk Edmonds, and Margaret Hsieh, to build a multi-robot entry for one of the robotics competitions at the AAAI conference held in Portland, Oregon. Loon was a system which consisted of two robots that had the same configuration of sensors and actuators. However, one was about 1/2 the size of the other one and could ride on the larger one's back. Communication was accomplished through a serial-line cable and the larger robot could deploy the smaller one when needed. For more info see the Loon page.	This is the Loon multi-robot system.
Surfbot In winter 1996, I developed an implementation of an Artificial Potential Field algorithm that ran on a small LEGO robot as my final project for CSCI 5512, Artificial Intelligence II. For more info go to my Surfbot page.	Surfbot -- A robot which navigates by potential fields.
Walleye In the 1995-96 school year, I assisted with the Walleye autonomous vision project. One undergrad student developed the vision code itself and I worked on some control/search algorithms. For more info see the Walleye page.	This is Walleye, built by John Fischer and other students in the AIRVL.
Autonomous LEGO 'bots The first robots that I worked with consisted primarily of LEGO parts with integrated sensor electronics and hobby servos. For control, I used two different 68HC11-based microcontrollers: the Handyboard and the Miniboard. For more info, see the Mini Robots home page.	6-leg walker is a simple LEGO robot designed to demonstrate a very simple dual-tripod gait.

Last updated Feb 20, 2008