15-494/694 Final Project Ideas

Visual Search

• The goal is for Cozmo to locate an object such as a cube or custom marker.
• An initial strategy is for him to rotate in place until he's swept through 360 degrees. But what if this doesn't find the object?
• Wandering randomly is dangerous because Cozmo cannot detect the edge of the table. He also risks running into obstacles (other than walls) he can't see, which will mess up his odometry.
• The solution is to maintain an occupancy grid representation of obstacles (walls) and safe space. Any space between the robot's current location and a visible wall marker is obviously unobstructed and can be marked as safe. Also, assume any space within 75 mm of a wall is safe.
• Cozmo may not know the initial wall configuration, and he can't see what's on other side of a wall, so he will have to explore by passing through doorways, discovering more walls, and checking each side of each wall. He will also need to look behind cubes in case another cube or marker is lurking there.
• The occupancy grid will also record which parts of the space have been inspected. Uninspected areas (such as the area behind a cube or on the other side of a wall) become candidates for exploration.

The Kidnapped Smart Robot Problem

• The "kidnapped robot problem" is when a robot is picked up and moved to a new place, and must then figure out where it ended up.
• The cozmo-tools particle filter responds to this by randomizing the particles and trying to use landmarks to re-localize. But what if no landmarks are visible in the camera image?
• This project will develop smart strategies for searching for landmarks to help the robot self-localize. This will involve moving both the head and the body.
• The problem is more difficult when the robot is in a tight space because collisions with walls or objects can throw off its odometry.

Cozmo's Magic Dream House

• We have code for navigating the Dream House and operating the elevator. See this video
• Develop new Dream House capabilities such as recognizing portraits on the wall (Cozmo has face recognition) and reporting who is in what room.
• Program Cozmo to lead a visitor to a specified room.
• Program Cozmo to give a tour of the house. Imagine it's some famous house like the Carnegie Mansion and give historical tidbits as part of the tour.

CNN Partial Cube Detector

• Cozmo can only detect a cube if it lies entirely within the camera field of view. If a cube is only partly visible in the camera frame, he can't see it.
• Train a convolutional neural network (CNN) to recognize partial cubes, so that Cozmo can know to make a small turn left or right to bring the cube entirely within the camera frame.
• This could greatly speed up visual search.

CNN Obstacle Detector

• Right now the only obstacles Cozmo can detect are cubes and walls.
• It would be nice to be able to make other types of obstacles, such as soda cans, detectable by the robot, so that we could construct an obstacle course.
• Develop a distinctive pattern (say, diagonal black and white stripes) that can be placed on an obstacle to make it visually distinct.
• Train a CNN to recognize obstacles and write code to add the obstacle to the world map.
• Demonstrate Cozmo navigating through an obstacle course

CNN Gesture Recognition

• Use a pre-trained deep neural network as the hidden layer for a new, rapid feature learner.
• See gesture recognition demo at the Teachable Machine.
• Teach Cozmo to recognize hand gestures using the GPU.
• Links: Teachable Machine source, and news blurb.
• The original version of Teachable Machine used SqueezeNet. The current version appears to use MobileNet.