15-494 Cognitive Robotics
Spring 2011
Course Links:  Main Lectures Labs Homeworks

Tekkotsu Links:   Wiki Tutorial Resources Bugs CVS

Cognitive Robotics: Lab 3/Homework 3

Part I: Visual Routines in the Simulator

Download the file ttt6.png to your ~/project/logged directory. (The image at right is an RGB version; click on it to see the YUV.) Since this is a PNG file, Tekkotsu expects it to be in YUV format instead of RGB. YUV format allows more accurate color segmentation, and can be used with the standard threshold file default.tm.

Type in the program from the visual routines lecture notes that finds the orange blob closest to the largest blue blob (examples 2 and 3), but change orange to pink.

What does findMinPlus() do? Where can you find the documentation for it in the Tekkotsu Reference section? (Hint: go to the DualCoding library sub-documentation from the main Reference page, and then click on the Class Members tab, then select "f".)

Although we'll be running Tekkotsu on the workstation instead of on the Chiara, it's the same code, so just type in your .cc.fsm file and type "make" to compile. Then, to run Tekkotsu, type the following on your workstation:
cd ~/project
./tekkotsu-CHIARA -c sim.plist
Start up a ControllerGUI, but instead of giving a robot name as the argument, use "localhost" to connect to the simulator. Then run your behavior, and click on "C" in the ControllerGUI to launch the camera space SketchGUI. Click on items in the SketchGUI derivation tree to see the corresponding sketch. Use control-click to view several items at once.

Part II: Solving Problems with Sketch Primitives

For each problem, write code using only sketch primitives (not shape primitives) to solve it. You may want to review the documentation for the Sketch primitives and the visops namespace. Note: the images below appear dark because the camera gain was set low. This reduces specular reflection and makes the images easier to segment.

Note: when you have multiple image files in your project/logged directory, the simulator will cycle through them at 30 frames per second. To prevent this, use the "freeze" command. Then, when you want to advance to the next image, use the "advance" command. Stop your behavior and start it again to pick up the new image.

Your answers to Parts II and III should be turned in as Homework 3, due Friday, Feb. 4.


Figure out whether there are more blue blobs or more pink blobs in each image, and display the blobs of the more numerous type. To "display" your result, construct a Sketch<bool> called result that contains only the desired blobs.


Which pink blob appears inside the closed curve in the images above? Note: you will want to use visops::fillInterior to find the interior of the curve.

Judging Distance

Solve this problem on the robot instead of the simulator. Which of two easter eggs is closer? Choose two easter eggs of different colors and place them in front of the Chiara in one of three arrangments: pink one closer, blue one closer, or both roughly the same. The robot should say which egg is closer (e.g., "blue is closer"), or "same distance" if neither is closer. Do not use blob area to judge distance; that won't work if the easter eggs are of different sizes. Instead, use the y-coordinate of the bottom edge of each blob. To calculate this, you'll need to take the logical AND of each pixel and the logical negation of its southern neighbor; you can index the sketch using idxS to get the southern neighbors.

Free Style

Make up an interesting geometric question of your own about any of the above images, and show how to solve it using visual routines.

Part III. Vision on the Robot

From now on, whenever you first boot a robot and start to use it, you should begin by running the "sendtekkotsu" script to load the latest copy of the Tekkotsu libraries. You only have to do this at the start of your session. Once you've updated your robot for the day, you can use the "sendmycode" script to upload your code as often as you wish.

Shell Game

Set up a Shell Game for the Chiara to play. Using 3 easter egg halves, hide a pice of tape under one of the eggs. The Chiara should point the arm at one of the three eggs (left, middle, or right). Then, when you remove the egg, the Chiara should make a happy noise if it wins (sees the tape) or a sad noise if it loses (no tape). Assume there are other random pieces of tape lying around; your robot should only care about a piece of tape that is (or is not) present under the easter egg it selects. To pick an egg at random, use a RandomTrans transition. To point the arm at a specific place, use an ArmNode to set the shoulder, elbow, and wrist joint angles directly. (We'll learn more sophisticated ways to move the arm later, when we cover kinematics.) Your behavior can assume that the eggs will be in fixed positions corresponding to the left, middle, and right thirds of the camera image, so it doesn't need to search for them. To select one third of the camera image, you'll want to make a Sketch<bool> to use as a mask. You can use a for loop to set the bits you want.

Dave Touretzky and Ethan Tira-Thompson