Calypso Module 1: Perception

Learning Goal: This module will show you how to use Calypso to investigate Cozmo's perception. Cozmo has a camera in his head; he uses computer vision to recognize objects and faces, and he keeps a map of the world in his "mind". You can see this map on your computer screen. Cozmo can also sense when you pick up or tap on one of his light cubes.

If you don't have a robot and are running Calypso in simulator mode, skip ahead to the "Editor Mode and Exection Mode" section.

Running Calypso

Wake up Cozmo by putting him on his charger. If he's already on it, remove him and put him back on it again. You should see his backpack lights pulse on and off, indicating that he has power and is awake.
Run the Cozmo app on the Fire tablet, connect to the robot, and put the app into SDK mode. (To put the app into SDK mode, press the Settings button in the top right corner of the app display, then scroll right until you get to the "SDK Mode" button.)
If Cozmo has driven himself off the charger, put him back on the charger. Once in SDK mode he will stay put until you tell him to move.
Connect the Fire tablet to your computer using the USB cable that came with the tablet.
Plug your Xbox 360 game controller into another USB port on your computer. On Windows you may have to wait a few seconds while the system locates and installs the appropriate device driver.
Run Calypso by clicking on the Calypso icon on your desktop.
Check your Fire for a pop-up message asking whether you want to allow USB debugging. If you see this pop-up, check the box that says "Always allow from this computer" and then press "OK" to proceed. You should only have to do this once per machine.
If Calypso starts successfully, you should see a Chrome browser tab that looks like this:
If Calypso pops up a window telling you that it could not connect to the robot, see the Troubleshooting section of the Calypso Help Center for help.
You should see the words "Novice Mode" in green in the bottom right corner of the screen. If you don't see them, it's a good idea to return to Novice Mode before proceeding further:
1. Mouse click on the Home icon in the top right corner of the screen.
2. Select "Settings..."
3. Click on "Normal Mode" to change to Novice Mode.

The Camera Viewer

Cozmo has a camera in his head, and built-in computer vision that allows him to see the world. The Camera Viewer lets you see what Cozmo sees.
When Calypso starts up, the Camera Viewer should be visible. If it's not, click on the camera icon on the right side of the screen to unhide the Camera Viewer.
You can change the size of the Camera Viewer, or hide it again, by moving your mouse over the Camera Viewer window so that the title bar appears, and then clicking on one of the buttons that appear in the title bar.
You can move the Camera Viewer to a different spot on the screen by moving your mouse over the Camera Viewer window and then clicking and dragging on any blank spot in the title bar.
If you hide the Camera Viewer by clicking on the close button (the "X") in the title bar, and then un-hide it again by clicking on the camera icon, the Camera Viewer remembers the size and location you selected previously.

What Can Cozmo See? Recognizing Light Cubes

Cozmo comes with three light cubes that have distinct markers on their faces. Cube #1 is called the "paperclip" because its markers look like paperclips. Cube #2 is called the "anglepoise lamp"; you can see why here. Some people call cube #3 the "deli slicer" because it sort of looks like this. Others think it looks like a child in a car seat, or like the letters "bT". We'll stick with "deli slicer". Here is a diagram showing all the cube markers:

To see Cozmo recognize a light cube:

Set the Camera Viewer display to the largest size.
Put a cube in front of Cozmo, about six inches away. You may need to tilt his head up or down so he can see it properly. (Just press on his head with your fingers and you can move it.)
When Cozmo recognizes the cube, you will see a yellow box around it in the camera viewer.
Slide the cube to the left or right until it starts to go out of the camera image. How much of the cube does Cozmo need to see in order to recognize it?
Put the cube back in front of Cozmo so he can see it. Put another cube next to the first one, so he can see both of them. Now you should see two boxes in the Camera Viewer, one for each cube.
Move the second cube so that it is slightly behind the first cube. When one object hides another from view, this is called "occlusion". Try sliding the second cube around and watching the result in the Camera Viewer. How much of the second cube has to be visible for Cozmo to still be able to see it?
Cozmo is rather near-sighted. If you move a cube further away from him, at some point he will stop detecting it even though you can still see it in the Camera Viewer. (If necessary, adjust his head angle so that the cube remains visible in the camera image.) If you move the cube closer, Cozmo will be able to see it again. Using a ruler or tape measure, determine how far away the cube can be before Cozmo loses sight of it.

Recognizing Faces

Cozmo can recognize human faces. Turn him so that he's facing you and tilt his head up so his camera can view your face. You should see a green box drawn around your face in the Camera Viewer.

Editor Mode and Execution Mode

The Calypso software you're using has two modes: editor mode and execution mode. Editor mode, which you're in right now, lets you write and edit programs. Execution mode runs the programs. Calypso always starts out in editor mode. Both modes offer the Camera Viewer.

Xbox Game Controller

You can switch between editor mode and execution mode by pressing the Back button on the game controller (see the colored square in the diagram above), or by pressing the backspace key (Delete key on a Mac) on the keyboard. Try it.

Notice that along the left edge of the Calypso window there is "button help" or "keyboard help" to remind you of the available options. The icon for the Back button is a leftward-facing triangle.

In execution mode the left half of the screen displays the program that is running, but you cannot edit the program. You can only observe it. Since we haven't entered a program yet, there is only one empty rule, and not much happens on the left half of the screen when you enter execution mode.

As in editor mode, execution mode allows you to control the size and placement of the Camera Viewer. You can hide the viewer, or move it to a more convenient location for execution, and this will not affect the viewer settings in editor mode. They're kept separate.

The right half of the execution mode screen shows the world map, which we'll now investigate.

Cozmo's World Map

Cozmo maintains a map of the world so he can remember where things are when he's not looking directly at them. The world map is shown on the right half of the screen in execution mode. Remember that you can enter execution mode by pressing the Back button on the controller or the backspace key on the keyboard.

Cozmo is shown on the map as a triangle. His map is constantly updated. Put a cube in front of Cozmo and tilt his head so he can see it. (Gently move the head up or down with your fingers.) When Cozmo sees the cube, a square will appear in the world map display. Turn the cube and you'll see the square rotate.

Put your hand between Cozmo and the cube so he can no longer see it, and notice the change of appearance of the square in the world map: it switches from a thick line to a thin one. If Cozmo can't see a cube, he assumes it is still whever he last saw it.

Slide the cube so it's at a slightly different location, then take your hand away so Cozmo can see it again. Notice that the cube "jumps" in the world map when Cozmo sees it at its new location.

Demo Program: AnnounceCubes

The file AnnounceCubes.calypso contains a demonstration program that has Cozmo say the name of each cube he sees. If you hide a cube from him and then show it to him again, he'll say the name again.

To load the program, simply click on the link above while Calypso is running.

Sending Cube Taps

A cube can feel when you tap on it. Let's try this out:

If you're running the AnnounceCubes program, press the Back button (or the backspace key) to return to editor mode.
Press the Start button (or click on the Home icon in the top right corner of the screen) to bring up the Home menu, and select New. This erases the current program.
Press the Back button again to return to execution mode.
Put a cube where Cozmo can see it.
Tap on the cube with your finger, and you'll see the square in the world map display pulse.
You can also simulate a tap by clicking on the square in the worldmap display with your mouse. What happens when you do this?

Review and Discovery

Do the Calypso Module 1 Review and Discovery activity to reinforce what you've learned.

Next Module

In the next module you'll write your first program for Cozmo, and learn how to drive him around using the game controller or keyboard. This is called teleoperation.

Back to Calypso Curriculum overview.