15-494/694 Cognitive Robotics Lab 8:
Calypso and ALVINN
I. Calypso for Cozmo
- Set up Cozmo and connect it to the workstation.
- Plug an Xbox controller into the workstation.
- Make sure the workstation's speakers are powered on and the volume is turned up.
- Run Calypso on the workstation by typing the following in a terminal window:
- Use the controller to enter the Let Me Drive rule. Note that
controller button help appears in the top left corner of the
Calypso window. If you switch to using the keyboard, the help
switches to keyboard help.
WHEN gamepad DO move
- Run the program by pressing the "back" button on the Controller
or the Backspace key on the keyboard.
- Use the controller to drive the robot around. Refer to the
button help: you can control the head and the lift too.
- Put a cube where the robot can see it and observe its appearance on the world map.
- Move or rotate the cube and observe the world map. Try putting
the cube on its side or upside-down and see how the world map
- Using the controller, make the robot pick up a cube.
- Leave execution mode by pressing the Back button on the controller or
the Backspace key on the keyboard.
- To turn off "Novice mode": bring up the Home menu by pressing
the Start button on the controller or clicking on the Home icon in
the top right corner of the screen. Then select Settings.
- Add a "Pursue and Consume" idiom to your program using the following rules:
Note: to indent a rule, put the pencil on the WHEN tile, pick up the rule using the A button,
slide it to the right using the left stick, then put it back down.
WHEN gamepad DO move
WHEN see cube DO move toward it
WHEN bumped cube DO grab it
WHEN DO say "I have a cube"
- Put one cube behind the robot and hide the other cubes. Run the program.
- Use the controller to turn the robot until it sees the cube.
II. Calypso Map Layouts
- You will need to complete this lab on the real robot, but you
can do some of your development work at home
using Cloud Calypso if you wish.
- Have a look at the Calypso curriculum.
- Show the robot the shack you constructed, and using the Let Me Drive idiom, have it explore the shack so
the walls appear on the world map.
- Click on the map menu in the top left corner of the world map and select Snapshot to copy the map to the Map Layout editor.
- Leave execution mode and switch to the Map Layout view.
- Clean up the map layout and add two blank walls to complete the shack.
- Drag a room onto the map layout and adjust it to cover most of
the interior of the shack. Rename the room to "interior".
- Drag two other rooms to locations outside of the shack, and give them names.
- Write a state machine to get the robot to visit each of your three rooms.
- Run your program and switch to the state machine view to see the robot change state.
- Save your program. The map layout will be saved with it.
III. Are You Smarter Than a Fifth Grader?
- Watch this video of the Cozmo baking companion for inspiration.
- Write your own Cozmo demo in Calypso. It must satisfy the following requirements:
- Program Cozmo plus at least one other character (a cube or the "world").
- Cozmo's program should involve multiple pages of code.
- Use at least two rooms. You can use the shack if you wish. If you choose not to use your shack,
you may want to keep it in the background to provide stable landmarks so the robot can reliably
navigate to the rooms.
- Take some photos and screenshots to document your demo.
IV. Experiment with Classic ALVINN
- Make a lab8 directory.
- Download the file data.zip into your lab8 directory and unzip it.
- Make a lab8/python subdirectory and download the
file alvinn1.py into it.
- Read the alvinn1.py source code.
- Run the model by typing "python3 -i alvinn1.py". The "-i" switch tells python not to exit
after running the program. Type
train_alvinn() to train the network.
- Examine the hidden unit weights by
show_hiddens(). You can examine an individual
hidden unit, e.g., unit 2, by typing
- How balanced is the training set? Generate a histogram plot of desired steering directions.
- Because these are single-lane roads, we can double the training
set size by flipping the input images and also the desired output
patterns. Modify alvinn1.py to do that. How does this affect the
- Retrieve the model paramters:
p = list(model.parameters())
- What do the parameters look like?
[param.shape for param in p]
- What do the output unit bias connections look like?
- Turn off weight decay; how does this affect the loss? How does
it affect the weights?
test_alvinn() to run the network on a test set of 97 similar road images. How
well does it do?
- Continue the training by typing
train_alvinn() again. Have we reached aymptote? How
well do we do on the test set now?
- Write a function test_alvinn2() to test the network on two lane
roads, which are also supplied in the ALVINN dataset. Note that you
should not flip the two-lane road images.
- Use the supplied function
closest_gaussian to compare the shapes of the
gaussians produced for two-lane roads to the ideal gaussians supplied to you in the variable
gaussians by plotting one against the other. Make a
similar comparison for the output patterns produced on the test
set of one-lane roads. This difference from an idealized gaussian
is what Pomerleau called Output Appearance Reliability
Hand in the following in a file called handin.zip:
- Your modified alvinn1.py file.
- A brief writeup describing your observations about performance of the network:
- What is the mean loss on the expanded training set?
- What is the mean loss on the test set?
- Compare with the mean loss on two-lane roads.
- How does OARE differ between the training set, test set, and two lane roads?
- Your Calypso program file.
- Screenshots of your Calypso world map when running your program.