15-494/694 Cognitive Robotics Lab 8:
Calypso and ALVINN

I. Calypso for Cozmo

1. Set up Cozmo and connect it to the workstation.

2. Plug an Xbox controller into the workstation.

3. Make sure the workstation's speakers are powered on and the volume is turned up.
4. Run Calypso on the workstation by typing the following in a terminal window: ~/Class/calypso

5. Use the controller to enter the Let Me Drive rule. Note that controller button help appears in the top left corner of the Calypso window. If you switch to using the keyboard, the help switches to keyboard help.

   WHEN gamepad DO move
       

6. Run the program by pressing the "back" button on the Controller or the Backspace key on the keyboard.

7. Use the controller to drive the robot around. Refer to the button help: you can control the head and the lift too.

8. Put a cube where the robot can see it and observe its appearance on the world map.

9. Move or rotate the cube and observe the world map. Try putting the cube on its side or upside-down and see how the world map representation changes.

10. Using the controller, make the robot pick up a cube.

11. Leave execution mode by pressing the Back button on the controller or the Backspace key on the keyboard.

12. To turn off "Novice mode": bring up the Home menu by pressing the Start button on the controller or clicking on the Home icon in the top right corner of the screen. Then select Settings.

13. Add a "Pursue and Consume" idiom to your program using the following rules:

    WHEN gamepad DO move
    WHEN see cube DO move toward it
    WHEN bumped cube DO grab it
        WHEN DO say "I have a cube"	
        

    Note: to indent a rule, put the pencil on the WHEN tile, pick up the rule using the A button, slide it to the right using the left stick, then put it back down.

14. Put one cube behind the robot and hide the other cubes. Run the program.

15. Use the controller to turn the robot until it sees the cube.

II. Calypso Map Layouts

1. You will need to complete this lab on the real robot, but you can do some of your development work at home using Cloud Calypso if you wish.

2. Have a look at the Calypso curriculum.

3. Show the robot the shack you constructed, and using the Let Me Drive idiom, have it explore the shack so the walls appear on the world map.

4. Click on the map menu in the top left corner of the world map and select Snapshot to copy the map to the Map Layout editor.

5. Leave execution mode and switch to the Map Layout view.

6. Clean up the map layout and add two blank walls to complete the shack.

7. Drag a room onto the map layout and adjust it to cover most of the interior of the shack. Rename the room to "interior".

8. Drag two other rooms to locations outside of the shack, and give them names.

9. Write a state machine to get the robot to visit each of your three rooms.

10. Run your program and switch to the state machine view to see the robot change state.

11. Save your program. The map layout will be saved with it.

III. Are You Smarter Than a Fifth Grader?

1. Watch this video of the Cozmo baking companion for inspiration.

2. Write your own Cozmo demo in Calypso. It must satisfy the following requirements:
   • Program Cozmo plus at least one other character (a cube or the "world").

   • Cozmo's program should involve multiple pages of code.

   • Use at least two rooms. You can use the shack if you wish. If you choose not to use your shack, you may want to keep it in the background to provide stable landmarks so the robot can reliably navigate to the rooms.

3. Take some photos and screenshots to document your demo.

IV. Experiment with Classic ALVINN

1. Make a lab8 directory.

2. Download the file data.zip into your lab8 directory and unzip it.

3. Make a lab8/python subdirectory and download the file alvinn1.py into it.

4. Read the alvinn1.py source code.

5. Run the model by typing "python3 -i alvinn1.py". The "-i" switch tells python not to exit after running the program. Type train_alvinn() to train the network.

6. Examine the hidden unit weights by typing show_hiddens(). You can examine an individual hidden unit, e.g., unit 2, by typing show_hidden(2).

7. How balanced is the training set? Generate a histogram plot of desired steering directions.

8. Because these are single-lane roads, we can double the training set size by flipping the input images and also the desired output patterns. Modify alvinn1.py to do that. How does this affect the model?

9. Retrieve the model paramters:
   p = list(model.parameters())

10. What do the parameters look like?
    [param.shape for param in p]

11. What do the output unit bias connections look like?

12. Turn off weight decay; how does this affect the loss? How does it affect the weights?

13. Type test_alvinn() to run the network on a test set of 97 similar road images. How well does it do?

14. Continue the training by typing train_alvinn() again. Have we reached aymptote? How well do we do on the test set now?

15. Write a function test_alvinn2() to test the network on two lane roads, which are also supplied in the ALVINN dataset. Note that you should not flip the two-lane road images.

16. Use the supplied function closest_gaussian to compare the shapes of the gaussians produced for two-lane roads to the ideal gaussians supplied to you in the variable gaussians by plotting one against the other. Make a similar comparison for the output patterns produced on the test set of one-lane roads. This difference from an idealized gaussian is what Pomerleau called Output Appearance Reliability Estimation.

Hand In

Hand in the following in a file called handin.zip:

1. Your modified alvinn1.py file.

2. A brief writeup describing your observations about performance of the network:
   1. What is the mean loss on the expanded training set?
   2. What is the mean loss on the test set?
   3. Compare with the mean loss on two-lane roads.
   4. How does OARE differ between the training set, test set, and two lane roads?

3. Your Calypso program file.

4. Screenshots of your Calypso world map when running your program.