15-494/694 Cognitive Robotics Lab 8:
Learning Color Classes

I. Software Update, SDK Update, and Initial Setup

Note: You can do this lab/homework assignment either individually, or in teams of two.

At the beginning of every lab you should update your copy of the cozmo-tools package. Do this:

$ cd ~/cozmo-tools
$ git pull

II. Unsupervised Learning: RGBClusterer Demo

Run the RGBClusterer.fsm demo, placing objects of several different colors in front of the camera. As discussed in lecture, this algorithm uses unsupervised learning (a Gaussian mixture model) to "discover" color classes. You may not get a good segmentation right away. Try adjusting the objects so they are well lit, and make sure that each color class is well-represented (not too small in the image).

Take a screenshot showing your original image and the color classified image.

III. Supervised Learning: Support Vector Machines

Support Vector Machines (SVMs) learn decision boundaries between classes by selecting from among the set of training points those points (vectors) closest to the decision boundary. They therefore avoid having to store all the training data. See the illustration here.

  1. Make a lab8 directory and download the files color_svm.py and sample_image.jpg into it.

  2. Run the demo by typing "python3 -i color_svm.py". The "-i" switch is necessary to keep Python in "interactive" mode so it doesn't quit when the main program finishes.

  3. To learn a "medium green" color class corrsponding to the bottle cap on the right side of the image, left click on some points on the bottle cap, and right click on some points of other colors.

  4. Try to maintain roughly equal numbers of positive and negative examples of "medium green". The order in which you pick points doesn't matter because the classifier is retrained from scratch every time a point is added.

  5. If your training set gets out of balance, the SVM may set the decision boundary to something crazy, and all the pixels may be selected or deselected. Just add some more training points to bring things into balance, and the model will recover.
  6. The SVM can also set bad decision boundaries if your data are not cleanly separable, and the fraction of misclassified points exceeds some threshold. Again, adding more training points will cure the problem.

  7. Modify the demo to include options for saving and reloading the trained classifier using pickle. Read this page to learn how to do that.

IV. Train Cozmo to Recognize An Object

Unfortunately the color_svm demo cannot run inside cozmo-tools due to problems with the tkinter GUI interface, which matplotlib relies on. So you will have to train your classifier offline. But you can still collect images from Cozmo using the new SaveImage node in nodes.py.

  1. Pick a uniformly colored object you want Cozmo to track.

  2. Get a good picture of the object through Cozmo's camera using the SaveImage node.

  3. Using your modified color_svm program, train the classifier.

  4. Write a state machine program to load the trained classifier, classify pixels in the latest camera image, and display that result with matplotlib. You won't be able to make this real-time interactive because of the tkinter problem; you will have to use plt.show() and further execution will be blocked until you type "q" in the plot window. So you can only process one image at a time.

V. Make Cozmo Find and Track Your Object

Write code so Cozmo looks for for your object if it's not in view, and drives up to it, maintaining a modest distance. If you gently move the object, Cozmo should continue to move so as to maintain the desired distance.

You can test for the presence of the object by counting the number of pixels that are in your desired color class.

Hand In

Hand in all the code you wrote above, plus relevant screen shots.

Dave Touretzky
Last modified: Wed Mar 8 01:47:37 EST 2017