15-494/694 Cognitive Robotics Lab 8:
Learning Color Classes
I. Software Update, SDK Update, and Initial Setup
Note: You can do this lab/homework assignment either
individually, or in teams of two.
At the beginning of every lab you should update your copy of the
cozmo-tools package. Do this:
$ cd ~/cozmo-tools
$ git pull
II. Unsupervised Learning: RGBClusterer Demo
demo, placing objects of several different colors in front of the
camera. As discussed in lecture, this algorithm uses unsupervised
learning (a Gaussian mixture model) to "discover" color classes. You
may not get a good segmentation right away. Try adjusting the objects
so they are well lit, and make sure that each color class is
well-represented (not too small in the image).
Take a screenshot showing your original image and the color classified image.
III. Supervised Learning: Support Vector Machines
Support Vector Machines (SVMs) learn decision boundaries between
classes by selecting from among the set of training points those
points (vectors) closest to the decision boundary. They therefore
avoid having to store all the training data. See the illustration
Make a lab8 directory and download the
and sample_image.jpg into it.
- Run the demo by typing "python3 -i color_svm.py". The "-i" switch
is necessary to keep Python in "interactive" mode so it doesn't quit
when the main program finishes.
- To learn a "medium green" color class corrsponding to the bottle
cap on the right side of the image, left click on some points on
the bottle cap, and right click on some points of other
- Try to maintain roughly equal numbers of positive and negative
examples of "medium green". The order in which you pick points
doesn't matter because the classifier is retrained from scratch
every time a point is added.
- If your training set gets out of balance, the SVM may set the
decision boundary to something crazy, and all the pixels may be
selected or deselected. Just add some more training points to
bring things into balance, and the model will recover.
- The SVM can also set bad decision boundaries if your data are
not cleanly separable, and the fraction of misclassified points
exceeds some threshold. Again, adding more training points will
cure the problem.
- Modify the demo to include options for saving and reloading the
trained classifier using pickle.
page to learn how to do that.
IV. Train Cozmo to Recognize An Object
Unfortunately the color_svm demo cannot run inside cozmo-tools due to
problems with the tkinter GUI interface, which matplotlib relies on. So
you will have to train your classifier offline. But you can still collect
images from Cozmo using the new SaveImage node in nodes.py.
- Pick a uniformly colored object you want Cozmo to track.
- Get a good picture of the object through Cozmo's camera using
the SaveImage node.
- Using your modified color_svm program, train the classifier.
- Write a state machine program to load the trained classifier,
classify pixels in the latest camera image, and display that
result with matplotlib. You won't be able to make this real-time
interactive because of the tkinter problem; you will have to use
plt.show() and further execution will be blocked until you type
"q" in the plot window. So you can only process one image at a
V. Make Cozmo Find and Track Your Object
Write code so Cozmo looks for for your object if it's not in view, and
drives up to it, maintaining a modest distance. If you gently move
the object, Cozmo should continue to move so as to maintain the
You can test for the presence of the object by counting the number of
pixels that are in your desired color class.
Hand in all the code you wrote above, plus relevant screen shots.