Will it roll ?
Learning environment property through action

Francois Chu

The idea for this project is to study what a robot can learn by itself when it only relies on its own senses and motor system. What kind of property can it learn, if it is limited by a black and white camera and a single arm ? Would it develop the same approach of spatiality if experienced through the sense of touch or through a stereoscopic vision system. In this project, we focused specifically on the property of rollability. We want to study if the robot can differentiate a rollable object from a non rollable one, not only by watching at them but also by interacting with them, since the rollability property is intrinsically a movement property than cannot be observed on still images (human can do it because they have several years experience with cubes and balls!).

This page present my project results for 15-494 Cognitive Robotics, taught by Dave Touretzky, by giving a brief description of the model I wanted to implement, the actual implementation, and some results and additional ideas.

Theoretical model

This problem can be expressed with the following model: we want to describe how a robot can attribute a property to what he describes as an object by interacting with the environment.

If the robot can use a video camera, he can then understand some static property:

With a black and white camera and an arm, he can understand the notion of compressibility, but cannot anymore grasps the idea of color.

In this project, we want to study a robot that learns the property of rollability, as a proof of concept of the ability of a robot acquiring knowledge by itself.


The general architecture of the main behavior reproduces the theoretical structure presented above. The states in the following sequence alternate between the input, the output and the process of information. The behavior can be divided in 2 main components : The Decider class, depending on the results of its algorithms, throw different SignalEvents. For instance, the loop indicated in red is activated just after an action is performed on the object, and captures the movement by computing the blob size.


Here we can see the (slightly) difference of reaction between a rollable and a non rollable object:

Non Rollable

The following video show a run of the Behavior against a rollable and a non rollable object, each focused on the robot or on the Sketch camera:


Although the classification is usually correct in good environment conditions, the accuracy drops significantly when these conditions are not met. There are multiple reasons for the behavior to fail classifying correctly :

Conclusion & Future work

Different aspects could benefit from further improvement : Also, this project can provide good insights for further research: