A framework for learning to recognize and segment object classes


Caroline Pantofaru
Martial Hebert


Input  Image
Classification mask (red = higher confidence; blue = lowest confidence)

This project explores techniques for pixel-level object segmentation. Two main ideas are explored. First, we use features computed over regions of image segmentations in addition to the usual local features used in recognition. We use these regaion features for recognition and good overall performance but, more importantly, we show that using regions from an over-segmentation enables pixel-level labeling of the image, instead of finding merely a bounding box or approximate outline as is commonly done.

Second, we show how recognition and region segmentation can be combined into a system which is trained by using weakly supervised training data. In order to achieve pixel-level labeling for rigid and deformable objects, we employ regions generated by unsupervised segmentation as the spatial support for our image features, and explore model selection issues related to their representation. We examined the influence that different model choices can have on its performance.  Pixel-level classification accuracy was evaluated on two challenging and varied datasets.


C. Pantofaru and M. Hebert,  A framework for learning to recognize and segment object classes using weakly supervised training data, British Machine Vision Conference, September, 2007.

C. Pantofaru ,  G. Dorkó, C. Schmid,   M. Hebert,  Combining Regions and Patches for Object Class Localization, The Beyond Patches Workshop in conjunction with the IEEE conference on Computer Vision and Pattern Recognition, June, 2006, pp. 23 - 30.


Copyright notice