Here is the subset of LabelMe used  in the paper:

Tomasz Malisiewicz, Alexei A. Efros. Recognition by Association via
Learning Per-exemplar Distances. In CVPR, June 2008.


Several things to note:

0.)The folder chosen for testing is the
  "05june05_static_street_boston" folder.

1.)I added some objects that were not present in the original LabelMe
  annotations by hand; however, the most recent LabelMe might now
  contain those objects as well as many more objects.

2.)Images and their annotations were resized to 300x400

3.)The really small objects have been removed.  I think tiny is
  defined as less than 2000 pixels in 300x400 image.

4.)Objects do overlap.  This is how labelme provides annotations. I
  did not use the LabelMe code that uses heuristics to resolve segment
  overlaps.  Things to note are roads overlapping car tires and trees
  overlapping buildings.

5.)There are a total of 1146 objects spanning 18 labels.  It ends up
  lot of different labels were present in the tiny objects which were
  removed.  Lots of examples of road, car, building, sky, tree.

