Description
Rich scene understanding from 3D point clouds is a challenging task that requires contextual reasoning, which is typically computationally expensive. The task is further complicated when we expect the scene analysis algorithm to also efficiently handle data that is continuously streamed from a sensor on a mobile robot. Hence, we are typically forced to make a choice between 1) using a precise representation of the scene at the cost of speed, or 2) making fast, though inaccurate, approximations at the cost of increased misclassifications. In this work, we demonstrate that we can achieve the best of both worlds by using an efficient and simple representation of the scene in conjunction with recent developments in structured prediction in order to obtain both efficient and stateoftheart classifications. Furthermore, this efficient scene representation naturally handles streaming data and provides a 300% to 500% speedup over more precise representations.
Videos
Datasets
 CamVid
 NYUScenes: [images] [ground truth]
 MPIVehicleScenes: [images] [ground truth]
Results from Miksik et al. ICRA'13 paper
Perframe classifications: [CamVid 05VD] [CamVid 16E5] [NYUScenes] [MPIVehicleScenes]Temporally smoothed classifications: [CamVid 05VD] [CamVid 16E5] [NYUScenes] [MPIVehicleScenes]
File formats:
 *_seg.txt: An MbyN matrix with values [0,S1], which maps each pixel to one of S segments.

*_infer.dat: A SbyK matrix where each row is the Kclass
probability distribution for the segment.
(Unfortunately the *_infer.dat files are not available for the temporally smoothed classifications.)  *_infer.txt: An MbyN matrix with values [0,K1], which maps each pixel to the class with highest probability.
Presentations
 ICRA 2013 (Hu) slides: [pptx] [pdf]
 ICRA 2013 (Miksik) talk: [pptx] [pdf]
 ICRA 2009 talk: [pptx] [pdf]
References
Efficient 3D Scene Analysis from Streaming Data
ICRA 2013 [pdf] [supplementary video] [project page] [bibtex] 

Efficient Temporal Consistency for Streaming Video Scene Analysis ICRA 2013 [pdf] [supplementary video] [project page] [bibtex] 

Onboard Contextual Classification of 3D Point Clouds with Learned Highorder Markov Random Fields ICRA 2009 [pdf] [project page] [bibtex] 
Funding
 QinetiQ North America Robotics Fellowship
 Collaborative Technology Alliance Program, Cooperative Agreement W911NF1020016