Spatiotemporal Bundle Adjustment for Dynamic 3D Reconstruction

Dance and Jump

Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. However, the triangulation constraint is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not purposed to estimate the temporal alignment between cameras. In this paper, we present a spatiotemporal bundle adjustment approach that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating 3D static points, as well as subframe temporal alignment between cameras and estimating 3D trajectories of dynamic points. The key to our joint optimization is the careful integration of physics-based motion priors within the reconstruction pipeline, validated on a large motion capture corpus. We present an end-to-end pipeline that takes multiple uncalibrated and unsynchronized video streams and produces a dynamic reconstruction of the event. Because the videos are aligned with sub-frame precision, we reconstruct 3D trajectories of unconstrained outdoor activities at much higher temporal resolution than the input videos.

Publications


"Spatiotemporal Bundle Adjustment for Dynamic 3D Reconstruction"
Minh Vo, Srinivasa Narasimhan, and Yaser Sheikh
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016.
[
PDF][Poster][Code (Coming soon)]

Physics-based Motion Prior

Among the infinitely many 3D trajectories hit by rays from different video cameras (and hence, perfectly satisfy the geometry constraint), only the true 3D trajectory correctly temporally alligns all the cameras. This motivates us to find a motion prior that ideally estimates a trajectory corresponding to the correct temporal alignment.

Our physics-based motion prior is compact. It can model complex 3D trajectories and faciliate accurate temporal alignment of multiple cameras.

Results on synthetic dataset



Results on real dataset









Acknowledgements


This research is supported by the NSF CNS-1446601, the ONR N00014-14-1-0595, and the Heinz Endowments "Plattform Pittsburgh".