Large Displacement Optical Flow Matlab Code

[Download ]

The code implements a coarse-to-fine variational framework for optical flow estimation between two image frames that incorporates descriptor matches in additiona to the standard brightness and gradient constrancy constraints, for dealing effectively with large displacements of small (and large) structures. The code is based on [1] and [2]. We denote with w(x)=(u,v) the displacement field between two images I_1 and I_2. Following [2], we obtain descriptor matches w_descr(x) by matching densely sampled HOG descriptors in the two images via approximate nearest neighbor search. We then compute w(x) by minimizing the energy functional:

In the above equation, the first term encodes intensity and gradient constancy, the second term encodes robust smoothness constraint and the last term biases the displacement field w(x) towards the confident descriptor matches w_descr(x). \delta(x) is a delta function indicating if a descriptor match is available in location and \rho(x) denotes confidence of the match. Descriptor matches are obtained by matching densely sampled HOG descriptors in the two images.
We minimize the above functional in a coarse-to-fine scheme: at each pyramid level the non-linear Euler-lagrange equations are solved via fix-point iterations. At each iteration, the sparse linear system is solved efficiently via succesive overelaxation (SOR) as described in [1]. Intuitively, at each pyramid level the pixel displacement estimates and pairwise/unary robust pixel potentials are updated iteratively: given the current pixel motion estimates w(x), the robust unary and pairwise terms are computed and vice versa. Once a fix-point has been reached, the flow field is upsampled to initialize the computation at a finer pyramid level.

Examples

The (back) warped second image should coincide with the first except for places that get occluded or dis-occluded. In those places optical flow is not well defined and may be arbitrary. Forward-backward consistency check can discard flow vectors in occluded regions.

RGB Flow field descriptor matching t t+1 image 1 & (back)warped image2

RGB Flow field descriptor matching t t+1 image 1 & (back)warped image2

How to use?

After downloading the folder, run start_up.m while being inside the folder, to compile the mex files. Then you should be able to run the demo. We observed that the image dowsampling factor for creating the image pyramid needs to be close to one for the linearizations to be valid and the small displacement assumption to hold at each pyramid level, to deal effectively with very large displacements. It can be varied according to the ammount of displacement, small downsampling factor results in fewer pyramid levels and thus smaller running time. Please report comments/bugs to katef@seas.upenn.edu.

References

[1]	T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. In European Conference on Computer Vision (ECCV), pages 25–36, 2004.
[2]	Thomas Brox, J. Malik. Large Displacement Optical Flow: descriptor matching in variational motion estimation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3): 500-513, 2011

Last update: Dec, 2012.