Rectification Without Lines

Carl Doersch and Alexei Efros

Abstract: When inferring scene geometry from a single image, one of the first steps is to estimate the vanishing points of each surface in the image in order to recover its orientation. If the surface contains detectable parallel lines then this is easy. But what if there aren't any? Another option is to make use of repeated patterns. Many surfaces, like the building above, contain small elements, like the stones, that are approximately repeated across the surface. If the surface is rectified (its vanishing line is at infinity), then these elements will all be approximately the same size.

This intuition leads to a simple way to 'score' a homography: we transform the image by that homography, extract random patches, and correlate them with the rest of the transformed image. If the homography comes close to rectifying the image, then there should be a large number of strong peaks in the correlation map. Given the scoring function, we can simply search over the space of homographies. We found that gradient ascent works reasonably well on this scoring metric, even though there are no guarantees of convexity.

The algorithm qualitatively worked well on a handful of test surface patches (for example, the one above, where the left is the original surface and the right is the output of our algorithm). However, we never found a good way to evaluate our algorithm in general, since the sorts of surfaces where this algorithm would be useful are somewhat rare.

pdf: under construction