• Home
• People
• Projects
• Publications
• Databases
• Courses
• Events

Matting and Depth Recovery of Thin Structures using a Focal Stack

Thin structures such as fences, grass and vessels are common in photography and scientific imaging. They contribute complexity to 3D scenes with sharp depth variations/discontinuities and mutual occlusions. In this paper, we develop a method to estimate the occlusion matte and depths of thin structures from a focal image stack, which is obtained either by varying the focus/aperture of the lens or computed from a one-shot light field image. We propose an image formation model that explicitly describes the spatially varying optical blur and mutual occlusions for structures located at different depths. Based on the model, we derive an efficient MCMC inference algorithm that enables direct and analytical computations of the iterative update for the model/images without re-rendering images in the sampling process. Then, the depths of the thin structures are recovered using gradient descent with the differential terms computed using the image formation model. We apply the proposed method to scenes at both macro and micro scales. For macro-scale, we evaluate our method on scenes with complex 3D thin structures such as tree branches and grass. For micro-scale, we apply our method to in-vivo microscopic images of micro-vessels with diameters less than 50 microns. To our knowledge, the proposed method is the first approach to reconstruct the 3D structures of micro-vessels from non-invasive in-vivo image measurements.

Publications " Matting and Depth Recovery of Thin Structures using a Focal Stack " Chao Liu, Artur W. Dubrawski and Srinivasa G. Narasimhan IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. [PDF] [supp] [poster]

Illustration Viewing geometry of a single pixel in a camera with finite aperture. The camera is focused between occluder k and occluder N-1. The pixel receives radiance contributions from rays within the double-sided cone determined by the focal plane and aperture size. The occluders are represented with the occlusion map M and radiance map L. Occluder k is partially occluded by the occluders in its near field and occludes the occluders/background in its far field. We use MCMC to estimate the matting. However, we have to re-render the image every time when the matting variables changes, which is too computationaly expensive. Instead, we render the differential image, which can be estimated efficiently without sacrificing the accuracy. To estimate the occluder's depth, we model the the scene as a set of planar surfaces. Each planar surface is described by a 3-by-1 vector (surface normal and depth). The set of vectors are solved by gradient descent method. We recovered the depth of thin structures, where high sptial frequency depth discontinuities are present. Note that the depth estimations using the traditional DFF method for points close to the occlusion boundaries are inaccurate due to high frequency depth discontinuity. As a result, the estimated depth map for the thin structures appears wider. In contrast, our mehotd recovers the depth discontinuities faithfully. We apply our method to in-vivo microscopic images of micro-vessels with diameters less than 50 microns. We reconstruct the 3D structure of the microvessels despite spatially varying blur and occlusions. To our knowledge, this is the first method to reconstruct the 3D structures of micro-vessels from a non-invasive in-vivo imaging system.

Video

We show the input focal stack, the depth & matting results and the 3D reconstruction of micro-vessels in the sublingual area of a living pig.

Acknowledgements

We thank the Disruptive Healthcare Technology Insti- tute (DHTI) supported by Highmark Inc. and Allegheny Health Network, and NSF (award 1320347) for supporting this work.