There already exist many single-view modeling programs for architectural scenes with heavy planar geometry. There are also non-view based modeling programs for non-geometric curved objects (particularly relevant to this paper is Takeo Igarashi's work on Teddy ). This work combines the two ideas to make an image based modeler for non-geometric curved objects. The program has been tested on images of human faces among other things. While the algorithm is much more general then a face modeler, I find that faces provide a rigorous test of my hypothesis, as a small error in a face image is readily apparent to a human viewer.
This paper presents a method for modelling simple curved surfaces from a single image. The premise of the work is that the geometry of an object can be radically simplified and still produce convicincing new views, provided the object is textured with a detailed photograph or image. The program described in this paper provides a simple interface to model a small set of shapes, and accurate texture them with an image or painting. The program then allows the user to rotate the resulting model to produce novel views of the image.
The first portion of the interface is the actual modeler itself. The user uses a mouse to sketch out closed loops, reminiscent of contour lines. The loops do not, however, act as contour lines per se. Rather the area enclosed by each loop is bumped out as on ovloid shape which is represented as a height field on top of the 2d image (this idea was based on Takeo Igarashi's work ). The resulting height fields from each loop are summed to form a final height field for the image. A screenshot of sample input is shown below.
The loop around the chin region is necessary, because the human face isn't a perfect oval. The slight "bumping out" of the chin area deforms the oval to be closer to the human face shape. A sample heightfield is shown below. In this heightfield, green represents heights closer to 0 (with respect to the image plane), while purple represents heights closer to the maximum of the heightfield.
The second portion of the interface is the 3d rotation tool, it takes in the photo-with-heightfield from the modeler, and allows the user to interactively rotate the image and save the result as a tiff.
Each loop that the user sketches is made into a bounding
polygon. This program differs significantly from Teddy in the
algorithmic details. First off, the polygon sides aren't
normalized to be the same length. Second, there are no tests
for self-intersection, the algorithm used here can use
self-intersecting polygons as easily as it can use single-loop
polygons. Membership in the polygon is determined by the parity
of the number of line segments crossed when drawing a line
between <0,0> and the pixel being tested (some additional tests
are used to catch the degenerate cases where two line segments
meet). The heightfield is initialy set to
Height(thisPixel due to thisPoly) := isInside(thisPixel, thisPoly) * closestDist(thisPixel, thisPoly)
This process could be greatly sped up by precomputing the constrained Delaunay triangulation of the polygon (in fact, could be converted to a really simple scanline algorithm). While the heights are being computed, a variable keeps track of the maximum distance of any interior point to the polygon border. This is stored in a variable called maxheight. The heights due to each polygon are transformed by
Height(thisPixel) := sqrt( maxheight*maxheight - (maxheight-Height(thisPixel))*(maxheight-Height(thisPixel)))
This makes the entire shape into a piecewise mesh of partial spheres, with radius equal to maxheight. The shape is C0 continuous, and as the number of points on the polygon approaches infinity, the shape approaches C1 continuity. The shape due to each polygon is then summed toghether (making it easy to get noses, etc.). Note that the sum of these height-fields (which is the final result) is usually not C1 continuous, but still has garrunteed C0 continuity.
Once the height-field has been computed, the heights are converted to "generalized disparity" values, so that they can be viewed using McMillan's algorithm for 3d viewing using image-warping .
Because the premise of the work is that detailed textures can
make up for poor or slightly innacurate shape information, the
algorithm does create poor / innacuratte shape information.
When too few contours are drawn, or when a shape is rotated to
extreme positions, this becomes readily apparent. Here is an
Here is the input image
Here is the image, rotated a little bit
Here is a view rotated to extreme positions
The first piece of future work to be done would be to speed up the heightfield creation process, by pre-processing polygons into Delaunay triangulations. The other remaining area of interest is texture synthesis for hole-filling. An approach that I did not have time to implement was to find a scalar field that yields the heightfield as an implicit surface, using numerical finite element methods, and then rewarp the surface onto a bounding cube by following the gradient of the scalar field from every point on the object surface, to the bounding cube. Texture synthesis can then be used on the bounding cube and reprojected back onto the 3d object. Also, missing shape detail could be synthesized by extending traditional texture-synthesis methods into 3d, possibly interpolating 3d shape and color as a hypertexture.