The idea of my final project is to remove the current sky in an image, and relight the image using an HDR image of the sky. Using HDR images of the sky at different periods during the day, it is possible to create a series of relit images at different times of day. These images can then be strung together into an animated time-lapse image of the sun moving throughout the day.
Because an image is only a 2D representation of the scene, I'm using a similar method to tour into the picture to create a 3D representation of the image. This 3D representation will have a floor, left and right walls, and a back wall, but no ceiling (because it is assumed to be an outdoor scene). The sky in the original image is removed using an alpha channel, and it is assumed that the sky was a constant illumination. Using this the illumination from this sky is subtracted out from the model, allowing for a new sky to be inserted and relight the image.
An ideal image would be an image of a city on an overcast day, with a single point perspective. It also would have the tops of buildings in the image so that the shadows can be determined better.
Here are some examples of this 3D construction from the original image:
|Original Image||Alpha Mask|
|User selected back plane and vanishing point||Calculated planes for each wall|
|3D Representation of the city with sky removed|
To relight the scene I used HDR light probe images from Jean-Francois Lalonde's research. Each of the light probes were rendered from a web camera image at different times of day. I used 151 images over the course of the day, and a sampling of these images can be seen in tone mapped versions below. The colors aren't entirely accurate due to distortions when tone mapped to rgb from hdr.
To create the visibility map for a given point in the image I did the following:
1. Cast a ray from the image point in 3D space to each pixel on the sky dome.
2. See how the angle of this ray compares with the normal at this point. This was caculated using rayDir dot normal
-If this value was less than zero, then the ray is coming from behind and therefore does light the point.
-Otherwise the ray may light the point if it's not blocked by another surface.
3. Check that another wall doesn't block this ray
-Loop through each of the 3 other walls and see where the ray intersects the plane each wall is in.
-If it intersects the plane before hitting the pixel, then check if it intersects where there are buildings.
-To check this I passed in a coded image that had each wall represented by a number and any part that had alpha=0 represented by a 0 (an example can be seen below).
4. If the ray isn't obstructed, then put the value of rayDir dot normal in the visibility map.
Otherwise, put a value of 0 in the visibility map.