Seeing through Water: Image Restoration using Model-based Tracking

A video sequence of an underwater scene taken from above the water surface suffers from severe distortions due to water fluctuations. In this paper, we simultaneously estimate the shape of the water surface and recover the planar underwater scene without using any calibration patterns, image priors, multiple viewpoints or active illumination. The key idea is to build a compact spatial distortion model of the water surface using the wave equation. Based on this model, we present a novel tracking technique that is designed specifically for water surfaces and addresses two unique challenges --- the absence of an object model or template and the presence of complex appearance changes in the scene due to water fluctuation. We show the effectiveness of our approach on both simulated and real scenes, with text and texture.


"Seeing through Water: Image Restoration using Model-based Tracking"
Yuandong Tian and Srinivasa G. Narasimhan,
Proc. of IEEE International Conference of Computer Vision (ICCV),
Oct, 2009.

"The Relationship between water depth and distortion,"
Yuandong Tian and Srinivasa G. Narasimhan,
Technical Report CMU RI,
Aug, 2009.

Code and Data

Codes and Data can be downloaded here.
Additional water-distorted test images can be downloaded here:
Middle Font, Small Font, Tiny Font.

Poster and Presentation

Presentation and Additional Results


Image distorted by wavy water surface:
Due to light refraction at the fluctuating water surface, a stationary scene immersed in water appears distorted in the camera. By Snell's law, the distortion function w(x, t) at position x and time t is related to the height h(x,t) of the water surface.
Typical image deformation by water distortion:
In each row, the first patch shows the undistorted image; the rest show the distorted ones. One can perceive severe distortions and topological changes in the examples. Note that all the images are synthesized by the wave equation.
The water bases B:
The extracted water bases using PCA on the synthesized data. The bases capture the local wave-like structures of water dynamics and reduce the number of distortion parameters in favor of the upcoming tracking procedure.
The reconstructed distortion using water bases:
The subspace spanned by the water bases has a dimension of ten. However, it captures the basic characteristics of distortion.
Two difficulties in water tracking:
1) There is no template; 2) Unlike affine transform that forms a group, the water distortion is not one-to-one. As a result, a novel tracking technique is proposed to handle these problems.


(Video Result Playlist)
Tracking results on synthesized images:
Water tracking validation using simulated data. The distortion estimations, although not perfect, are reasonable given the complex appearance changes. Details are in the paper.
Image restoration results:
The proposed method is tested on four different sizes of text fonts, as well as check board and brick textures. The first column shows a sample frame from the input video, which is severely distorted. Then the second and third column shows the result by pixel-wise mean/median, which is less distorted but severely blurred. Finally our results are shown on the last two columns using two patch partitions (detailed in the paper). Notice our method alleviates the distortion but still retains image details.
Image synthesis using estimated distortion:
Given the scene with still water surface (the first column), a distorted image can be synthesized using estimated distortion (the second column), which is similar to the corresponding frame of the input video sequence (the third column).

Video stabilization results
In addition, the estimated distortion can be used to stabilize a distorted video by applying the tracking procedure to a temporal sliding window.