Video Completion by Motion Field Transfer

Takaaki Shiratori, Yasuyuki Matsushita, Sing Bing Kang, Xiaoou Tang
(Joint work with Microsoft Research Asia)

  • In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), June 2006 [paper] [poster]


Abstract

Existing methods for video completion typically rely on periodic color transitions, layer extraction, or temporally local motion. However, periodicity may be imperceptible or absent, layer extraction is difficult, and temporally local motion cannot handle large holes. This paper presents a new approach for video completion using motion field transfer to avoid such problems. Unlike prior methods, we fill in missing video parts by sampling spatio-temporal patches of local motion instead of directly sampling color. Once the local motion field has been computed within the missing parts of the video, color can then be propagated to produce a seamless hole-free video. We have validated our method on many videos spanning a variety of scenes. We can also use the same approach to perform frame interpolation using motion fields from different videos.


Results


Hole-filling

We set the spatio-temporal hole to the input video sequences. Then using our motion field transfer, the hole is filled with the estimated local motion field. Finally, video completion is achieved by propagating color.


Walking (DivX avi)



Performer (DivX avi)

Object removal

This movie shows a useful application of video completion: object removal. Here, we manually removed the foreground person and automatically filled the hole using our motion field transfer technique. Even though there is complex motion in the background, the fill-in was accomplished well.


Performance (DivX avi)

Frame Interpolation

This movie shows another result of video completion for a breakdance scene. Unlike previous examples, here we recover entire frames. The input video contains rather complex motions that are not periodic. Despite this, our method was able to interpolate frames by treating the new intermediate frames as holes.


Breakdance (DivX avi)

Frame rate recovery with prior video

Given a low-frame rate input video, we recompute the local motion in the prior video with the same frame rate as the input video. Motion field transfer is then performed to find out the most similar motion patch from the prior video. Once the patch is found, the full-resolution motion field is warped to the low-frame rate video to achieve a higher frame rate. In this way, the frame rate recovery is performed. The video shows the video chat scenario where prior video is especially effective.


Video chat (DivX avi)

(The latest DivX encoder is downloadable from DivX website)


Media


Acknowledgement

This work was done while Takaaki Shiratori was visiting Microsoft Research Asia.



Top
Home