Spanish Dancing (Flamenco) with Music

German Cheung, Sundar Vedula, Takeo Kanade

As essential as having good visual images in the virtualized reality project, sound is another important sense in conveying realistic dynamic events to human. Here, we investigated the possibility of adding sound into our synthesized video. Microphones are placed uniformly inside the 3D room together with the cameras. The cameras and microphones are synchronized so that all the sound from the dynamic event is recorded at the same time as the images are captured. The recordings are then incorporated into the synthesized fly-through video. The volume and directions of the sound are carefully controlled so that it is consistent with the virtual camera positions of the synthesized image sequence.

The Matrix-like fly through around a dancer with acoustic effect:

This synthesized video with sound effect can be downloaded here. (Lower Quality) MPEG movie file (4MB) or (Higher Quality) Quick-time movie file (13.6MB)

How do we create it?

Data Collection: To produce the above synthesized video, we have asked a dancer to perform Flamenco in our 3D room with a CD player playing Spanish dancing music on the background. In this particular production, we used 49 cameras for images and six uniformly spaced microphones for sound. The following are images from one of the cameras. The whole video of this camera can be downloaded here MPEG movie file (480KB). In this camera view, two of the six microphones can be seen behind the dancer. The sound clip recorded by one of the microphones can also be heard here WAV Sound Clip file (552KB).

3D Shape Reconstruction & Sound Source Location : By using the image sequences from some of the cameras, the 3D shape of the dancer is reconstructed by using a volumetric method called Shape from Silhouette which creates a voxel model. The voxel model is then converted to surface model by using marching cubes and smoothed. Moreover, we assumed there is only one sound source (the CD player) and its location is determined by considering the relative magnitudes of the recorded sound clips from all the six microphones. In the following video, a red cube is added artificially to indicate the approximate position of the sound source. The video can be downloaded here. MPEG movie file (852KB)

Incorporating the virtualized video sequence with synthesized sound clip: By using the reconstructed models, an image-based rendering technique is used to create new views of virtual cameras. A virtual camera path is designed and new views are generated by combining (consistently with respect to the 3D shape information) image pixels from images of the closest cameras (to the virtual camera). On the other hand, a new sound clip is constructed by using virtual camera path so that the loudness of the sound is inversely proportional to the square of the distance between the sound source location and the position of the virtual camera. Since the sound and images are synchronized during recording, they can be combined easily by comparing the time stamps recorded on the images and sound clips. This is how we create the Matrix-like fly through video at the beginning of this page.

For further technical details, please contact German Cheung ( german+@cs.cmu.edu) or Sundar Vedula( srv+@cs.cmu.edu)

Return to the Virtualized RealityTM Home Page

written and prepared by German Cheung ( german+@cs.cmu.edu)