Translating Faces

German Agent Speaking

Japanese Agent Speaking
When people communicate across video conference links we may not only want to hear the foreign speaker speak in our own language, but also see his lips move accordingly. Vision processing, face and eye tracking combined with a little computer graphics manipulation make it possible. Above, we see our travel agent Max speak in German at the transmitting end. ....And in the US, voilà: hear and see him speak English. The technique also saves bandwidth, since only preselected codes for lip shapes (and/or facial gestures) need to be transmitted.

Translated Agent Speaking

Translated Agent Speaking

Related Publications:
  • X.W. Chen and J. Yang, "Visual Speech Synthesis Using Quadtree Splines," Proceedings of ICASSP 2001, Salt Lake City, May 2001.
  • J. Yang, J. Xiao, M. Ritter, "Automatic Selection of Visemes for Image-based Visual Speech Synthesis," Proceedings of First IEEE International Conference on Multimedia (IEEE ME2000) .
  • M. Ritter, U. Meier, J. Yang, A. Waibel, "Face Translation: a Multimodal Translation Agent," Proceedings of AVSP 99 .