Imagine you want to take your partner down the memory lane and show them the place where you grew up -- the lake, the greenery and the mountains, and make them hear the melodies of chirping birds, the dancing trees, and the rustling leaves. How would the place look like on a sunny day? How mesmerizing it becomes with clouds and rain? How important mountains and forests were to the beauty of the area? and how gloomy it became when the lake once dried up due to the drought? What if I give you a tool to do that.
My research is primarily about building the Computational Studio, a computational machinery that continually learns the 4D audio-visual world from the sparse real-world samples in an unsupervised manner and enables audio-visual social communication for non-experts on their everyday computational devices. There are three essential aspects of the Computational Studio: (1) capturing 4D visual world along with audio; (2) example-based audio-visual synthesis; and (3) interactively synthesizing the audio-visual world.
My work on the Computational Studio lies at the intersection of Computer Vision and Graphics, Machine Learning, Robotics, Human-Computer Interaction, and Psychology.