Gestures Application

Computational Photography Final Project

My goal here is to make a cool app using a camera and a projector.

Basically the notion of projector based graphics, with a camera and a projector facing the same image plane, can lead to a number of applications that I find really interesting. The example that got me started on was this video of people interacting with a projected box: here.

The final product I wanted was an interactive photo manipulation program. The first feature of this program will be an iphone or microsoft surface where you move a pile of pictures around with your hands:

From there there are other manipulation tools, like air brush, blur, smudge, etc seem like a natural fit for this format.

Initial results:
For both of these the upper right is the program window, which will eventually be full screen and on a projector. The lower right is what the camera sees, and the lower left is the computed output.

Camera calibration detecting checkerboard pattern (note that the camera is paused to get a better screenshot):

Background subtraction for finger detection:

Next I did a color calibration, since this is what black looks like:

Here is background subtraction on the final app without the color correction on top, and with color correction on bottom.

Now the blob based finger detection, basically a threshold of the bottom of the image:

Then I just calculate the tip of the finger, and use it. To do this I use a really dumb algorithm, wich just looks for the highest line that has finger points in it. This is a safe assumption though since the projected screen is above my head. A more intelligent algorithm that calculated the tip of the blob intelligently could be used. Here you can see three images that you can move back and forth, change the order of the pile, etc.:

Next I added the blur image feature. Here are two examples, the first i am blurring the top of the pole, and the second i am blurring the bush in the center. In each the 4 panels are (starting in the upper left, going clockwise) blurred expected image, unblured camera photo, finger detection (filtered threshold of the difference of first two), and program output screen:

Next a quick reset of the image and a second example:

Now in all of these examples I've shown I wasnt really using a projector. For ease of testing, I used either my laptop or my desktop monitors, but this works depending on the lighting in the area. However I have tested it (and presented it) on a projector, and it works just as well there.