Computer Vision for the Visually Impaired
I am working on algorithms to improve the BrainPort vision device which is manufactured by WicabThis is Joint work with Karl Hellstern, Yaser Sheikh, and Takeo Kanade.
With this device blind users can perceive the approximate size, shape and location of objects in their surrounding. Visual information is gathered by a camera that is mounted on a pair of sunglasses. It is then translated into electric pulses that are translated to the surface of the tongue. We have designed a system that allows users of the BrainPort device to recognize faces. The system detects faces in the image captured by the camera, compares them with a "prototypical" or "average" face, and produces a difference map which allows the user to literally feel what is unique about the face of the person in front of it. A prototype of our system was depicted in an episode of the BBC show Frontline Medicine.
The BrainPort device has a steep learning curve to it. It requires the user to create a mapping between what is rendered on their tongue and the visual world around them. This requires a lot of practice, can create frustration in users, and might limit the device's usefulness. Recently we have started work on ways to make this training period shorter, more effective, and also more enjoyable. We are working on a game environment for the device which might be considered as a virtual reality game. As the user moves her head the system matches frames taken from the camera to understand the relative motion of the users head. It can then render what the user "Sees" in the game to the tongue interface. This will allow users to play training games wherever they desire and will increase their engagement with the device, making the training sessions more effective.