OUR RESEARCH

Carnegie Mellon Goes to the Super Bowl

Football fans tuning into this year’s Super Bowl were treated to a unique new view of the action during playbacks. CBS Television used a new technology co-developed by the network and Carnegie Mellon computer vision expert Takeo Kanade.

“Eye Vision”, as CBS calls it, involves shooting multiple video 01_images of a dynamic event, such as a football game, from multiple cameras placed at different angles. The video streams from these cameras are combined by computer and the resulting 01_images reach viewers in a format that will make them feel as if they are flying through the scenes they see.

The action at Super Bowl XXXV was captured by more than 30 cameras, each poised some 80 feet above the field at Raymond James Stadium in Tampa, Fla. Each camera, with computer-controlled zoom and focus capabilities, was mounted on a custom-built, robotic pan-tilt head, which could swing the camera in any direction at the command of a computer. These camera heads were controlled in concert so that cameras pointed, zoomed and focused at the same time on the same spot on the field, where a touch down or fumble occurred.

How does the system work? One of the camera heads is designated as the master camera. A human camera operator manipulates a movable pan-tilt tripod attached to a TV screen on which the video from the master camera is constantly displayed. The pan-tilt tripod is equipped with sensors to constantly measure its angle. The master camera head moves by mimicking the motion of the tripod as the camera operator moves it to capture a moving object on the field on his or her TV screen.

In the meantime, information collected from the master camera’s pan-tilt angles, zoom and focus is fed to a computer, which quickly computes the appropriate control signal for each of the remaining cameras. This causes all of them to converge on the same target and capture its image from a variety of angles.

Live action on the football field is continuously captured up to 30 times per second by the video cameras. The video of each camera is synchronized and time stamped so that all the views at the most critical and interesting moments can be played back in sequence, as if a viewer had flown around the action.

Kanade explained his technology in an interview from Tampa, which aired during the Super Bowl Pre-Game Show. He noted that the “Eye Vision” demonstration that will appear on Super Bowl Sunday is only a small part of this new technology, which he calls “Virtualized Reality™” as opposed to virtual reality. It is the product of more than six years of research.

For this modeling technology to achieve its full impact, the set of captured, multiple video 01_images must be processed beyond the play back. The detailed geometrical information about a scene is extracted by computer, which enables a person to choose how to view a scene, even from a perspective that was not shot by any camera.

To bring this concept to life, Kanade and his students built a “3D room” equipped with more than 50 video cameras, and experimented by filming people involved in a variety of sports activities. He also spun off a company named Zaxel Systems, Inc., for commercialization of the technology. Much of this work can be viewed at the Virtualized Reality™ Web site.

In contrast to virtual reality, in which synthetic environments are created, Virtualized Reality™ technology, and to a lesser extent, Eye Vision, are based on events taking place in the real world, which are captured and processed by computer manipulation.

“Because our models are derived from real 01_images,” Kanade says, “the models look much more real than typical virtual worlds.”

Kanade says the output from these multiple cameras shooting a scene together from many angles actually can create totally new views that were not captured by any camera. As this technology develops, he believes it will create a completely new way to view sports and entertainment events. People will be able to customize the perspective from which they watch, for example, from that of a particular player or the ball.

Kanade is the former director of Carnegie Mellon’s Robotics Institute. He has been a leader in the development of video-based vision systems used in the university’s autonomous vehicles and exploration robots. His team has developed a vision-based autonomous helicopter, which ultimately may be able to aid in search and rescue operations, fire fighting and inspection tasks.

He is also a pioneer in medical robotics and computer-assisted surgery, working with surgeons and medical professionals to develop smart tools capable of performing medical procedures better than a physician or machine could do alone.

Kanade earned his undergraduate and graduate degrees in electrical engineering from Kyoto University, Japan. He has been on the Carnegie Mellon faculty since 1980 and director of the Robotics Institute since 1991. He is a member of the National Academy of Engineering. Most recently he received a $100,000 award from the NEC Foundation for C&C Promotion for “fundamental and broad contributions to the development of multimedia through the advancement of robotics and computer vision.”

Reprinted with kind permission from Lisa Kulick, Carnegie Mellon associate director of public relations - Web development