Qualcomm has accepted Gengshan Yang and N Dinesh Reddy into its current class of Innovation Fellows for the pair's work on creating computer-generated 3D models of traffic, people, animals and their interactions in cities. They join 15 other projects selected from the hundreds that applied, and will receive $100,000 in funding and mentoring from top Qualcomm engineers.

Understanding the varied activities in urban centers is a challenge facing smart cities, whose components — humans, buildings, modes of transit, animals, etc. — interact in many ways. Some of those ways are common, like vehicles stopping at an intersection to allow pedestrians to cross, and some rare, like construction, festivals or accidents. These rare events are hard to reconstruct with current computer vision algorithms.

Yang and Reddy, both Ph.D. candidates in the Robotics Institute, gathered unlabeled videos from mobile devices and city surveillance cameras and built a framework that can reconstruct these scenes in a 3D model usable by smart city applications and other industries. Their framework allows models to be created from the videos themselves, rather than relying on predefined shapes. It can model rare and uncommon objects and generalize scenes beyond existing methods.

Computers will use these 3D models to sense and see what is happening in cities. This ability will be important to connected and self-driving vehicles, sharing data over 5G networks and in augmented reality and virtual reality applications.

Yang and Reddy's fellowship project, "Open World 3D Dynamic Reconstruction for Smart Cities" will be based on prior work by the pair, including Yang's paper, "LASR: Learning Articulated Shape Reconstruction From a Monocular Video," and Reddy's paper, "Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision."