We present an approach to capture the 3D structure and motion of a group of people engaged in a social interaction.
The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion
needs to be measured over a space large enough to host a social group; and (3) human appearance and con-
figuration variation is immense. The Panoptic Studio is a system organized around the thesis that social interactions
should be measured through the perceptual integration of a large variety of view points. We present a modularized
system designed around this principle, consisting of integrated structural, hardware, and software innovations. The
system takes, as input, 480 synchronized video streams of multiple people engaged in social activities, and produces,
as output, the labeled time-varying 3D structure of anatomical landmarks on individuals in the space. The algorithmic
contributions include a hierarchical approach for generating skeletal trajectory proposals, and an optimization
framework for skeletal reconstruction with trajectory reassociation.