Hanbyul Joo
(Also known as "Han")
Ph.D. candidate

The Robotics Institute
School of Computer Science
Carnegie Mellon University
Office: EDSH 201
Email: hanbyulj at cs dot cmu dot edu

CV (updated in Dec. 2017)


I am a Ph.D. candidate (expect to graduate in May 2018) in the Robotics Institute at Carnegie Mellon University, under the supervision of Prof. Yaser Sheikh. I am also actively collaborating with Prof. Mina Cikara in Harvard University and Prof. Hyun Soo Park in UMN. I interned at Oculus Research Pittsburgh (Summer and Fall, 2017) and Disney Research Zurich (Summer, 2015). Before joining CMU, I spent three years as a researcher at ETRI, a government-funded research organization in Korea. I received my M.S. in Electrical Engineering (under the supervision of Prof. In So Kweon), and B.S. in Computer Science, both from KAIST, Korea. The Samsung Scholarship is supporting my graduate study.

Research

The goal of my research is the creation of a new scientific discipline of computational behavioral science, by measuring the full spectrum of social signals transmitted during an interpersonal social interaction. I have been developing a unique sensing system at CMU, the Panoptic Studio, which is composed of more than 500 synchronized cameras. My research is based on the tools of computer vision, machine learning, computer graphics, and robotics.

   
Markerless Total Motion Capture
  The Panoptic Studio
  Panoptic DB (2D/3D body keypoint annotations)

News

Dataset/Library

CMU Panoptic Studio Dataset

OpenPose Library

Tutorial

DIY A Multiview Camera System: Panoptic Studio Teardown
Hanbyul Joo, Tomas Simon, Hyun Soo Park, Shohei Nobuhara, and Yaser Sheikh
In Conjunction with CVPR 2017
[Tutorial Page] [Video]

Publications

Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies
Hanbyul Joo, Tomas Simon, and Yaser Sheikh
In CVPR 2018 (to appear)
[arXiv] [video] [Project Page]

Structure from Recurrent Motion: From Rigidity to Recurrency
Xiu Li, Hongdong Li, Hanbyul Joo, Yebin Liu, Yaser Sheikh
In CVPR 2018 (to appear)

Hand Keypoint Detection in Single Images using Multiview Bootstrapping
Tomas Simon, Hanbyul Joo, Iain Mattews, and Yaser Sheikh
In CVPR 2017
[arXiv] [Project Page] [Code] [Dataset]

Panoptic Studio: A Massively Multiview System for Social Interaction Capture
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh
In TPAMI 2017 (Extended version of ICCV15)
[published version] [arXiv version] [Dataset]

Panoptic Studio: A Massively Multiview System for Social Motion Capture
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh
In ICCV 2015   (Oral) - Acceptance ratio: 56/1698~3.3%
[Paper(PDF)] [Supplementary Material] [Project Page]
Press Coverage:

MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction
Hanbyul Joo, Hyun Soo Park, and Yaser Sheikh
In CVPR 2014   (Oral) - Acceptance ratio: 104/1807~5.8%
[Paper(PDF)] [Project Page] [Dataset]
Press Coverage: ...

Graph-based Shape Matching for Deformable Objects
Hanbyul Joo, Yekeun Jeong, Olivier Duchenne, and In So Kweon
In ICIP 2011
[Paper(PDF)]

Graph-Based Robust Shape Matching for Robotic Application
Hanbyul Joo, Yekeun Jeong, Olivier Duchenne, Seong-Young Ko, and In So Kweon
In ICRA 2009
[Paper(PDF)]

Statistical Background Subtraction Based on the Exact Per-pixel Distributions
Youngbae Hwang, Hanbyul Joo, Jun-sik Kim, and In So Kweon
In IAPR workshop on Machine Vision Applications (MVA) 2007
[Paper(PDF)]

Talks

  • "Measuring and Modeling Social Signals for Computational Behavioral Understanding"

    • MIT, Media Lab, Nov 2017.
  • "The Panoptic Studio: A Massively Multiview System for Social Interaction Capture"

    • UC Berkeley, Computer Vision Group, Dec 2016 (hosted by Prof. Alexei A. Efros).
    • Stanford, Computer Vision and Geometry Lab (CVGL), Dec 2016 (hosted by Prof. Silvio Savarese).
    • Adobe Research, Dec 2016 (hosted by Dr. Joon-Young Lee).
    • ASSP4MI workshop of ICMI, Nov 2016.
    • CMU, Machine Learning Lunch Seminar, Oct 2016.
    • ICCV Oral Talk, Dec 2015 (video link).
    • CMU, VASC Seminar, Dec 2015.
    • ETH Zurich, Computer Vision and Geometry lab, Oct 2015 (hosted by Prof. Marc Pollefeys).
    • Seoul National University, June 2015 (hosted by Prof. Kyoung Mu Lee).
    • ETRI, CG Team, May 2015 (hosted by Dr. Seong-Jae Lim).
    • KAIST, May 2015 (hosted by Prof. In So Kweon).
  • "The Panoptic Studio (CVPR14)"

    • Graduate Seminar, Civil & Environmental Engineering, CMU, Feb 2015 (hosted by Prof. Hae Young Noh).
    • People Image Analysis Consortium, CMU, Nov 2014.
    • Reality Computing Meetup, Autodesk, Nov 2014.
  • "MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction"

Press Coverage

Patents

  • Motion capture apparatus and method (Patent No.: US 8805024 B2)
    Hanbyul Joo, Seong-Jae Lim, Ji-Hyung Lee, Bon-Ki Koo

  • Method for automatic rigging and shape surface transfer of 3D standard mesh model based on muscle and nurbs by using parametric control (Patent No.: US 7171060 B2)
    Seong Jae Lim, Ho Won Kim, Hanbyul Joo, Bon Ki Koo

  • 3D model shape transformation method and apparatus (Patent Application No.: US 20120162217 A1)
    Seong-Jae Lim, Hanbyul Joo, Seung-Uk Yoon, Ji-Hyung Lee, Bon-Ki Koo.