Bot Intelligence Group (BIG)
Members | Research | Publications

The main theme of research in Bot Intelligence Group (BIG) is to develop robotic intelligence ranging from the low-level autonomy to the high-level cognitive abilities. We aim to develop robots that can cooperate or collaborate with humans in shared environments, learning to improve themselves over time through continual on- and off-line training, exploration, and interactions with humans and/or environments. Towards this general goal, we strive to answer research questions on how to make robots to understand various semantic contexts of physical and/or social environments, act in both task-effective and socially-compliant manners, and communicate their internal states with other agents in intuitive ways.

Research areas: semantic robot navigation, social robot navigation, human-robot interaction/collaboration, vision language planning, creative AI, arts and robotics, simulation-to-real adaptation, robotic intelligence, cognitive robotics

Research projects: Social Robot Navigation | Creative AI | Semantic Robot Navigation


Jean Oh
, PI
Hyeonwoo Yu (postdoc) Xuning Yang Abby (Xinjie) Yao Sam Shum Tanmay Shankar Advaith Sethuraman Peter Schaldenbrand (HCII)
Ingrid Navarro Mayank Mali Xiaopeng Lu Felix Labelle Nariaki Kitamura Jon Francis (LTI) Yuning Wu

Meet our robots that can follow natural language commands.
(Ballbot by Ralph Hollis)


Social Navigation

March 22-24, 2021. Jean Oh is co-organizing a symposium, Machine Learning for Mobile Robot Navigation in the Wild (ML4NAV), at the AAAI 2021 Spring Symposium Series.

C. Mavrogiannis, P. Trautman, A. Steinfeld, D. Zhao, A. Wang, F. Baldini, and J. Oh.  Core Challenges of Social Navigation: A Survey, 2021. [ Arxiv:2103.05668]

D. Zhao and J. Oh. Noticing Motion Patterns: Temporal CNN with a Novel Convolution Operator for Human Trajectory Prediction. In: IEEE Robotics and Automation Letters (RA-L), Special Issue on Long-Term Human Motion Prediction (2020). [Arxiv]

T.-E. Tsai and J. Oh.  A Generative Approach for Socially Compliant Navigation, In: IEEE Conference on Robotics and Automation (ICRA). 2020. [Arxiv]

X. Yao, J. Zhang, and J. Oh.  Autonomous Human-Aware Navigation in Dense Crowds, In: Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). (Late breaking results). 2019.

X. Yao, J. Zhang, and J. Oh.  Following Social Groups: Socially-Compliant Autonomous Navigation in Dense Crowds, In: Cognitive Vehicles Workshop at IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2019.

A. Vemula, K. Muelling, J. Oh. Social Attention: Modeling Attention in Human Crowds. In Proc. of IEEE Conference on Robotics and Automation (ICRA), 2018. *Best Paper Award in Cognitive Robotics* [ArXiv].

A. Vemula, K. Muelling, J. Oh. Modeling cooperative navigation in dense human crowds. In Proc. of IEEE Conference on Robotics and Automation (ICRA), 2017 [pdf].

A. Vemula, K. Muelling, J. Oh. Path Planning in Dynamic Environments with Adaptive Dimensionality. In Proc. of International Symposium on Combinatorial Search (SoCS) 2016. [pdf]

3D Reconstruction for Immersive HRI

H. Yu and J. Oh  Anchor Distance for 3D Multi-Object Distance Estimation from 2D Single Shot, IEEE Robotics and Automation Letters (RA-L), 2021 (to be presented at the IEEE Conference on Robotics and Automation (ICRA'21)). [Arxiv]

H. Yu and J. Oh  A Missing Data Imputation Method for 3D Object Reconstruction using Multi-modal Variational Autoencoder, 2021. [Arxiv]

Creative AI, Arts & Robots

We have been co-organizing a series of workshops on the topic of Creative AI to be held at multidisciplinary venues including arts, AI, machine learning, graphics, and robotics.

June 2021. Peter Schaldenbrand and Jean Oh are co-organizing the second Creative AI workshop, Computational Measurements of Machine Creativity (CMMC): Bridging the Gap between Subjective and Computational Measurements of Machine Creativity at Conference on Computer Vision and Pattern Recognition (CVPR 2021).

October 2020. Jean Oh co-organized the first Creative AI workshop: Measuring Computational Creativity: Collaboratively Designing Metrics to Evaluate Creative Machines at Inter-Society for Electronic Arts (ISEA 2020).

P. Schaldenbrand and J. Oh.  Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agent. In: Proc. of AAAI Conference on Artificial Intelligence (AAAI). 2021. [Arxiv] [Code repo]

A. Bidgoli, M. L. De Guevara, C. Hsiung, J. Oh, and E. Kang.  Artistic Style in Robotic Painting; a Machine Learning Approach to Learning Brushstroke from Human Artists. In: The 29th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN). 2020. [Arxiv]

Semantic Autonomous Navigation in Unknown Environments

Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro,and Jean Oh. Core Challenges in Embodied Vision-Language Planning, 2021. [ Arxiv:2106.13948]

F. Labelle, X. Lu, N. Kitamura, and J. Oh.  Modular Pretraining for Vision Language Navigation. In: Embodied Vision, Actions & Language (EVAL) Workshop at European Conference on Computer Vision (ECCV). 2020.

J. Tian and J. Oh  Image Captioning with Compositional Neural Module Networks, International Joint Conference on Artificial Intelligence (IJCAI) 2019 [pdf].

T.-H. Lin, T. Bui, D. S. Kim, and J. Oh.  A Multimodal Dialogue System for Conversational Image Editing. In The Second Workshop on Conversational AI at the Thirty-second Conference on Neural Information Processing Systems (NeurIPS), 2018.

S.-R. Shiang, A. Gershman, and J. Oh   A Generalized Model for Multimodal Perception. AAAI Fall Symposium, November 2017 [pdf].

J. Hu, D. Fan, S. Yao, and J. Oh   Answer-Aware Attention on Grounded Question Answering in Images. AAAI Fall Symposium, November 2017 [pdf].

S. Shiang, S. Rosenthal, A. Gershman, J. Carbonell, J. Oh. Vision-Language Fusion for Object Recognition.  In Proc. of AAAI Conference on Artificial Intelligence (AAAI), 2017. [pdf]

J. Hu, J. Oh, A. Gershman. Learning Lexical Entries for Robotic Commands using Crowdsourcing. In Proc. of AAAI Conference on Human Computation (HCOMP), 2016 (short paper).[pdf]

J. Oh, M. Zhu, S. Park, T.M. Howard, M.R. Walter, D. Barber, O. Romero, A. Suppe, L. Navarro-Serment, F. Duvallet, A. Boularias, J. Vinokurov, T. Keegan, R. Dean, C. Lennon, B. Bodt, M. Childers, J. Shi, K. Daniilidis, N. Roy, C. Lebiere, M. Hebert, and A. Stentz. Integrated intelligence for human-robot teams. In Proc. of International Symposium on Experimental Robotics (ISER) 2016 [pdf].

A. Boularias, F. Duvallet, J. Oh, and A. Stentz. Learning Qualitative Spatial Relations for Robotic Navigation. in  Proc. of International Joint Conference on Artificial Intelligence (IJCAI), 2016 [pdf].

J. Oh, L. Navarro-Serment, A. Suppe, A. Stentz, M. Hebert, Inferring door locations from a teammate's trajectory in stealth human-robot team operations. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015 [pdf].

J. Oh, A. Suppe, F. Duvallet, A. Boularias, J. Vinokurov, L. Navarro-Serment, O. Romero, R. Dean, C. Lebiere, M. Hebert, and A. Stentz. Toward mobile robots reasoning like humans. In Proc. of AAAI Conference on Artificial Intelligence (AAAI), 2015. [pdf]

A. Boularias, F. Duvallet, J. Oh, and A. Stentz. Learning to ground spatial relations for outdoor robot navigation. In Proc. of IEEE Conference on Robotics and Automation (ICRA), 2015. [pdf] *Best Cognitive Robotics Paper Award*

 Team pictures
BIG 2021BIG 2020BIG 2018Robot Autonomy, Project Andy team, 2016
RI PhD student
Eagle (Dapeng) ZhaoMSRMilwaukee Tool
Felix LuMEOff World
Xin Shen ME MSAmazon
Kyungdon Joo Postdoc Assistant Professor, UNIST, Korea
Hae-Gon Jeon Postdoc (With Martial Hebert) Assistant Professor, GIST, Korea
Sunghoom Im Visiting scholar Assistant Professor, DGIST, Korea
Kyunghyun Lee Visiting scholar PhD student, KAIST, Korea
Byeong-Uk Lee Visiting scholar PhD student, KAIST, Korea
Jiyoung Ohn Design MS
Jisoo Shon Design MS
Aarati Noronha ME MS Realtime Robotics
Brendon BoltLTI MS LTI PhD student
Xinyu WangLTI MSRiiid Lab.
Andy (Chih-En) Tsai RI MSWaymo
Junjiao TianRI MSGeorgia Tech. PhD student
Yeeho SongRI MSRI PhD student
Satyen RajpalME MS Argo AI
Krishna Toshniwal ME MS
Jagjeet Singh ME MSArgo AI
Shuhan Yang ME MS
Balarama Buddharaju
Brandon Trabucco RISS
Pranav Dheram LTI MSAmazon
Chris Hazard RI MSNREC
Kai-Chi Huang
Sz-Rung ShiangLTI MSBloomberg Research, NYC
Junjie HuLTI MSLTI PhD student
Anirudh VemulaRI MS RI PhD student
Matthew WilsonRISS Summer ScholarUniv. of Utah undergraduate student
Jie-Eun HwangVisiting scholarProfessor, Univ. of Seoul, Korea
Kevin Zhang Summer intern CMU graduate student
Michael Jason Gnanasekar Summer intern Facebook
Sonia Appasamy Summer intern Cornell undergraduate student

Acknowledgement: Our research is funded by U.S. Army Research Lab., Air Force Office of Scientific Research, Defense Advanced Research Projects Agency, DiDi Chuxing, U.S. Army AI Hub, US Army Ground Vehicle Systems Center (GVSC) and Software Engineering Institute (SEI).