Ruta Desai

I am a Research Scientist in Fundamental AI Research (FAIR) at Meta. At FAIR, I am exploring perception and long-horizon planning problems towards embodied agents that can partner with humans. Prior to that, I was a Tech Lead Manager at Meta Reality Labs Research. At Reality Labs, I led an embodied AI team to develop contextual Augmented Reality (AR) systems that can extend their users' capabilities.

I obtained my PhD at the Robotics Institute, Carnegie Mellon University in 2018 working with Stelian Coros and Jim McCann. For my thesis, I devised human-AI systems that enabled casual users to build and program robots — towards increasing accessibility of robotics. In the past, I have also had not-so-brief stints in the area of human motor control and biomechanical simulations with Hartmut Geyer and Jessica K. Hodgins.

Email: rutadesai[at]meta.com

CV  /  Meta AI  /  Scholar

Publications

Embodied AI

Learning Human Preferences Over Robot Behavior as Soft Planning Constraints
Austin Narcomey, Nathan Tsoi, Ruta Desai, and Marynel Vazquez
International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, UAE (2024).                         
Under review
arXiv

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavi Puig*, Eric Undersander*, Andrew Szot*, Mikael Cote*, Ruslan Partsey*, Jimmy Yang*, Ruta Desai*, Alexander Clegg*, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimir Vondrus, Vincent-Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai**, Roozbeh Mottaghi**
International Conference on Learning Representations (ICLR), Vienna, Austria (2024).                         

PDF | Code | Webpage

Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai
International Conference on Machine Learning (ICML), Hawaii, USA (2023).                         

PDF | Code

Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments
Engin Tekin, Elaheh Barati, Nitin Kamra, and Ruta Desai
AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).                         
arXiv | bib

Cross-Domain Imitation Learning via Semantic Skills
Karl Pertsch, Ruta Desai, Franziska Meier, Vikash Kumar, Dhruv Batra, and Akshara Rai
Conference on Robot Learning(CoRL), Aukland, New Zealand (2022).                         

PDF | bib | Video | Webpage

Embodied AI for AR

Pretrained Language Models as Visual Planners for Human Assistance
Dhruvesh Patel, Hamid Engbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai
International Conference on Computer Vision (ICCV), Paris, France (2023).                         

PDF | Code

EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
Rishi Hazra, Brian Chen, Akshara Rai, Nitin Kamra, and Ruta Desai
International Conference on Computer Vision (ICCV), Paris, France (2023).                         

PDF | Code | Webpage

Human-Centered Planning
Yuliang Li, Nitin Kamra, Ruta Desai, and Alon Halevy
arXiv Preprint (2023).                         

arXiv

Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks
Weichao Mao, Ruta Desai, Michael Louis Iuzzolino, and Nitin Kamra
AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).                         

arXiv | bib

Egocentric Scene Context for Human-centric Environment Understanding from Video
Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grauman
Advances in Neural Information Processing Systems (Neurips), New Orleans, USA (2023).                         

PDF | bib | Webpage

Episodic Memory Question Answering
Samyak Datta, Sameer Dharur, Vincent Cartellier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh
The Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA (2022).                         

PDF | bib | Video

How You Move Your Head Tells What You Do: Self-supervised Video Representation Learning with Egocentric Cameras and IMU Sensors
Satoshi Tsutsui, Ruta Desai, and Karl Ridgeway
International Conference on Computer Vision (ICCV), EPIC Workshop (2021).                         

PDF | bib | Video

Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
Benjamin Newman, Kevin Carlberg, and Ruta Desai
arXiv (2020).                         

PDF | bib | Video

Towards Inferring Cognitive State Changes from Pupil Size Variations in Real World Conditions
Naga Venkata Kartheek Medathati, Ruta Desai, and James Hillis
ACM Symposium on Eye Tracking Research and Applications (ETRA), Stuttgart, Germany (2020).                         
PDF | bib

Human-AI Collaborative and Mixed-Initiative Systems

Optimizing the Timing of Intelligent Suggestion in Virtual Reality
Difeng Yu, Ruta Desai, Ting Zhang, Hrvoje Benko, Tanya R. Jonker, and Aakar Gupta
ACM User Interface Software and Technology Symposium (UIST), Bend, USA (2022).                         

PDF | bib

Geppetto: Enabling Semantic Design of Expressive Robot Behaviors
Ruta Desai, Fraser Anderson, Justin Matejka, Stelian Coros, James McCann, George Fitzmaurice, and Tovi Grossman
ACM Conference on Human Factors in Computing Systems (CHI), Glasgow, UK (2019).                         
Best paper award (top 1%) [Details]
PDF | bib | Video | Fastforward | Supplementary (PDF)

Assembly-aware Design of Printable Electromechanical Devices
Ruta Desai, James McCann, and Stelian Coros
ACM User Interface Software and Technology Symposium (UIST), Berlin, Germany (2018).                           
PDF | bib | Video | Fastforward

Skaterbots: Optimization-based Design and Motion Synthesis for Robotic Creatures with Legs and Wheels
Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros
ACM Transactions on Graphics (Proc. ACM SIGGRAPH), Vancouver, Canada (2018).                           

PDF | bib | Video

Automatic Design of Task-specific Robotic Arms
Ruta Desai, Margarita Safonova, Katharina Muelling, and Stelian Coros
ICRA Workshop on Autonomous Robot Design, Brisbane, Australia (2018).                           

arXiv | bib | Slides | Video

Interactive Co-Design of Form and Function for Legged Robots using the Adjoint Method
Ruta Desai, Beichen Li, Ye Yuan, and Stelian Coros
International Conference on Climbing and Walking Robots (CLAWAR), Panama city, Panama (2018).  

Best technical paper (second place)  
arXiv Preprint | Video

Computational Abstractions for Interactive Design of Robotic Devices
Ruta Desai, Ye Yuan, and Stelian Coros
IEEE International Conference on Robotics and Automation (ICRA), Singapore (2017).                          
PDF | bib | Slides | Video | Code
Robot models for fabrication: car, walking robot

Human Motion Modeling, Human Motor Control, Biomechanics

Patents and Articles

Academic Service

Conference Program Committee Chair and Reviewer

• Organizer, CVPR Workshop on Causal and Object-centric Representations for Robotics (2024).
• Reveiwer for IEEE CVPR, IEEE ICCV, ICLR, Neurips, ICML (2023-2024).
• Program committee, ACM UIST 2019, 2020.
• Associate Chair (AC), ACM CHI 2020, 2021.
• Reveiwer for IEEE IROS, IEEE ICRA, ACM TOCHI, ACM UIST, ACM CHI, ACM GI, IEEE WHC (2015-2022).

Talks

• Guest Lecture in Generating Expressiveness in Intelligent Agents and Avatars, University of Florida, 2022.
• DUB Seminar, University of Washington, 2020.
• BID Seminar, University of California, Berkeley, 2019.
• GRASP Seminar, University of Pennsylvania, 2019.

Outreach

Women in Technology

I am always looking to support and contribute towards encouraging women in science and technology. In the past:

• I have led roadshows and volunteered in teaching middle school girls at Technights at CMU. We have smashed computers, fought combats in code, and have even gazed stars through apps!
• I helped organize OurCS 2015, a conference to encourage undergraduate women in research. I was also a part of Google Anita Borg Scholars Alumni Planning committee.

Selected Press

• Silicon Angle, Meta's Habitat 3.0 simulates real-world environments for intelligent AI robot training, 2023.
• Techcrunch, Embodied AI spins a pen and helps clean the living room in new research, 2023.
• Techcrunch, New toolkit makes it easy to drag and drop your own robot, 2017.
• ACM Communications, Robot Design For Dummies, 2017.
• EurekAlert, CMU's interactive tool helps novices and experts make custom robots, 2017.
• NSF, Graduate Student Earns Prestigious Scholarships for Women - Ruta Desai, 2012.
• CMU SCS, Five SCS Students Named Siebel Scholars, 2012.