SCS Special Seminar

  • Assistant Professor
  • Computer Science Department
  • Cornell University

Robot Learning Through Humans

Now we have several types of robots in the human environments --- for example, assistive robots in the healthcare industry, manipulators in co-robotic assembly tasks, and so on. In order for such robots to perform tasks, they need to learn a functional understanding of the environment.  This includes learning about object affordances (i.e., how the objects could be used) and learning about humans--all the way from learning their low-level kinematics to their high-level intents.

In order to do so, I will present new machine learning algorithms that model the rich environment context, and at the same time allow learning the relevant latent factors (e.g., of object affordances, physics, and so on).  Our learning algorithm does so by modeling the probability distributions over the possible graph structures (in contrast to the standard CRFs where the graph structure is fixed).  It can learn the parameters in both unsupervised and co-active learning settings. We then show that these ideas significantly improve the robotic tasks of 3D scene understanding, human activity anticipation, and task and path planning.  Furthermore, we demonstrate that such human-based modeling makes several robots (e.g., assistive robots and grocery checkout robots).


Ashutosh Saxena is an Assistant Professor in the Computer Science department at Cornell University. His research interests include perception and machine learning for robots working in human environments. He received his M.Sc. in 2006 and Ph.D. in 2009 from Stanford University, and his B.Tech. in 2004 from Indian Institute of Technology (IIT) Kanpur. He is the recipient of the National Talent Scholar award in India, Google Faculty award, Alfred P. Sloan Fellowship, Microsoft Faculty Fellowship, and NSF Career award. Ashutosh developed Make3D (, an algorithm that converts a single photograph into a 3D model, which was used by tens of thousands of users to convert pictures to 3D. He has also developed algorithms that enable robots to perform chores such as unloading items from a dishwasher, placing items in a fridge, or checkout items at a grocery store (see His work has received substantial amount of attention in popular press, including the front-page of New York Times, BBC, ABC, New Scientist, Discovery Science, Wired Magazine, and FOX News Studio B. He has won best paper awards in 3DRR, IEEE ACE and R:SS, and was named a co-chair of the IEEE technical committee on robot learning.

Faculty Host: Jeff Schneider


For More Information, Please Contact: