The goal of my research is to enable robots to autonomously produce behavior that reasons about function and interaction with and around people. I aim to develop a formal understanding of interaction that leads to algorithms which are informed by mathematical models of how people interact with robots, enabling generalization across robot morphologies and interaction modalities.

My thesis focuses specifically on collaborative manipulation, and on motion planners informed by Bayesian models of interaction. I focus on the challenges that interacting and collaborating with a human raises specifically for the way we plan robot motion. Most motion in robotics is purely functional: industrial robots move to package parts, vacuuming robots move to suck dust, and personal robots move to clean up a dirty table. This type of motion is ideal when the robot is performing a task in isolation. When working with a user, however, the robot's motion has an observer, watching and interpreting the motion. As a result, functionality alone is not sufficient: some functional motions are better than others in how surprising they are or how easy they make it to coordinate with the robot. The fact that not all motions are the same, and certain motions are "better", demands a transition from merely functional to optimal motion.

The additional requirement of optimality is, however, a difficult one: optimizers are known to suffer from convergence to high-cost local minima in the complex, high-dimensional spaces induced by manipulation tasks. The first part of my work focuses alleviating this issue by (1) widening good basins of attraction -- taking advantage of the flexibility induced by manipulation tasks (i.e. that they are described by sets of goals rather than single goal configurations), and (2) learning from their previous experiences to initialize the optimizer in a good basin of attraction.

With improvements in how motion is optimized, robots can turn their attention to what should be optimized. When it comes to motion for collaboration and co-existence, what is the right optimization cost? The second part of my work studies the inferences human observers make about robot motion, and proposes motion planners that enable the right inferences, e.g. that produce intent-expressive or legible motion. My collaborators and I also study the generalization of this beyond legible motion, to other channels and interaction modalities.

Finally, sharing the workspace with a human is only one type of collaborative manipulation. I have also been exploring assistive teleoperation as a shared autonomy concept, in which the robot attempts to predict the operator's intent and plan the appropriate motion to assist in completing the task. This is particularly useful in assitive robotics, where the user remains in the loop while benefiting from existing advances in autonomy.

Constrained Trajectory Optimization

Trajectory optimization was seeded in the works of Euler, Lagrange and Hamilton with the principle of least action, and continued with the variational calculus view on optimal control. Traditionally, it has been too computationally expensive to be used in high-dimensional spaces. CHOMP (Covariant Hamiltonian Optimization for Motion Planning) [IJRR] performs functional gradient descent in the Hilbert space of trajectories using a non-Euclidean inner product and signed distance fields to quickly and smoothly bend trajectories out of collision.

CHOMP can also handle hard constraints [ICRA] along the trajectory, like keeping a glass of water upright or maintaining the same relative position of the two hands to carry a box. In particular, goal set constraints -- constraints on the end-point of the trajectory -- free the optimizer from the typical constant end-point assumption and enable it to optimize to any goal in the goal set. By taking advantage of this natural flexibility that occurs in manipulation tasks -- that they are described by an entire region of goals as opposed to a single goal configuration -- the optimizer converges to much better solutions, in a lot of cases making the difference between successful completion of the task and failure. [Short video]

Learning from Experience

Local trajectory optimizers like CHOMP produce locally-optimal solutions, and can get stuck in high-cost local minima. We are working on alleviating this problem by enabling the robot to learn from its past experiences, from what worked and what did not in the past, to inform the optimization process. Although the typical way to do so is to store previously successful solutions in a library of trajectories and recall the one corresponding to the most similar scene, our approach is to learn a lower-dimensional "recipe for success": rather than predicting a full trajectory, predict a set of trajectory attributes that place the optimizer in good basins of attraction. We started by focusing on the goal choice as an important attribute in my ISRR paper, and we are interested in extending this framework to attributes along the trajectory.

Motion Planning with an Observer

What should the robot optimize when collaborating with a human? The theory of action interpretation in psychology talks about two inferences humans make when observing another agent's actions. An "action-to-goal" inference is based on understanding the function of an action, and refers to the observer's ability to infer someone's goal state from his ongoing actions (e.g. because he is walking towards the coffee machine, he will eventually hold a cup of coffee). A "goal-to-action" inference refers to an observer'ss ability to predict the actions that someone will take based on his goal (e.g. because he wants coffee, he will walk over to the coffee machine).

Applied to motion, these lead to two properties: motion is predictable if it matches what the observer expects (i.e. his "goal-to-action" inference), and motion is legible if it expresses intent, enabling the observer's "action-to-goal" inference. Our HRI2013 paper formalizes predictability and legibility in terms of these inferences in opoosing directions, and proposes models for them that enable a robot to evaluate how predictable or legible a motion is. Our RSS2013 paper introduces a trajectory optimization algorithm based on functional gradient descent for generating motion that is legible (but that stays within a trust region of predictability).

Finally, inferences change over time, as the observer watches the robot move. Our HRI2014 paper studies familiarization to robot motion.


Even though we originally created the legibility formalism for motion, it can transform any cost function for efficiency into a legible behavior. This makes it impactful beyond goal-directed motion, and beyond manipulator arms. We used the same formalism to produce deceptive motion [RSS]. My undergraduate student Rachel Holladay has applied the same formalism to pointing gestures [RoMan], and we are working with Stefanie Tellex at Brown to show it applies to natural language.

Learning Motion from Demonstration

We've recently also focused on the problem of learning motion from demonstration. Imagine that user or observer is willing to provide the robot demonstrations of how it wants or expects it to move. The robot then needs to generalize these demonstrations to new situations, such as a new start or goal configuration.

Among serveral tools for addrssing this problem, a commonly used one is a Dynamic Movement Primitive (DMP). Our work introduces a generalization of DMPs. We provide a variational characterization of the problem by formalizing the generalization of a demonstrated trajectory to new endpoints as an optimization over a Hilbert space of trajectories. We find the closest trajectory to the demonstration, in the linear subspaced induced by the new endpoint constraints. Distance (the notion of "closer") is measured by the norm induced by the inner product in the space.

Different choices for the inner product lead to different adaption processes. DMPs implement this optimization for a particular such innter product choice. Our paper on this topic is currently in review.


Handovers too are instances of collaborative manipulation, where actors coordinate in time and space to transfer control of an object. This coordination comprises two processes: the physical process of moving to get close enough to transfer the object, and the cognitive process of exchanging information to guide the transfer.

Our work studies how people hand objects to each other in order to understand their coordination process and the signals and cues that they use and observe with their partners. We use a learner to identify the communication signals used before a handover happens. Based on these studies, we propose a coordination structure for human-robot handovers that considers both the physical and social-cognitive constraints of the interaction.

Assistive Teleoperation

Sharing the workspace with a human is only one way of collaborative manipulation. The robot can also share its autonomy with a human, as is the case with wheelchair mounted arms assisting a spinal cord injury patient.

In assistive teleoperation, the robot helps the user accomplish the desired task, making teleoperation easier and more seamless. Rather than simply executing the user's input, which is hindered by the inadequacies of the interface, the robot attempts to predict the user's intent, and assists in accomplishing it. We proposed a policy blending formalism for this type of shared control, and analyzed its main components: predicting the user's intent, and arbitrating between the two. I am particularly excited about applications of this for assistive, wheelchair-mountable arms. For more details, see RSS2012.

Motion Retargetting

Teleoperation, as well as animation, often requires retargetting human motion or input onto the robot, which has different kinematics. In ROMAN2012, we proposed an online learning method for customizing the retargetting function. Novice users would start teleopeating the robot, pause whenever they found the robot's configuration unsatisfactory, and provide an example of the desired mapping. The robot would use each example to update the function using kernel machine regression, where the width of the kernel would automatically be adjusted.