\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}

\begin{document}

\lecture{Statistical Techniques in Robotics (16-831, F08)}{Lecture \#28 (Dec 02, 2008)}{Lecturer: Drew Bagnell}{Scribe: Arvind Rao}{Imitation Learning}

\section{Imitation Learning via Inverse Optimal control}

what is it? : recall SVM for ranking footsteps,
want the robot to recover the \textit{high-level} behavior shown to the robot

Imitation Learning is different from supervised learning 
\begin{itemize}
\item
Decisions have consequences
\item
 Actions and decisions are purposeful
\end{itemize}

Consider the ALVINN system: If you predict with all but $\epsilon$ mistakes the correct direction of car driving -- how bad can $\epsilon$ be?. If the solutions you see are part of the training process, you'll be ok, but if you see something unexpected, then you might make some big mistakes. If the error is correlated with the direction that you are heading before, then the errors compound. Such situations are indicative of the problems that arise when the testing data has nothing to do with the training data.

A potential solution to the above problem is: reasoning over trajectories, i.e. instead of predicting single actions, predict trajectories of actions . Recall that in supervised learning we only take a decision one step into the future.

Question: How do we make a ``sequence" of decisions?. This is a ``planning problem" in robotics, i.e. think of a sequence of motions that takes you to a goal. There are two goals in imitation learning:
\begin{itemize}
\item
Take advantage of the planning algorithm to take a sequence of actions,
\item
Learning tells us how to get the planning algorithm to do what we want.\
\end{itemize}

Observe a sequence of actions to go from point A to point B  and train robot to go on any arbitrary trajectory (as taken by the human).

\textbf{Goal}: come up with algorithm that sets costs so that the human trajectory is the best cost path.

Why is this  \textit{inverse} optimal control?
In typical control problems, you find a planner/controller that minimizes some objective cost function over time. Whereas in "inverse" optimal control, you find what performance metric makes a particular plan (sequence of actions) optimal.

\textbf{Formulation}:
Consider the set $\xi$ of all possible paths between two points A and B. We need that, the cost of the example(expert) path $i$ be less than the cost of any arbitrary path in the set $\xi$. Here, $i \in (1,2,\ldots,N)$, $N$ being the total number of data points over which data is available. Therefore, we have,

$\sum_{i=1}^{N}$ [cost of desired path for data point $i$ - min cost over all paths for data point $i$ ] 

if we instead consider a linear cost function of the form $w^{T}f_j$, as the cost of the $j^{th}$ cell (recall the step planning setup from previous class). Let the optimal plan for data point point $i$ be $\xi_i$, therefore, we now have,

$ min_w [\sum_{j\in \xi_i} w^{T}f_j - min_\xi \sum_{j \in \xi} w^{T}f_j ] $

adding a margin (recall the analogous SVM trick), and looking over all $N$ data points, this becomes,

$ min_w \sum_{i=1}^{N} [\sum_{j\in \xi_i} w^{T}f_j - min_\xi \sum_{j \in \xi} (w^{T}f_j -l_j) ] $

\textbf{Solution}:
Use gradient descent (taking gradient with respect to the weight vector $w$),

$w_{t+1} = w_t - \alpha [ \sum_{j\in \xi} f_j - \sum_{j \in \xi^{*}}  f_j]$ (where $\xi^{*}$ is the argmin of $\sum_{j \in \xi} [w^{T}f_j - l_j]$  above)

A limitation of this approach is that it needs non-negative costs, but this is sidestepped if we use exponentiated gradient.

This is the linear version of max-margin planning approach , a non-linearized version can be constructed using kernelization.

\end{document}
