\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}
\usepackage{algorithmic}
\usepackage{algorithm}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}

\begin{document}

\lecture{Statistical Techniques in Robotics (16-831, F08)}{Lecture \#11 (Tuesday September 30th)}{Lecturer: Drew Bagnell}{Scribe: Breelyn Kane}{Support Vector Machines and Learning by Constraints}

\section{Support Vector Applications}
\subsection{Little dog walking robot}
		One example of a Support Vector Application is shown through the video of the 'little dog walking robot'.  There are methods which can be applied for choosing where to next place the robot's feet. 

\subsubsection{Goal}

	Find the next footstep that is likely to be successful, given where the foot is put and how all the joints move. In this case, the goal is to choose a lowest cost sequence of footsteps in a greedy way. Each subsequent footstep would minimize the cost. 
	
	The simplest case is a feedback algorithm with coaching. There are two places to go, right or left, and it isn't clear which way is better. A coach would determine which terrain patch is better than another. It would be best to never pick a position where the robot might fall. 
	
		Determining the cost of where to put each foot next turns into a regression problem. Cost functions may represent any set of features. Possible features in this particular example: terrain, or the triangle of support. Two feature vectors represent where to place a foot next. They are given a rank of which is the 'best'. 
	
\subsubsection{Example Cost function}
A learned cost function related to terrain:  Red areas may be expensive, and blue areas are cheap. An area that is steep = expensive, and concave regions, that allow more stability, = cheap. A human 'coach' would pick areas of stability more. 

The terrain could also be a series of steps where a person may determine it's less expensive to step close to the bottom of a step than close to the top of a next step. The flat areas away from steps are most ideal.

\subsection{Sports Examples}
  
  	There are continuous or discrete rankings of teams in sports. In football/basketball rankings are determined to decide which team might be 'better', or more likely to win. Cost features can be generated for two teams that affect their pairwise comparison.
	

\section{Support Vector Framework}

	\begin{itemize}
\item Feedback Mechanism - instead of random choices (best of two), human picks the next best place (best of all)
\item  Greedy Version - (one-step inverse optimal control) how do you get the planner to match the humans next best prediction.
\item  Converging - make minimum of the cost function agree with what human said, the next best step is more informative since it places a constraint on the next possible footstep.
\end{itemize}

\section{Implementation}
 
 For a simple example footsteps are ranked linearly. Think in terms of reward instead of cost. Note: this is picking the highest value item causing a sign flip.
 
 \begin{itemize}
  \item Features of terrain path: vector $x_{i}$,  ith example.
   \item Reward = $w^{T}x_{i}$ , weight on vector of features.
   			
   \item   
    \begin{tabbing} 
   Series of data points \= $w^{T}x_{1}\leq w^{T}x_{2}$ \\
   	\> $w^{T}x_{2}\leq w^{T}x_{3}$ \\
   	\> $w^{T}x_{4}\leq w^{T}x_{4}$ \\
   \>\ldots \\
   \end{tabbing}
      
     There is something wrong with this constraint when $w=0$, since this solution has triviality.      
     \item To offset this add marginality $c$ \\ 
     		$w^{T}x_{1}\leq w^{T}x_{2} + c$
	
  
    \item Maximum Margin Approach \\
     Since the scale of the reward function is arbitrary, the weight vector can be constrained to unit norm: \textbar \textbar$w$\textbar\textbar$^{2}\leq1$ \\
       The maximum margin approach is where you are trying to find the largest c that makes the constraint true. Instead equivalently it's possible to keep c fixed and try to minimize w. \\
       objective function = min\textbar \textbar$w$\textbar\textbar$^{2}$ where constraint is now $w^{T}x_{i}^{1}\leq w^{T}x_{i}^{2}+1$ 
        \begin{tabbing} 
   	 \= $w^{T}x_{i}^{1}\leq w^{T}x_{i}^{2}+1$ \\ 
   	\> $w^{T}x_{i+1}^{1}\leq w^{T}x_{i+1}^{2}+1$ \\ 
	\>\ldots \\
   	\>  $w^{T}x_{T}^{1}\leq w^{T}x_{T}^{2}+1$  \\
   \end{tabbing}

    
    \item Noise is something that can happen when a judgement may be wrong. A limitation of linearity is that the feature sets can't interact or 'affect' each other. The problem goes from constrained to unconstrained when the notion of error is added. \\
    Slack variable $\xi \geq 0$

        \begin{tabbing} 
   	 \= $w^{T}x_{i}^{1}\leq w^{T}x_{i}^{2}+1-\xi_{i}$ \\
   	\> $w^{T}x_{i+1}^{1}\leq w^{T}x_{i+1}^{2}+1-\xi_{i+1}$ \\
	\>\ldots \\
   	\>  $w^{T}x_{T}^{1}\leq w^{T}x_{T}^{2}+1-\xi_{T}$  \\
   \end{tabbing}
   
   \item Objective function becomes: \\  
   	min$\lambda$\textbar \textbar$w$\textbar\textbar$^{2} + \sum_{i=1}^n \xi_{i}$ \\ \\
	where lambda: \\
	 $\lambda \leftarrow$ smaller = larger (growing) set of weights, longer to learn but do better. \\
	$\lambda \leftarrow$ larger = smaller (constrained) set of weights, learn initially faster. \\
	Bigger weight = tiny margin, and smaller weight = bigger margin.
	
	\item If the constraint is met within margin then $\xi_{i} = 0$ \\
	otherwise noise becomes $\xi_{i} \geq w^{T}(x_{i}^{2}-x_{i}^{1})+1$

	\item Revised objective function: \\
	min$\lambda$\textbar \textbar$w$\textbar\textbar$^{2} + \sum_{i=1}^n 
									max(0,w^{T}(x_{i}^{2}-x_{i}^{1})+1) $ \\ \\

	
     \end{itemize} 
    
\subsection{Explanation Of Loss}
In the case of conflicting assumptions the slack variable is either needed or not. If the constraint is met $\xi$ shouldn't affect it so the slack variable equals zero.  Similarly if $w^{T}(x_{i}^{2}-x_{i}^{1})+1> 0$ then the assumption is wrong within the margin and a penalty is paid. 

At every time step a loss is paid where the loss function is:
\begin{equation} 
 \sum_{i=1}^n max(0,w^{T}(x_{i}^{2}-x_{i}^{1})+1) 
 \end{equation}

Since this value is always positive it represents a convex function, and the max of a convex function is convex. 

\subsection{Online Gradient Descent}
At every time step get loss $l^{t} = \lambda$ \textbar\textbar$w$\textbar\textbar$^{2} +  \sum_{i=1}^n max(0,w^{T}(x_{i}^{2}-x_{i}^{1})+1) $

Then the gradient is taken of this, where if the function is differentiable it agrees with the gradient. 
\begin{itemize}
\item When \\
	$w^{T}(x_{i}^{2}-x_{i}^{1}) +1\leq0$, means correctly ranked by a margin of one, 
	the gradient=0 \\
\item else\\
 $ (x_{t}^{2}-x_{t}^{1})$ is the gradient with respect to w.
 \end{itemize}

\section{Summary}
\begin{itemize}
\item At each time step initialize support vector ranker \\
$w_{i}^{0}= \vec{0}$
\item for each sample $i$ predict\\
$w_{T}x_{1}$ , $w_{T}x_{2} $, next two
\item If ranked correctly and makes the margin\\
$w_{t+1} \leftarrow (1-\eta_{t} \lambda)\vec{w_{t}}$,  where $\eta$ = learning rate.\\

\item Else (ranking wrong or they were within one of each other) \\
$w_{t+1} \leftarrow (1-\eta_{t} \lambda)\vec{w_{t}} - \eta_{t}(x_{t}^{wrong prediction} - x_{t}^{should predicted})$ \\ \\
Value on the right is applied to shrink the weight vector. 

\end{itemize}

A Batch algorithm is where the subset of data is randomized. Above was a generalization of the standard support vector machine by using binary classification. Data points are ranked as positive or negative, and progress is observed linearly one data point at a time.

\end{document}
