\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage[tight]{subfigure}
\newcommand{\trans}[1]{{#1}^{\ensuremath{\mathsf{T}}}}           % transpose

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}

\begin{document}

\lecture{Statistical Techniques in Robotics (16-831, F09)}{Lecture \#26 (Nov. 19th)}{Lecturer: Drew Bagnell}{Scribe: Eric Whitman}{Gradient Boosting}

\section{Sparsity Misconceptions}
Last time, we said that we should only run Exponentiated Gradient Descent on life if a small number of things are good.

To analyze this claim, we look at the regret bounds:

\underline{Gradient Descent}

$R \le \vert F\vert_2\vert G\vert_2 \sqrt{T}$

\underline{Exponentiated Gradient Descent}

$R \le \vert F\vert_1\vert G\vert_\infty \sqrt{T}\sqrt{\log(d)}$

There also exist algorithms that interpolate between these two and have bounds of the form $\vert F\vert_p\vert G\vert_q$, where $p$ and $q$ are dual norms ($\frac{1}{p}+\frac{1}{q}=1$).

By comparing the bounds, we can debunk two common misconceptions about sparsity

\subsection{Misconception 1}
Claim: $w^*$ must be sparse for exp-grad to make sense.

Look at the example of the continuously rebalanaced portfolio.  We also have $\sum{w} = 1$, which gives the $L_1$ ball.  In the best case for the non-exponential version, we evenly distribute the assets ($w_i  = w_j$).  This gives $\vert w \vert_1 = 1$ and $\vert w \vert_2 = \frac{1}{\sqrt{d}}$

We also know that $\vert G \vert_\infty \ge G_i$, which gives us

$\displaystyle \vert G \vert_2 = \sqrt{\sum_{i = 1}^d G_i^2} \le \sqrt{\sum_{i = 1}^d {\vert G \vert}_\infty^2} = \sqrt{d}{\vert G \vert}_\infty$

We can now substitute and compare the bounds:

$\vert G \vert_\infty\sqrt{\log{d}}$ vs. $\frac{1}{\sqrt{d}}\sqrt{d}{\vert G \vert}_\infty$

This simplifies to $\sqrt{\log{d}}$ vs. $1$, meaning that even in the worst case in terms of $w^*$, exponentiated gradient descent can be (depending on the gradient) only worse by a factor of $\sqrt{\log{d}}$, which is small.


Truth: If weights vary a lot and $\vert G \vert_\infty$ is much different from $\vert G \vert_2$, then exponentiated-gradient is good.

\subsection{Misconception 2}
Claim: If there are lots of features, then do exp-grad.

This is false because the difference in volume between the $L_1$ and the $L_2$ ball is larger in high dimensions.  The result is that $\vert G \vert_2 >> \vert G \vert_\infty$.  An example is that exp-grad is bad for bag-of-words.

\section{Greedy Algorithms}
\subsection{$\epsilon$-Boosting}
\begin{enumerate}
\item $w=0$
\item compute $l$,$\nabla l$
\item identify $\displaystyle i^* = \arg \max_i \vert \nabla l^i \vert$
\item $w_{i^*} -= \epsilon \nabla l^{i^*}$
\end{enumerate}
Repeat steps 2-4 until convergence.

One possible loss function is the squared error loss:

$\displaystyle l = \frac{1}{2}\sum_t(y_t-\trans{w}x_t)^2$

$\displaystyle \nabla l = \sum_t(y_t-\trans{w}x_t)(-x_t)$

$\displaystyle \nabla l_i = -\sum_t(y_t-\trans{w}x_t)(x_t^i)$

\subsection{Orthogonal Matching Pursuit}
The name comes from signal processing.  This is normally used with a squared error loss function.
\begin{enumerate}
\item $w=0$; active set = $\emptyset$
\item compute $l$,$\nabla l$
\item identify $\displaystyle i^* = \arg \max_i \vert \nabla l^i \vert$
\item active set $\cup=i^*$ (add $i^*$ to the active set)
\item minimize over $w$ restricted to active set
\end{enumerate}
Repeat steps 2-5 until convergence.

Squared loss gives orthogonal residuals, which can make this algorithm quite efficient.  It is identical to AdaBoost for squared loss.  $\epsilon$-boosting is almost always better than AdaBoost.

Greedy methods are actually sparse whereas exp-grad is only pseudo-sparse.  This makes greedy methods better computationally.  Usually, most of the computational effort is in step 3: find $\displaystyle i^* = \arg \max_i \vert \nabla l^i \vert$.  For many problems, we have ``oracle'' access to $i^*$, meaning there is some trick to get $i^*$ without checking all possible $i$'s.

\subsection{``Oracle'' Access to $i^*$}

Suppose the actual features on the data, $f$, are 10 dimensional.  Then $x$ might be all small (depth 2) decision trees on f or $x$ might be all hyperplane seperators on $f$.  In either of these cases, $x$ is extremely high dimensional, but we have some other way of finding the best one.

Take the loss function $\displaystyle l = \sum_t(y_t-f(x_t)+\epsilon h(x_t))^2$, where $f(x_t)$ is what we have so far (the sum of the previously added classifiers), and $h(x_t)$ is a new classifier $h(x): x\Longrightarrow \{-1,1\}$.

We can then split the part that does not involve $h$ ($y_t-f(x_t)$) into two parts:

$o_t = \operatorname{sign}(y_t-f(x_t))$ = target output of $h$
$\alpha_t = \vert y_t-f(x_t) \vert$ = weight

If we substitute these in to the loss function, drop the $\epsilon^2$ term, and drop the term that does not depend on $h$, we can see that minimizing the original loss function is equivalent to minimizing:

$\displaystyle \arg \min_h \sum_t\alpha_to_th(x_t)$

This is what we normally do when we train classifiers: we give it correct labelings ($o_t$) and weights ($\alpha_t$), so we know how to minimize this efficiently.

This method is not quite correct because the classifier actually minimizes some other cost function rather than the squared loss that we want.

\subsection{A Different Loss Function}
Rather than squared loss, we can use $l = \sum\max(0,1-y_tf(x_t))$

$l'=(1-y_tf(x_t)>0)?y:0$(In C programming langauge notation.)

Splitting this into 2 parts gives:

$o_t = \operatorname{sign}(l_t')$

$\alpha_t = \vert l_t' \vert$

\end{document}