\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

%\renewcommand{\baselinestretch}{1.25}

\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document header: change the lecture number, date, scribe name, and
% lecture topic.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\lecture{Statistical Techniques in Robotics (16-831, F10)}
{Lecture\#10 (Thursday September 23)}
{Lecturer: Drew Bagnell}
{Scribe:Anca Dr\u{a}gan \footnotemark}
{General Weighted Majority, Online Learning as Online Optimization}

% Be sure to acknowledge all previous scribes.
\footnotetext{Some content adapted from previous scribes: Jared Goerner.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document body: Scribe your notes here.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{General Loss Function Weighted Majority}

\subsection{Recap}
In the previous lectures, we have developed on the rather philosophical statement that induction is a provably near-optimal strategy. More precisely, the predictions as a function of time come near the optimal strategy.

What this means in fact is not that we can prove any bounds on the difference between reality and our predictions (loss), but rather that we can make the regret of our predictions small enough: we can make the difference between our loss and the loss of the best expert (in retrospect) small.

\begin{equation}
 R = \sum_{t}l_t(alg)-l_t(e^*)
\end{equation}

To find such an algorithm with small regret, we have introduced ``Weighted Majority'', a method that penalizes experts that made incorrect predictions. By a simple analysis, we were able to show that the number of mistakes this method makes is bounded by:
\begin{equation}
 m\leq 2.4(m^*+\log N)
\end{equation}
 where $N$ is the number of experts and $m^*$ is the number of mistakes the best expert makes over the entire time horizon.


\subsection{Weighted Majority with a General Loss Function}
The classical ``Weighted Majority'' mentioned above uses a binary loss function: $l\in\{0,1\}$. What follows is a strict generalization of this, where we allow a more flexible form of the loss function. 

\begin{enumerate}
 \item Set $w_i^0=0$.
 \item At time $t$, predict expert $e_i$ in proportion to its weight, $w_i$.
 \item Adjust the weight of experts: $w_i^{t+1}=w_i^t e^{-\epsilon l_t(e_i)}, \forall i$, with $l_t(i)\in\{0,1\}$
\end{enumerate}

This algorithm has many applications: computational geometry, Adaboost (where the experts are the data points), and even Yao's XOR lemma (see \cite{yao} for more details) in complexity theory.

The bound on the regret of this algorithm becomes
\begin{equation}
\label{eq:regbound}
 E[R]\leq \epsilon \sum_t l_t(e^*) + \frac{1}{\epsilon}\ln N
\end{equation}

Ideally, we would like this algorithm to be what is called ``No Regret'', defined as the average regret over time converging to 0:
\begin{equation}
\label{eq:noreg}
 \frac{R_T}{T} \rightarrow 0
\end{equation}
To do so, we need to make sure that $\epsilon(T)$ decays in such a way that the regret grows less than linearly as a function of $T$. Since $l(e^*)\leq 1$ by definition, we have that $\sum_t l_t(e^*) \in O(T)$. Therefore, applying this to (\ref{eq:regbound}), we get:

\begin{equation}
 E[R]\in O(\epsilon T + \frac{1}{\epsilon} \ln N)
\end{equation}

Setting $\epsilon=\frac{1}{\sqrt{T}}$, we get that
\begin{equation}
 E[R]\in O(\sqrt{T} + \sqrt{T} \ln N)
\end{equation} 
Not only does this grow less than linearly as required by (\ref{eq:noreg}) for obtaining No Regret, but it also makes the ration approach 0 the fastest (up to constants).

Note: \emph{This is the point where Drew says how this algorithm can solve any problem in the universe. For more information on General Weighted Majority, refer to the original paper by Arora et.al\cite{genWM}.}
 


\subsection{Example: Linear Classifiers}
The general form of a binary linear classifier is
$$\theta^Tf \geq 0  \Rightarrow 1  $$
$$\theta^Tf < 0  \Rightarrow -1  $$

Here, $f$ is a feature vector and $\theta$ is the weight vector. Like any other problem, we can come up with a weight vector using Weighted Majority: simply set the experts to be sampled $\theta$s, and the loss to be whether the linear classifier using $\theta$ as a weight vector made the right binary prediction. The time steps in Weighted Majority become the data points. 

The problem with this approach is that we need to discretize this $n$-dimensional space (where $n$ is the number of features), which requires having $O(e^n)$ experts. This makes the algorithm very impractical.

On the other hand, the approach allows us to derive useful bounds on linear classifiers, by looking at the scaling in terms of the number of features. Plugging in the number of experts into (\ref{eq:regbound}), we get that
\begin{equation}
E[R]\leq \epsilon \sum_t l_t(e^*) + \frac{1}{\epsilon}\ln e^n = \sum_t l_t(e^*) + \frac{1}{\epsilon} n
\end{equation}
Therefore, the regret scales linearly in the dimension of the feature space. It turns out it is generally true that we need $O(n)$ samples for a linear classifier. (When introducing the constant it becomes about $10n$).


\section{Online Learning as Online Optimization}

Motivation: we can solve convex problems efficiently. 

\subsection{What are Convex Sets and Functions?}
A convex set is a set such that any linear combination of two points in the set is also in the set: if $A\in C,B\in C$, then
\begin{equation}
\label{eq:consets}
 \theta A + (1-\theta)B \in C
\end{equation}

 
For example, a disc in 2D is convex, but a circle isn't. Likewise, a star-shaped set would not be convex, since if you picked a point in one star tip and a point in another star tip, not all of the linear combinations of those two points would be in the set. An interesting property of convex sets is that their intersection is convex.


Convex functions are the functions for which the epigraph (the area above the curve) is a convex set. Intuitively, you can think about it as if the function will hold water if it was poured in from the top of the graph. The property corresponding to (\ref{eq:consets}) for functions is:
\begin{equation}
 f(A)\theta + (1-\theta f(B) \geq f(\theta A + (1-\theta)B)
\end{equation}
 

This directly generalizes to Jensen's inequality:

\[if\sum_i{{\theta }_i=1}\] 

\[then\ f\left({\theta }_1x_1+\ \dots +\ {\theta }_nx_n\right)\le \ \sum_i{{\theta }_if(x_i)}\] 


\subsection{Subgradients}  
Convex functions have subgradients at every point in their domain. A subgradient $\nabla f(x)$ is a subgradient at $x$ if it is a line s.t. the entire function is above it:
\begin{equation}
 f(y) \geq f(x) + \nabla f(x)^T (y-x),\forall y
\end{equation}

This property simply re-states that any point in the domain of the function, the value of the function at that point needs to be higher than the value of the subgradient line at the same point.

If a function is differentiable at a point, then it has a unique subgradient at that point. Furthermore, convex functions are the max over all subgradients. This is an interesting property that will be used later in the class, because the maximum of convex functions is convex. This is an equivalent statement to the one about the intersection of convex sets, because the maximum of two functions is a function whose epigraph is the intersection of the epigraphs of the two original functions.


\subsection{The Online Convex Programming Problem - Intro}
Online Convex Programming was proposed by Martin Zinkevich\cite{oco} in 2003. It is framed in the same context of time steps, loss function, experts and weighted majority, preserving all the same qualities form WM, while being computationally feasible.

The idea is that the experts are elements of some convex set, and that the loss at time $t$ is convex over the set of experts and thus has a subgradient. At every time step, we need to predict an expert $x_t$ and receive the loss $l_t(x_t)$ and $\nabla l_t(x_t)$.

Example: for the case of the linear classifier, where the experts are $x_t=\theta_t$ in some convex set, the loss function could be
$$l_t(\theta_t) = (\theta_t^Tf_t-y_t)^2$$
where $y_t$ is the actual label for the data point $f_t$, in $\{-1,1\}$. This loss is convex and is in fact a parabola in terms of $\theta_t$. 

The next lecture will formalize the Online Convex Programming Problem better, and explain its applications to No Regret Portfolio creation.


\begin{thebibliography}{3}
\bibitem{genWM}
Sanjeev Arora, Elad Hazan, and Satyen Kale, ``The multiplicative
weights update method: A meta algorithm and its applications.''
Technical report, The Princeton University

\bibitem{yao}
O Goldreich, N Nisan, A Wigderson, ``On Yao's XOR-lemma''

\bibitem{oco}
Martin Zinkevich, ``Online Convex Programming
and Generalized Innitesimal Gradient Ascent'', ICML 2003
\end{thebibliography}
\end{document}