\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}

\begin{document}

\lecture{Statistical Techniques in Robotics (16-831, F08)}{Lecture \#25 (Nov. 18th)}{Lecturer: Drew Bagnell}{Scribe: Matthew Tesch}{Sparsity}

\section{Sparsity}

Sparsity is important in two situations:
\begin{enumerate}
\item when you are interested in a lot of features and only a few are relevant.
\item when you have a small amount of data relative to the number of features.
\end{enumerate}

Some examples of these situations include:
\begin{enumerate}
\item {\bf Spam Filtering:} The feature vector is all words and all combinations of words, which is huge; it is likely that there are only a few which correlate strongly with spam.
\item {\bf Computer Vision:} There are often a lot of features (filter outputs, patches, etc), but only a few images.
\item {\bf Image Denoising:} When we want to remove lots of noise and recover an underlying image.
\end{enumerate}

\subsection{Minimizing Regret in Sparse Problems: Choosing the right norm}

Regret bound for online gradient descent:
\begin{equation}
R \le \vert F\vert_2\vert G\vert_2 \sqrt(T),
\end{equation}
where
\begin{enumerate}
\item $R$ is the regret,
\item $G$ is the gradient,
\item $F$ is the size of our space, and
\item $T$ is the number of training points.
\end{enumerate}

Many things tie in into the size of $G$, but it is roughly proportional to $sqrt(d)$, where $d$ is the amount of features.  Therefore, we need order $d$ examples if we have $d$ features, so $T$ is fixed.  Also, if $\vert G\vert_2$ is big, we pay a lot of regret.

One way to reduce $R$ is by using a smaller set of hypotheses. Ideally, we want a constant dependence on $d$, but this is unrealistic. Also, we would like to only be dependent on the largest element of $G$; for this we use the infinity norm. And, reducing the size of the hypothesis space is useful as well; to do so, we take the 1 norm of $F$ instead of the 2 norm (see Figure~\ref{fig:norms}).  Note that this reduces the number of possible hypotheses, possibly removing the `best' hypothesis, but the increase in performance is worth it for sparse problems.  After these assumptions, our regret bound is now $R \le \vert F \vert_1\vert G \vert_\infty log(d)$, which is only a logarithmic dependence on $d$.


\begin{figure}
\centering
\includegraphics{norms2}
\caption{By choosing a different norm as a bound on a space, the volume can drastically change.  The shaded blue area is an example of a region that contains points whose two-norms are less than some constant $c$, but one-norms are greater than $c$.  Although in two dimensions, this is not a tremendous area difference, as the dimensionality of the space increases, this area represents a drastic reduction in size of the space of points with norm less than a set value.}
\label{fig:norms}
\end{figure}



\subsection{Where Sparsity Plays a Role: Total-Variation Method}

In this problem, we have images that have accrued noise (through transmission, reproduction, or any other source), and we want to reconstruct the original, denoised picture. One method that could be used is a Markov random field; essentially, a lattice of pixels with neighbors connected (see Figure~\ref{fig:lattice}). 


\begin{figure}
\centering
\includegraphics{lattice}
\caption{An example of a Markov random field that represents a noisy image.  The $p_{ij}$ is the observed value of the pixel, while $x_{ij}$ is the value of the pixel in the original image.}
\label{fig:lattice}
\end{figure}

One important assumption is that most real images have a sparse number of changes; this is our prior on the image.  Essentially, in a big grid of pixels, there are regions of constant pixel value, and only at the borders of these regions do the pixel values change; i.e. we assume sparsely-piecewise constant input images.

Therefore, to use this assumption to obtain the original image, we penalize changes in neighboring pixel values. However, we want large changes to not be that much worse than small changes, so we penalize the 1-norm $\vert\Delta x\vert_1$ instead of the two norm.

To complete the algorithm, we use our input image as evidence: since this provides a good estimate of the desired image, we penalize deviations from this image with a 2-norm: $\sum_i(p_i-x_i)^2$.

This forms the basis of total variation denoising.  In this algorithm, we simultaneously minimize the 1 norm between our estimates of original pixels and the 2 norm between our observation and our estimated original image.  In other words, we minimize $\sum_{i} (p_{i}-x_{i})^2 + \lambda\vert\Delta x\vert_1$.  A high value for $\lambda$ emphasizes smoothing; a low value for $\lambda$ emphasizes keeping original image.

Simplest solution: compute subgradient, do gradient descent, iterate, get answer.

\subsection{The moral of the story:}

If you want sparsh-ish solutions, choose the $L_1$ norm, not $L_2$.  The $L_2$ norm penalizes large deviations much more thatn the $L_1$, and so produces a smoothing effect.  The $L_1$ norm allows a sparse number of larger shifts, instead.

\subsection{Another Method for Sparsity: Exponentiated Gradient Descent}

The basic Exponentiated Gradient Descent algorithm is as follows:

Let $\alpha_t$ be step size.  Then, for each gradient step:
\begin{enumerate}
\item Predict $w_t$.
\item Recieve loss $l_t$ and $\nabla l_t$.
\item Use one of the following multiplicative update rules: 
\begin{enumerate}
\item $w_{t+1} = w_t(1-\alpha_t\nabla l_t)$.
\item $w_{t+1} = w_te^{-\alpha_t\nabla l_t}$.
\item $w_{t+1} = w_t(1-\nabla l_t)^{\alpha_t}$.
\end{enumerate}
\item Optional normalization step (usually used for probability distributions): $w_{t+1} = w_{t+1}/z$, where $z$ is sum of the weights.
\end{enumerate}

Although three are listed, they are not equally common; the third almost never gets used.  Also, note that there is a `matlab dot-multiply' in these updates (i.e. element-wise vector multiplication), and that the exponentiation is element-wise as well.

Some sanity checks, or `does this algorithm behave as we would like?':
\begin{enumerate}
\item Grad = 0 $\rightarrow$ same weights.
\item Our update move is within 90 degrees of the `correct' direction (the direction of the gradient).  In other words, $\langle w_{t+1}-w_t,-\nabla l_t\rangle$ should be positive.  This is easy to test for at least the first multiplicative update rule above.  We plug in for $w_{t+1}$ to get: $\langle w_t - w_t-w_t\alpha_t\nabla l_t,-\nabla l_t\rangle >0$. Since the left side of the inner product is just a weighted version of the right, we ensure positive correlation.
\end{enumerate}

One way to think of exponentiated gradient is that it is just like grad. descent, except with 'distance' replaced by 'KL-divergence'.  Also, we can bound the regret as $R \le k \vert F \vert_1 \vert G \vert_\infty \sqrt{\log d T}$ where $k$ is a constant.

The advantage to this multiplicative approach is that irrelevant features are killed really fast.  One disadvantage is that we can only have positive weights; if we have features that might be anti-correlated, we can double the feature vector by adding a negative of each feature.  Another problem with this algorithm is that every feature must be about at the same scale; this way the gradient is about the same magnitude for each.  Large differences is scale can cause domination of the gradient by a single feature.

\subsection{The problem in document recognition}

The one-norm encourages high weights for a small number of features; low weights for high number of features.  This never helps on text; almost always makes it worse. Feature vector is REALLY sparse, so adding thousands of features only slightly increases $\vert G \vert_\infty$ or $\vert G \vert_2$; there is not much difference between the two. The regular gradient rule works just fine and considers more experts in this case, and is a better choice because of this.

\subsection{Winnow}
For the binary case, winnow is just the exponentiated gradient with support vector loss (the loss without margin for true winnow).

\section{Review:}
Two cases of 'sparsity':
\begin{itemize}
\item {\bf Subject identification from text:} the input is sparse, the weight vector isn't necessarily
\item {\bf TV denoising:} the weight vector is sparse, but the input is dense
\end{itemize}


Also, in the denoising problems, we are learning a basis, then we use it for classify later if desired.  Important: we are not classifying original input image, but a set of weights that help you reconstruct original input image.

Overall, exponentiated gradient is good for sparse solution; bad when input is sparse, b/c we are constraining ourselves to small hypothesis small, even though 2-norm approximates infinity norm.
\end{document}
