\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

%\renewcommand{\baselinestretch}{1.25}

\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document header: change the lecture number, date, scribe name, and
% lecture topic.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\lecture{Statistical Techniques in Robotics (16-831, F10)}
{Lecture\#12 (Thursday September 30)}
{Lecturer: Drew Bagnell}
{Scribe:Heather L. Jones \footnotemark}
{Random fields for Ladar data and estimation}

% Be sure to acknowledge all previous scribes.
\footnotetext{Some content adapted from previous scribe: Javier Hernandez Rivera}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document body: Scribe your notes here.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Portfolio Optimization Continued - No Regret Portfolio}

Note: Do not actually use the investment algorithm we discussed in the last lecture - you will lose money!

Recall the algorithm from last time:
Start with
\begin{equation}
w_0 \leftarrow \frac{1}{n}
\end{equation}

Then
\begin{equation}
w_{t+1} \leftarrow Proj[w_t + \frac{\alpha r^t}{w_t^T r_t}]
\end{equation}

How can this go wrong?

1. We made a no junk bond assumption
\begin{equation}
%r_{min} \leq r_t * \sum{r_{max}}
r_{min} \leq r_t  \leq r_{max}
\end{equation}
This assumption fails if there is a catastrophic market crash.

2. How do you do the projection?

If $w^i \leq 0$, then $w^i = 0$.  Now everything doesn't sum to 1.  To do the projection correctly, sort the weights,
smallest to largest.  Then look at $\sum_i{max(w_i - a,0))} = 1$, and find an $a$ such that this is true.

Gain function: $log(w^T r)$ (take the negative to get the loss)

L2 Projection (minimize squared difference between point before and after projection)

\begin{equation}
R \leftarrow \sqrt{G} * \sqrt{T} * \sqrt{D(w_0,w*)}
\end{equation}

3. Transaction costs are not accounted for

4. Moving the market - if you put a lot of money in, you affect the price. 
We had regret $\sum_t{l_t(w_t) - l_t(w*)}$.  If these losses are a function 
of decisions you made, then it doesn't mean a lot if you have small regret 
(you just made the best expert perform worse).  This shows up in other 
robotics applications too: the robot may see something different because
it makes actions based on what it sees.  For example, the robot decides 
that a green thing far away is bad and wants to stay as far away from it
as possible.  Then the robot never learns that maybe green isn't so bad.

All these problems exist in every application of no regret.  You're still minimizing
your regret, but maybe you aren't minimizing what you hoped you were 
minimizing.  So, no regret is cool, but not magic (ok, maybe a little magic).

Why is projection OK?  When you project onto a convex set, you get closer to 
every point in the convex set.

\begin{equation}
R_t \leq \bigtriangledown{ l_t^T(w_t-w*)} \leq \frac{\alpha G}{2}+[D(\tilde{w_t},w*)-D(w_{t-1},w*)]
\end{equation}


\begin{figure}
\centering
\includegraphics[width=5cm,angle=0]{img/StatTechProjectionImage.png}
\caption{Projection onto a convex set}
\label{fig:proj}
\end{figure}

The term $D(\tilde{w_t},w*)-D(w_{t+1},w*)$ is always positive (See Figure \ref{fig:proj}).

Not only does projection not hurt, it could help.  We have to get $D(\tilde{w_t},w*)$ to go down - 
for this we need the L2 distance, not L1.  Even if the set is huge, no regret with respect to 
$D( w_{0} , w* )$. 

$D()$ is the L2 distance. \em Mirror descent \em is for other $D$'s besides $(w_0 - w*)^2$.

$\alpha = \sqrt{\frac{F}{GT}}$, if you don't know the horizon, use little $t$ instead of $T$, $\alpha_t$ instead of a constant $\alpha$.

$R \leq \sqrt{FGT}$

This is a strongly convex function (bounded by a quadratic) - the quadratic has some steepness.

Every strongly convex function can be written as a convex function.

$f(x)=g(x)+\frac{H}{2}*||x||^2$, (with f(x) strong)

$abx(x) + ||x||^2$

The market problem we just did does not have this property.

$\alpha = \frac{1}{Ht}$, and for this kind of function, 
$R \leq \frac{G}{2H}(1+log(T))$, where $\sum{\frac{1}{T}}$ is bounded by $log(T)$ - notice no dependance on F, so regret grows only logarithmically with the number of time steps.

\section{Markov Random Field Applications}
\subsection{Elevation Map from Laser Data}

[Projected slides showing terrain as sensed from a plane]

The laser in this example has issues - sometimes it does not penetrate vegetation, and sometimes it appears to go through the ground.

\begin{figure}
\centering
\includegraphics[width=5cm,angle=0]{img/StatTechSlopeChangeImage.png}
\caption{Change in slope is bad}
\label{fig:slope}
\end{figure}

\begin{figure}[h!t]
\centering
   \includegraphics[width=9cm, angle=-90,trim=0.25in 0.25in 0.25in 0.25in]{img/mrf_depth.pdf}
\caption{Markov Random Field for Range Sensing}
\label{fig:mrfdepth}
\end{figure}

We know that a change in slope is bad for a robot on the ground (See Figure \ref{fig:slope}).  We can represent the ground as an MRF 
(See Figure \ref{fig:mrfdepth}). The blue nodes represent the height of the ground $h_i$, and the orange nodes represent the 
range readings, $z_i$. The potential functions are defined as:
\begin{equation}
\Phi_{ij} = e^{-\lambda (h_i-h_j)^2}
\end{equation}
\begin{equation}
\Psi_i = e^{-\sigma (z_i-h_i)^2}
\end{equation}
where $\lambda$ is a constant weight placed on the depth measurements.

\begin{equation}
Pr(h) = \frac{1}{Z}\prod_i{\Psi_i}\prod_{i,j \in neighboring pairs}{\Psi_{ij}}
\end{equation}

With Z as the normalizing factor.

Today: $argmax_h Pr(h|z)$

Previously: $h_{sample}$ \textasciitilde{} $Pr(h|z), E[h_i]$

These two need not agree, but they will here because for a Gaussian distribution, mean = mode.

There is also a third thing (which we won't ever do): $argmax_{h_i} Pr(h_i|z)$, (this is the max marginal distribution)

Now take the log of today's formulation:

$log(Pr(h)) = C - \lambda \sum{(h_i-h_j)^2}-\sigma_i \sum{(z_i - h_i)^2}$

This is convex.

$\bigtriangledown_{l_t}^i = -2 \lambda \sum_{j \in neighbor i}{(h_i - h_j)} - 2 \sigma (z_i - h_i)$, 
where the superscript i indicates that the gradient is with respect to $h_i$.

After applying this, the terrain map [on projected slide] looks smoother.

\subsection{Bookshelf with Image and Laser (Depth) Data}

The trick in this application is to realize that the image is higher resolution than the depth data.  
We can use essentially the same MRF model as in Figure \ref{fig:mrfdepth}, but now the $z_t$ nodes only
connect to every third (or tenth, etc.) $h_i$ node, but there is a camera pixel for each $h_i$ node with color $c_i$.

From the equation we had before:
\begin{equation}
\Phi_{ij} = e^{-\lambda (h_i-h_j)^2}
\end{equation}

We now set $\lambda^{ij} = (constant)*e^{-D(c_i,c_j)}$.

\subsection{Terrain Mapping for a Ground Robot}

\begin{figure}[h!t]
\centering
   \includegraphics[width=9cm, angle=-90,trim=0.25in 0.25in 0.25in 0.25in]{img/mrf_terrain.pdf}
\caption{Markov Random Field for Terrain Mapping}
\label{fig:mrfterrain}
\end{figure}

The objective in this problem is to estimate the height of the terrain observed with range sensors. Let's assume the set of heights can be described by a function $Z = f(x,y)$ where $Z=0$ corresponds to the ground level. Figure \ref{fig:mrfterrain} illustrates a MRF to solve this problem. Note that in this case, $Z$ is defined in 1D.

Similarly to the previous problem, we can define the potential functions as:
\begin{equation}
\Psi = e^{-(\widetilde{D}_i-D_i)^2}
\end{equation}
\begin{equation}
\Phi = e^{-w_{ij} (D_i-D_j)^2}
\end{equation}

Moreover, we have the set of constraints $\forall D_i \leq \text{Ray height}_i$ that require a projection function to be satisfied. Note that the set of solutions is convex because it is defined with inequalities.


This assumes no reflections (a puddle could make the robot think there was a hole).

In this case, we get new heights at every time step, so we have a convex set that changes at every time step.  This is still OK, because 
the sequence of convex sets keeps getting smaller.
\end{document}
