\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex


\providecommand{\abs}[1]{\lvert#1\rvert}
\providecommand{\norm}[1]{\lVert#1\rVert}

\let\oldsqrt\sqrt
% it defines the new \sqrt in terms of the old one
\def\sqrt{\mathpalette\DHLhksqrt}
\def\DHLhksqrt#1#2{%
\setbox0=\hbox{$#1\oldsqrt{#2\,}$}\dimen0=\ht0
\advance\dimen0-0.2\ht0
\setbox2=\hbox{\vrule height\ht0 depth -\dimen0}%
{\box0\lower0.4pt\box2}}




%\renewcommand{\baselinestretch}{1.25}

\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document header: change the lecture number, date, scribe name, and
% lecture topic.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\lecture{Statistical Techniques in Robotics (16-831, F14)}
{Lecture\#14 (Thursday October 16)}
{Lecturer: Drew Bagnell}
{Scribe: Sezal Jain\footnotemark}
{Portfolio Optimization}

% Be sure to acknowledge all previous scribes.
\footnotetext{Notes based on work from previous scribes: Ji Zhang}




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document body: Scribe your notes here.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Portfolio Optimization - No Regret Portfolio}

We want to invest in $n$ different stocks given a set of investment weights $w_i$ s.t. $w_i \in R$, $w_i \geq 0$, and $\sum w_i = 1$. We also know the market returns ratios $r_i = \frac{value^i_{t+1}}{value^i_t}$. So the daily increase in wealth is $w_t^Tr_t$, and the total wealth over time is $\prod_t w^T_t r_t $ We want to maximize $\log\Pi w_t^Tr_t = \sum \log w_t^Tr_t$. We can use use the following algorithm:
\begin{equation}
w_0 \leftarrow \frac{1}{n}
\label{eqn_w0}
\end{equation}
\begin{equation}
w_{t+1} \leftarrow Proj[w_t + \frac{\alpha r_t}{w_t^T r_t}]
\end{equation}

We shall compare the policy to a constantly rebalancing portfolio that maintains a set constant investment ratios. We explain the details in the following.

Here, we control the  $w_t$, the objective function is $r^i_t$ (the return of the $i^{th}$ investment during timestep $t$) and this is no regret with respect to a constant weight $w^\star$ which is the CRP in return (This means it is a constantly rebalanced portfolio, after every timestep, the assets are moved around such that their starting weight each day is the same - it is not the portfolio where you pick the best asset each day from some oracle).

So for each day $w^Tr_t$ is the total return  for that day and your total return at the end is $\prod_t w^T_t r_t $.

We will optimize $\sum \log w^T_t r_t$.  (As an aside, it seems people actual care about money in a "sort of" logarithmic way, but we are really doing this because it is convenient mathematically).

1) On day 1, buy at some arbitrary set of weights as in equation \ref{eqn_w0} where $w_0$ can be set using any a priori knowledge available.

2) We compute interest in $w_t$

3) Compute gain $\log w_t^T v_t$

4) For each stock $i$, compute 
\begin{equation}
w^i_{t+1} \leftarrow Proj[w^i_t + \frac{\alpha r^i_t}{w_t^T r_t}]
\end{equation}

Actual regret bound for the optimal rate
\begin{equation}
R\leq\sqrt{F^2G^2T}
\end{equation}
The optimal learning rate is
\begin{equation}
\alpha = \sqrt{\frac{F}{TG}} 
\end{equation}

Filling in the numbers:

1. What is F?

F is largest distance between points in the set.
In case of a general simplex 
$$F=\sqrt{2}$$

2. What is G?

G is the maximum gradient you can get where gradient is $\frac{r_t}{{w_{t}}^{T}r_{t}}$
If total investment goes to zero i.e $ {w_{t}}^{T}r_{t} \rightarrow 0$, we cannot get the value of $G$. But using the assumption $r_{min} \le r_t \le r_{max}$ (i.e no junk bond assumption)
\begin{equation}
|G|^2\leq \sqrt{{(\frac{r_{max}}{r_{min}})}^2N}
\end{equation}
\begin{equation}
\rightarrow R\leq \sqrt{2{(\frac{r_{max}}{r_{min}})}^2NT}
\end{equation}
\begin{equation}
\alpha= \sqrt{\frac{2}{TN \frac{r_{max}}{r_{min}}}}
\end{equation}
How can this go wrong?

1. We made a no junk bond assumption
\begin{equation}
%r_{min} \leq r_t * \sum{r_{max}}
r_{min} \leq r_t  \leq r_{max}
\end{equation}
This assumption fails if there is a catastrophic market crash, but holds for general cases. Intuitively, we can consider the stock price in an economic system won't go more than twice or less than half of its original price in a single day.

2. How do you do the projection?

If $w^i \leq 0$, then $w^i = 0$.  Now everything doesn't sum to 1.  To do the projection correctly, sort the weights,
smallest to largest.  Then look at $\sum_i{max(w_i - a,0))} = 1$, and find an $a$ such that $\sum w_i = 1$.

This process is called projecting on a simplex, simplex being the weights that sum to one in this case. 
Renormalizing weights is the wrong way to take a projection as geometrically the method explained above gives the closest point in the simplex to the initial weights.

Gain function: $log(w^T r)$ (take the negative to get the loss)

$L_2$ Projection (minimize squared difference between point before and after projection)

\begin{equation}
R \leftarrow \sqrt{G} * \sqrt{T} * \sqrt{D(w_0,w*)}
\end{equation}

3. Transaction costs are not accounted for. The assumption is generally true for big investigation such that transaction cost is neglectable.

4. Moving the market - if you put a lot of money in, you affect the price. 
We had regret $\sum_t{l_t(w_t) - l_t(w*)}$.  If these losses are a function 
of decisions you made, then it doesn't mean a lot if you have small regret 
(you just made the best expert perform worse).  For example, if one wants to 
buy fifty percent of the stock of a company, the stock of that company will 
correspondingly go higher, and as a result, the person won't be able to buy 
with the same amount of money. This shows up in other 
robotics applications too: the robot may see something different because
it makes actions based on what it sees.  For example, the robot decides 
that a green thing far away is bad and wants to stay as far away from it
as possible.  Then the robot never learns that maybe green isn't so bad.

All these problems exist in every application of no regret.  You're still minimizing
your regret, but maybe you aren't minimizing what you hoped you were 
minimizing.  So, no regret is cool, but not magic (ok, maybe a little magic).

Why is projection OK?  When you project onto a convex set, you get closer to 
every point in the convex set.

\begin{equation}
R_t \leq \bigtriangledown{ l_t^T(w_t-w*)} \leq \frac{\alpha G}{2}+[D(\tilde{w_t},w*)-D(w_{t-1},w*)]
\end{equation}


\begin{figure}
\centering
\includegraphics[width=5cm,angle=0, bb=0 0 171 189]{img/StatTechProjectionImage.png}
\caption{Projection onto a convex set}
\label{fig:proj}
\end{figure}

The term $D(\tilde{w_t},w*)-D(w_{t+1},w*)$ is always positive (See Figure \ref{fig:proj}).

Not only does projection not hurt, it could help.  We have to get $D(\tilde{w_t},w*)$ to go down - 
for this we need the $L_2$ distance, not $L_1$.  Even if the set is huge, no regret with respect to 
$D( w_{0} , w* )$. 

$D()$ is the $L_2$ distance. \em Mirror descent \em is for other $D$'s besides $(w_0 - w*)^2$.

$\alpha = \sqrt{\frac{F}{GT}}$, if you don't know the horizon, use little $t$ instead of $T$, $\alpha_t$ instead of a constant $\alpha$.

$R \leq \sqrt{FGT}$

This is a strongly convex function (bounded by a quadratic) - the quadratic has some steepness.

Every strongly convex function can be written as a convex function.

$f(x)=g(x)+\frac{H}{2}*||x||^2$, (with f(x) strong)

$abx(x) + ||x||^2$

The market problem we just did does not have this property.

$\alpha = \frac{1}{Ht}$, and for this kind of function, 
$R \leq \frac{G}{2H}(1+log(T))$, where $\sum{\frac{1}{T}}$ is bounded by $log(T)$ - notice no dependance on F, so regret grows only logarithmically with the number of time steps.

\section{Applications}
\subsection{Elevation Map from Laser Data}

[Projected slides showing terrain as sensed from a plane]

The laser in this example has issues - sometimes it does not penetrate vegetation, and sometimes it appears to go through the ground.

\begin{figure}
\centering
\includegraphics[width=5cm,angle=0, bb=0 0 138 91]{img/StatTechSlopeChangeImage.png}
\caption{Change in slope is bad}
\label{fig:slope}
\end{figure}

\begin{figure}[h!t]
\centering
   \includegraphics[width=9cm, angle=-90,trim=0.25in 0.25in 0.25in 0.25in, bb=0 0 612 792]{img/mrf_depth.pdf}
\caption{Markov Random Field for Range Sensing}
\label{fig:mrfdepth}
\end{figure}

We know that a change in slope is bad for a robot on the ground (See Figure \ref{fig:slope}).  We can represent the ground as an MRF 
(See Figure \ref{fig:mrfdepth}). The blue nodes represent the height of the ground $h_i$, and the orange nodes represent the 
range readings, $z_i$. The potential functions are defined as:
\begin{equation}
\Phi_{ij} = e^{-\lambda (h_i-h_j)^2}
\end{equation}
\begin{equation}
\Psi_i = e^{-\sigma (z_i-h_i)^2}
\end{equation}
where $\lambda$ is a constant weight placed on the depth measurements.

\begin{equation}
Pr(h) = \frac{1}{Z}\prod_i{\Psi_i}\prod_{i,j \in neighboring pairs}{\Psi_{ij}}
\end{equation}

With Z as the normalizing factor.

Today: $argmax_h Pr(h|z)$

Previously: $h_{sample}$ \textasciitilde{} $Pr(h|z), E[h_i]$

These two need not agree, but they will here because for a Gaussian distribution, mean = mode.

There is also a third thing (which we won't ever do): $argmax_{h_i} Pr(h_i|z)$, (this is the max marginal distribution)

Now take the log of today's formulation:

$log(Pr(h)) = C - \lambda \sum{(h_i-h_j)^2}-\sigma_i \sum{(z_i - h_i)^2}$

This is convex.

$\bigtriangledown_{l_t}^i = -2 \lambda \sum_{j \in neighbor i}{(h_i - h_j)} - 2 \sigma (z_i - h_i)$, 
where the superscript i indicates that the gradient is with respect to $h_i$.

After applying this, the terrain map [on projected slide] looks smoother.

An alternative way of writing this is:

\[
\min_{\bar{x}} ( -\log P(\bar{x} | observations)) \rightarrow \min_{\bar{x}} \sum_i [ \gamma (x_i - h_{i,min})^2 + \lambda \sum_{j \in Neighbors} (\abs{x_i - x_j} ) ] \]

\[ \rightarrow ( \sum_i 2 \gamma(x_i-h_{i,min}) + \lambda \sum_j (sign(x_i - x_j))(\gamma-\alpha) \]

where \[
 sign(z) = \begin{cases}
        1  & $if $z \ge 0\\
        -1 & $if $z < 0
        \end{cases}
\] 

\subsection{Generating Super Resolved Range Images}

This is based on the paper An Application of Markov Random Fields to Range Sensing by James Diebel and Sebastian Thrun.

In this problem, we have a color image and a depth image, for example taken from an Xtion 3D camera sensor. The depth image as a much lower resolution than the color image. Now the task is to enhance the resolution of the depth image to the same level as the color image. The way of doing this is by assuming that two consecutive pixels in the depth image share the same depth if in the color image they share the same color.
\begin{figure}[h!t]
\centering
   \includegraphics[width=9cm]{img/super_resolution.png}
\caption{Markov Random Field for the depth and color image}
\label{fig:super_res}
\end{figure}

We can use essentially the same MRF model as in Figure \ref{fig:mrfdepth}, but now the $z_i$ nodes only
connect to every third (or tenth, etc.) $h_i$ node, but there is a camera pixel for each $y_i$ node with color $c_i$.

$\delta(color(r_1), color(r_2)) = \sum(p_i - p_j)^2$ for pixels $p_i$ and $p_j$.

The natural depth measurement potential function is an unnormalized gaussian.
\begin{equation}
\Psi = e^{\frac{-\sigma}{2}{(z_i-y_i)}^2}
\end{equation}
From the equation we had before, the depth smoothness prior:
\begin{equation}
\Phi_{ij} = e^{-\lambda_{ij} (y_i-y_j)^2}
\end{equation}
Taking squared or absolute value has a different effect on the outcome.
We now set $\lambda_{ij} = (constant)*e^{-D(c_i,c_j)}$. Here, $D$ reflects how important color is for depth constancy.

Optimize:
\begin{equation}
\max_{x}(log P(y|z,c))
\end{equation}
The gibbs field looks like a giant product of $\Psi$ and $\Phi$ functions.

\textbf{Squared vs Absolute}

What difference does it make if we choose
\begin{equation}
\Phi_{ij} = e^{-\lambda_{ij} |y_i-y_j|}
\end{equation}
While computing the total optimization cost (after taking log):
\begin{equation}
\sum_{i\in N} \frac{\sigma}{2}{(z_i-y_i)}^2+\sum_{i,j\in N} \lambda_{ij}|y_i-y_j|
\end{equation}
Calculating gradients:
\begin{equation}
-\sigma (z_i-y_i) + \lambda_{ij} sign(y_i-y_j)
\end{equation}
\subsection{Terrain Mapping for a Ground Robot}

\begin{figure}[h!t]
\centering
   \includegraphics[width=9cm, angle=-90,trim=0.25in 0.25in 0.25in 0.25in, bb=0 0 612 792]{img/mrf_terrain.pdf}
\caption{Markov Random Field for Terrain Mapping}
\label{fig:mrfterrain}
\end{figure}

The objective in this problem is to estimate the height of the terrain observed with range sensors. Let's assume the set of heights can be described by a function $Z = f(x,y)$ where $Z=0$ corresponds to the ground level. Figure \ref{fig:mrfterrain} illustrates a MRF to solve this problem. Note that in this case, $Z$ is defined in 1D.

Similarly to the previous problem, we can define the potential functions as:
\begin{equation}
\Psi = e^{-(\widetilde{D}_i-D_i)^2}
\end{equation}
\begin{equation}
\Phi = e^{-w_{ij} (D_i-D_j)^2}
\end{equation}

Moreover, we have the set of constraints $\forall D_i \leq \text{Ray height}_i$ that require a projection function to be satisfied. Note that the set of solutions is convex because it is defined with inequalities.


This assumes no reflections (a puddle could make the robot think there was a hole).

In this case, we get new heights at every time step, so we have a convex set that changes at every time step.  This is still OK, because 
the sequence of convex sets keeps getting smaller.

\subsection{Self-supervised Learning}

In this example, a robot vehicle is trying to find a specific target using self-supervised Learning.

\subsection{Self-training in Nutshell}

Here, a robot vehicle navigates in an unknow environment. The robot learns which part of the terrain is drivable and which part is obstacles. The robot looks at the terrain far away and make a guess, then, later on when the robot drives closer, it gets the ground truth. That way the system is trained. The system uses portfolio optimization to adjust weights assigned to different parts of the terrain, and hence converge to a save place to drive on.

\end{document}
