\documentclass[11pt]{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{epsfig}
\usepackage[tight]{subfigure}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {#1} \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[5]{\handout{#1}{#2}{#3}{#4}{#5}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}

\begin{document}

\lecture{Adaptive Control and Reinforcement Learning (16-899C, S09)}{Lecture \#21 (4/07/09)}{Lecturer: Bagnell }{Scribe: Eric Whitman}{Conservative Policy Iteration (CPI)}


\section{Policy Search by Dynamic Programming}
\begin{itemize}
\item fixed time horizon - rarely what we want - so not often useful
\item performance bounds (tightest known)
\item does not require the bellman equation - can be used to make it faster - overly general
\end{itemize}
\paragraph{Quote about Optimality:}
``An optimal policy hast he property that whatever the initial decision may be, the remaining decisions constitute an optimal policy with regard to ... [the resulting state].''

Implication: You can work backwards from the last time step without considering anything earlier, and without losing optimality.

\subsection{Bounds}
Assume reward $\in [0,1]$

{\bf Define:}

 $V_{\pi_0...\pi_{T-1}} = \frac{1}{T}E[r(s_0)+r(s_1)...+r(s_{T-1})|\pi_0...\pi_{T-1}] \in [0,1]$

Bellman: $V_{\pi_t...\pi_{T-1}}(s) = \frac{1}{T}r(s)+E_{s'}[V_{\pi_t...\pi_{T-1}}(s')]$

where $s'$ is the result of $\pi_t$

$Q_a,\pi_{t+1},\pi{t+2},...,\pi{T-1}(s)$ fix policy at t to choose $a$, then follow the policy from then on.

$\mu_t(s) = p(s_t)$ probability in state, need one distribution for each time step - used to focus the approximation effort

$\mu_t(s)$ is ``similar'' to the $p(s_t|\pi^*)$

$\mu_t$ comes from:
\begin{itemize}
\item trajectory
\item teacher
\item other policy, $\pi^{\operatorname{adequate}}$
\item $\mu$ close to flat
\end{itemize}

\subsection{Procedure}
for each $t \in (T-1, T-2, ... , 1, 0)$

1) $d_t = \operatorname{generate samples}(\pi_{t+1}, ... , \pi_{T-1}, \mu)$

2) $\pi_L = \operatorname{Learn}(d_t)$

\subsubsection{Details on step 1 - generating samples}
Generate samples of the form (policy sequence, $\mu$)

\hspace{0.5in} many ($k$) times:

\hspace{1in} sample $s$ from $\mu$

\hspace{1in} for each action:

\hspace{1.5in} $\tilde{Q}_{s,a}=\operatorname{rollout}(a,\vec{\pi})$

\hspace{1in} return $\{s_i,\tilde{Q}_{s_i,a1},\tilde{Q}_{s_i,a2},\tilde{Q}_{s_i,a3},...\}_{i=1:k}$

\subsubsection{Details on Rollout}
\begin{figure}[h]
\centering
\includegraphics[width=.7\textwidth]{rollout.png}
\caption{Figure 1: Rollout}
\label{fig:rollout}
\end{figure}

Starting from state $s$ at time $t$, perform all available actions.  Run the existing policies from each or the resulting states.  The cumulative rewards starting from state $s$ and performing action $i$ is represented by $\tilde{Q}_{s,a_i}$.

\paragraph{Stochastic Policies}
There is never a reason for a stochastic policy in finite horizon.  This is not necessarily true in the infinite horizon case or in adversarial situations.  For non-adversarial finite time horizon problems, time varying policies are better than stochastic policies.

Argument1:  For the last time step, the expected value of the stochastic policy is the the average of the values of what the policy might do.  The deterministic policy picks the best one, which must be at least as good as the average.  We can inductively make the same argument about the previous time step all the way back to the beginning.

Argument2: (Probablistic method)  We can look at a stochastic policy as taking the state and another input ($\pi(s,\operatorname{rand})$).  A deterministic policy just picks the best $\operatorname{rand}$.  This argument is similar to the mean value theorum.

\subsubsection{Details of step 2 - Learning}
$\operatorname{Learn}(d)$

find $\displaystyle \pi^*_t=\arg\max_{\pi'\in\Pi}E_d[V_{\pi',\pi_{t+1},...,\pi_{T-1}}(s)]$

{\bf Brute Force} \newline
for $\pi'$ in $\Pi$\newline
compute $\displaystyle \Sigma_i \tilde{Q}^i_{s_i,\pi'(s_i)}$\newline
pick biggest

{\bf Learn Classifier($d$)}

Felix: $\displaystyle d'=\{(s_i,\arg\max_a \tilde{Q}_{s_i,a})\}$\newline
return $\pi=\operatorname{classifier}(d')$

doesn't understand importance of particular samples

breaks ties in bad ways, which can be important

{\bf Learn Weighted Classifier($d$)}

$d'=(s_i,\tilde{Q}_{s_i,a^*}-\tilde{Q}_{s_i,a_1},\tilde{Q}_{s_i,a^*}-\tilde{Q}_{s_i,a_2},...)_{i=1:k}$\newline
return $\pi=\operatorname{classifier}(d')$

\subsubsection{Regression}
Regression is harder than classification (worse performance for the same number of examples).

Learn $\operatorname{Regression}(d)$

$d'=(s_i,a_1,\tilde{Q}_{s_i,a_1})$\newline
$\hspace{.5in}(s_i,a_2,\tilde{Q}_{s_i,a_2})$\newline
$\operatorname{reg} = \operatorname{regression}(d')$

$\displaystyle \pi=\arg\max_a \operatorname{reg}(s,a)$

\subsection{Bounds}
$\displaystyle V_{\pi \operatorname{learn}}(s_0)\ge V_{\pi_{ref}}(s_0)-\epsilon\Sigma_t||\frac{\mu_{\operatorname{ref},t}}{\mu_t}||_\infty$

$\epsilon$ is the error of the classifier, $\epsilon = p(\operatorname{wrong})$

$\pi_{\operatorname{ref}}$ is usually $\pi^*$.

implication: err on the side of broadness

Can't get bounds for other methods.
\section{Compare to Value Iteration}

start at last time step \newline
get some reward \newline
learn function $V_{T-1}(s)$ based on samples \newline
samples: ($s$,reward)

effort is O(T).  caches values at each timestep - linear work

goes wrong with error, need to do max over actions $Q_{s,a}$

Q will make errors (function approximator).  These will accumulate as previous steps head towards areas that they thought were good but are really bad

\subsection{Policy Search by Dynamic Programming on the other hand}

Only cache policy - much safer.

However rollout takes more time than looking up values.

Effort is O($T^2$).  A factor of T worse than Value Iteration.

There are non-parametric local minima - local minima in terms of states.

It assumes $\mu$, so it does no global exploration, just local exploration.

``If willing to give up exploration, we can do function approximation.''






\end{document}
