%
% This is a borrowed LaTeX template file for lecture notes for CS267,
% Applications of Parallel Computing, UCBerkeley EECS Department.
% Now being used for CMU's 10725 Fall 2012 Optimization course
% taught by Geoff Gordon and Ryan Tibshirani.  When preparing 
% LaTeX notes for this class, please use this template.
%
% To familiarize yourself with this template, the body contains
% some examples of its use.  Look them over.  Then you can
% run LaTeX on this file.  After you have LaTeXed this file then
% you can look over the result either by printing it out with
% dvips or using xdvi. "pdflatex template.tex" should also work.
%

\documentclass[twoside]{article}
\setlength{\oddsidemargin}{0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}

%
% ADD PACKAGES here:
%

\usepackage{amsmath,amsfonts,graphicx}

%
% The following commands set up the lecnum (lecture number)
% counter and make various numbering schemes work relative
% to the lecture number.
%
\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf 10-725: Optimization
	\hfill Fall 2012} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1: #2  \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: #3 \hfill Scribes: #4} }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1: #2}{Lecture #1: #2}

   {\bf Note}: {\it LaTeX template courtesy of UC Berkeley EECS dept.}

   {\bf Disclaimer}: {\it These notes have not been subjected to the
   usual scrutiny reserved for formal publications.  They may be distributed
   outside this class only with the permission of the Instructor.}
   \vspace*{4mm}
}
%
% Convention for citations is authors' initials followed by the year.
% For example, to cite a paper by Leighton and Maggs you would type
% \cite{LM89}, and to cite a paper by Strassen you would type \cite{S69}.
% (To avoid bibliography problems, for now we redefine the \cite command.)
% Also commands that create a suitable format for the reference list.
\renewcommand{\cite}[1]{[#1]}
\def\beginrefs{\begin{list}%
        {[\arabic{equation}]}{\usecounter{equation}
         \setlength{\leftmargin}{2.0truecm}\setlength{\labelsep}{0.4truecm}%
         \setlength{\labelwidth}{1.6truecm}}}
\def\endrefs{\end{list}}
\def\bibentry#1{\item[\hbox{[#1]}]}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{SPACE-IN-INCHES}{CAPTION}
\newcommand{\fig}[3]{
			\vspace{#2}
			\begin{center}
			Figure \thelecnum.#1:~#3
			\end{center}
	}
% Use these for theorems, lemmas, proofs, etc.
\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
\newenvironment{proof}{{\bf Proof:}}{\hfill\rule{2mm}{2mm}}

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

\newcommand\E{\mathbb{E}}

\begin{document}
%FILL IN THE RIGHT INFO.
%\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}
\lecture{8}{September 20}{Geoff Gordon/Ryan Tibshirani}{Avinava Dubey}
%\footnotetext{These notes are partially based on those of Nigel Mansell.}

% **** YOUR NOTES GO HERE:

% Some general latex examples and examples making use of the
% macros follow.  
%**** IN GENERAL, BE BRIEF. LONG SCRIBE NOTES, NO MATTER HOW WELL WRITTEN,
%**** ARE NEVER READ BY ANYBODY.
%This lecture's notes illustrate some uses of
%various \LaTeX\ macros.  
%Take a look at this and imitate.

\section{Review of Subgradient Method}

One of the methods to optimize a convex but not necessarily differentiable function is to use subgradient methods. Thus the objective is to
\begin{eqnarray}
\min_{x \in \mathbb{R}^n} f(x)
\end{eqnarray}
where $f(x)$ is convex, but may not be differentiable. The update step after choosing an initial point in $x^{(0)} \in \mathbb{R}^n$ is 
\begin{eqnarray}
x^{(k)} = x^{(k-1)} - t_k.g^{(k-1)}, \ k = 1,2,3,\ldots
\end{eqnarray}
where $g^{(k-1)}$ is a subgradient of the function $f$ at $x^{(k-1)}$. This is similar to gradient descent step with the gradient replaced by sub-gradient.

A major problem with subgradient methods is the rate of convergence which is $O(1/\sqrt{k})$ when the function is Lipschitz on a bounded set containing its minimizer. A bit of the structure of the objective function can be used to get a better convergence rate. In generalized gradient decent the knowledge of certain form of the objective will lead to a faster convergence rate.


\section{Generalized Gradient Descent}

Suppose the objective function $f(x)$ has the following form 
\begin{eqnarray}
f(x) = g(x) + h(x)
\label{eq:decomp}
\end{eqnarray}
such that $g(x)$ is convex and differentiable while $h(x)$ is only convex but not necessarily differentiable. When the function $f(x)$ was differentiable we made quadratic approximation to $f$ around $x$ to get gradient descent update. The update step is of the form $x^{+} = x - t \nabla f(x)$ where $x^{+}$ is the next value of $x$ and $t$ is the step size. To recap the quadratic approximation around $x$ is
\begin{eqnarray}
 f(x) + \nabla f(x)^T(z-x) + \frac{1}{2}(z-x)^T \nabla^2 f(x) (z-x) 
\end{eqnarray}
To get to the gradient update step replace the $\nabla^2f(x)$ by $\frac{1}{t}I$ leading to the following approximation 
\begin{eqnarray}
x^{+} = \arg \min_z f(x) + \nabla f(x)^T(z-x) + \frac{1}{2t}\| z - x \|^2 
\end{eqnarray}

In the case when $f$ is not differentiable but can be decomposed into sum of a convex differentiable function ($g$)  and a convex non-differentiable function ($h$) then $g$ can still be approximated by a quadratic approximation while $h$ can be used as it is to get the following:-
\begin{eqnarray}
\nonumber x^{+} &&= \arg\min_z g(x) + \nabla g(x)^T(z-x) + \frac{1}{2t}\|z-x\|^2 + h(z) \\
\nonumber &&= \arg\min_z \frac{1}{2t}( \|z-x\|^2 +  2t \nabla g(x)^T(z-x) + t^2\|\nabla g(x) \|^2) + g(x) - \frac{2}{t}\|\nabla g(x) \|^2 + h(z) \\
&& = \arg\min_z \frac{1}{2t} \| z-(x-t\nabla g(x)) \|^2 + h(z)\quad \mbox{removing terms independent of $z$}
\end{eqnarray}
The effect of the above update scheme is two fold. The first term keeps the update as close to the gradient update for $g$ while the second term makes $h$ small. 

The above intuition leads to the following method for solving functions represented in \ref{eq:decomp}. 
\begin{enumerate}
	\item Initialize $x^{(0)} \in \mathbb{R}^n$.
	\item Let $x^{+} = x^{(k-1)} - t_k \nabla g (x^{(k-1)})$
	\item Define $\mbox{prox}_t(x) = \arg \min_{z \in \mathbb{R}^n} \frac{1}{2t} \|x-z\|^2 + h(z)$. Then 
	\[ x^{(k)} = \mbox{prox}_{t_k}(x^{+})\]
\end{enumerate}
The update step can also be made to look similar to the update step of gradient descent by writing it as 

\begin{eqnarray}
x^{(k)} && =  x^{(k-1)} - t_k G_{t_k}(x^{(k-1)})
\end{eqnarray}
where $G_t(x)$ is the generalized gradient and is given by
\[ G_t(x) = \frac{x - \mbox{prox}_t(x-t\nabla g(x))}{t} \]

\textbf{Discussions}
\begin{enumerate}
	\item \textit{One optimization step has been replaced by another. So does this help?} Yes since for a number of important functions $h$ the $\mbox{prox}_t$ can be computed analytically.  
	\item \textit{What if $g$ is complicated?} Note that the prox function does not depend on the function $g$. For the update only the value of the gradient of $g$ is required.
\end{enumerate}

\subsection{ISTA}
Lets consider a case when $h(x)$ is $L1$ penalty on $x$ ie $h(x) = \lambda \|x\|_1$ where $\lambda \geq 0$. Then the prox function is defined as 
\begin{eqnarray}
\mbox{prox}_t(x) &=& \arg \min_{z \in \mathbb{R}^n} \frac{1}{2t} \| x - z \|^2 + \lambda \|z\|_1 \\
\nonumber &=& S_{\lambda t}(x) \quad \quad \quad  \mbox{refer to lecture 7 slide 16 }
\end{eqnarray}
where $S_{\lambda t}(x)$ is the soft-thresholding operator.

\[ [S_{\lambda}(x)]_{i} = \left \{ \begin{array}{ll}
x_i - \lambda & \mbox{if } x_i > \lambda \\
0 & \mbox{if } -\lambda \leq x_i \geq \lambda \\
x_i + \lambda & \mbox{if } x_i < -\lambda
\end{array}
\right .
\]

\subsubsection{LASSO}
In lasso the objective function can be written as 
\begin{eqnarray}
 f(x) &=& \frac{1}{2}\|y - Ax \|^2 + \lambda \|x\|_1 \\
\nonumber  &=& g(x) + h(x)
\end{eqnarray}
Where $g(x) = \frac{1}{2}\|y - Ax \|^2$. Thus $\nabla g(x) = -A^T(y-Ax)$. Hence the generalized gradient update is 
\begin{eqnarray}
x^{+} = S_{\lambda t}(x + t A^T(y-Ax))
\end{eqnarray}

The resulting algorithm is faster than sub-gradient method as shown in the figure \ref{fig:ista_sub}.
\begin{figure}[h]
\begin{center}

	\includegraphics[scale=0.4]{ista_vs_sub.png}
	\caption{ISTA vs subgradient decent \label{fig:ista_sub}}
	\end{center}
\end{figure}

\subsection{Convergencce Analysis}
For $f(x) = g(x) + h(x)$, lets assume that 
\begin{itemize}
	\item $g$ is convex, differentiable, $\nabla g$ is Lipshitz continuous with constant $L > 0$.
	\item $h$ is convex, $\mbox{prox}_t(x) = \arg \min_z \{ \frac{\|x-z\|^2}{2t} + h(z)\}$ can be evaluated
\end{itemize}
Then the following holds
\begin{theorem}
 Generalized gradient descent with fixed step size $t \leq \frac{1}{L}$ satisfies 
 \[ f(x^{(k)}) - f(x^*) \leq \frac{\| x^{(0)} - x^* \|^2}{2tk} \] where $x^*$ is the optimal solution.
\end{theorem}

\textbf{Discussions}
\begin{enumerate}
	\item This has the same convergence rate $O(1/k)$ as that of gradient descent but \textit{this counts the number of iteration but not the number of operations}.
	\item Why does generalized gradient descent converge at $O(1/k)$ where as subgradient converges at $O(1/\sqrt{k})$. It is because of the additional knowledge about $\nabla g$ being Lipschitz continuous. 
\end{enumerate}

\begin{proof}
We begin by first showing that 
\[ f(y) \leq g(x) + \nabla g(x)^T (y-x) + \frac{L}{2}\|y-x\|^2 + h(y) \quad \forall x,y \]
Since $\nabla g$ is Lipschitz with constant $L$, $\nabla^2 g \preceq LI$, we have $\forall x,y$ 
\[ (x-y)^T(\nabla^2 g (x) - LI)(x-y) \leq 0\] which implies that 
\[ L\|x-y\|^2 \geq (x-y)^T\nabla^2 g(x)(x-y) \]
By Taylor's expansion 
\begin{eqnarray}
\nonumber g(y) &=& g(x) + \nabla g(x)^T (y-x) + \frac{1}{2}(x-y)^T\nabla^2 g(x)(x-y) \\
\nonumber &\leq & g(x) + \nabla g(x)^T (y-x) + \frac{L}{2}\|y-x\|^2
\end{eqnarray}
therefore 
\begin{eqnarray}
f(y) \leq g(x) + \nabla g(x)^T (y-x) + \frac{L}{2}\|y-x\|^2 + h(y) \quad \forall x,y
\end{eqnarray}
substituting $y = x^+ = x - tG_t(x)$ we have
\begin{eqnarray}
f(x^+) \leq g(x) -t\nabla g(x)^T G_t(x) + \frac{Lt}{2}\|G_t\|^2 + h( x - tG_t(x)) \label{eq:f_uper}
\end{eqnarray}

Now 
\begin{eqnarray}
\nonumber x - tG_t(x) &=& \arg\min_z \frac{1}{2t} \| z - (x-t\nabla g(x))\|^2 + h(z) \\
\nonumber &=& \arg \min_z \nabla g(x)^T (z-x) + \frac{1}{2t}\|z-x\|^2 + h(z)
\end{eqnarray}
This implies that there exist a $v \in \delta h(z)$ such that at minima $\nabla g(x) + \frac{1}{t}(z-x) + v = 0$, but the minimum occurs at $z = x - tG_t(x)$, thus
\begin{eqnarray}
\nonumber \nabla g(x) - G_t(x) + v =0, \quad v \in \delta h(x-tG_t(x)) \\
\Rightarrow G_t(x) - \nabla g(x) \in \delta h(x - tG_t(x))
\end{eqnarray}
Since $h$ is convex,
\begin{eqnarray}
\nonumber h(x) \geq h(x-tG_t(x)) + (G_t(x) - \nabla g(x))^TtG_t(x) \\
h(x-tG_t(x)) \leq h(x) - t(G_t(x) - \nabla g(x))^TG_t(x) \label{eq:h_uper}
\end{eqnarray}

Substituting equation \ref{eq:h_uper} in \ref{eq:f_uper} we get
\begin{eqnarray}
f(x^+) \leq f(x) - (1-\frac{Lt}{2})t\|G_t(x)\|^2
\end{eqnarray}
Since $f$ is convex, $f(x) \leq f(x^*) + G_t(x)^T (x - x^*)$, substituting it above we get
\begin{eqnarray}
\nonumber f(x^+) &\leq & f(x^*) + G_t(x)^T(x-x^*) - (1-\frac{Lt}{2})t\|G_t(x)\|^2 \\
\nonumber & \leq & f(x^*) + G_t(x)^T(x-x^*) -\frac{t}{2}\|G_t(x)\|^2 \quad \mbox{since } t < 1/L \\
&=& f(x^*) + \frac{1}{2t}(\|x-x^*\|^2 - \|x^+ - x^*\|^2)
\end{eqnarray}
Summing over iterations we have 
\begin{eqnarray}
\nonumber \sum_{i=1}^k (f(x^{(i)} - f(x^*) ) &\leq & \frac{1}{2t}(\| x^{(0)} - x^* \|^2 - \| x^{(k)} - x^* \|^2) \\
 &\leq & \frac{1}{2t}\|x^{(0)}  - x^* \|^2 
\end{eqnarray}

Since the difference is non-increasing we have
\begin{eqnarray}
f(x^{(k)}) - f(x^*) \leq \frac{1}{k} \sum_{i=1}^k (f(x^{(i)} - f(x^*) ) \leq  \frac{1}{2tk}\|x^{(0)}  - x^* \|^2 
\end{eqnarray}
\end{proof}

\subsection{Backtracking Line Search }
The procedure for back tracking is similar to gradient descent:
\begin{itemize}

	\item Fix $ 0 < \beta < 1$
	\item Then at each iteration start with $t=1$, and while 
	\[ f(x - tG_t(x)) \geq f(x) - \frac{t}{2}\|G_t(x)\|^2 \] update $ t= \beta t$
\end{itemize}

\begin{theorem}
Generalized gradient descent with backtracking line search satisfies 
\[ f(x^{(k)}) - f(x^*) \leq \frac{\|x^{(0)} - x^* \|^2}{2t_{min}k} \] where $t_{min} = \min \{ 1, \beta/L\}$
\end{theorem}

\subsection{ Example: Matrix Completion}
Given matrix $A,$ $m \times n $, only some entries are observed $A_{ij},\ (i,j) \in \Omega$, the objective is to fill in the missing entries. This can be used to predict user preferences such as user rating for unseen movies. The objective is to 
\begin{eqnarray}
\min_{X \in \mathbb{R}^{m \times m}} \frac{1}{2} \sum_{(i,j) \in \Omega} (A_{ij} - X_{ij})^2 + \lambda \|X\|_*
\end{eqnarray}
where $\|X\|_*$ is the nuclear norm of $X$ and is given by \[ \|X\|_* = \sum_{i=1}^r \sigma_i(X)\] where $r = rank(X)$ and $\sigma_i$ is the $i$th singular value.

To solve this first lets define a projection operator onto the observed set 
\[ [P_{\Omega}(X)]_{ij} = \left \{ \begin{array}{ l l}
X_{ij} & (i,j) \in \Omega \\
0 & \mbox{otherwise}
\end{array}
\right .
\]
The objective is to minimize the following function 
\[ f(X) = \frac{1}{2}\|P_{\Omega}(A) - P_{\Omega}(X) \|^2_F + \lambda \|X\|_* \] which has the same form as $g(x) + h(x)$. Now projection function is convex and the Frobenius norm is differentiable and convex and the nuclear norm is convex but not differentiable. Here we can apply generalized gradient descent. The gradient is $\nabla g(X) = - ( P_{\Omega}(A) - P_{\Omega}(X) )$ and the prox function is 
\[ \mbox{prox}_t(X) = \arg \min_{Z \in \mathbb{R}^{m \times n}} \frac{1}{2t} \|X - Z \|^2_F + \lambda \|Z\|_* \]
If we select $\mbox{prox}_t(X) = Z$ then $Z$ should satisfy 
\begin{eqnarray}
0 \in Z-X + \lambda t . \delta \|Z\|_* \label{eq:dif_cond}
\end{eqnarray}
 Note that if $Z = U\Sigma V^T$ then \[ \delta \|Z\|_* = \{ UV^T + W: W \in \mathbb{R}^{m \times n}, \|W\| \leq 1, U^TW = 0, WV = 0 \} \]
If we let $Z = U\Sigma_{\lambda} V^T$  then equation \ref{eq:dif_cond} holds, here $X = U\Sigma V^T$ is the SVD and $\Sigma_{\lambda}$ is given by the diagonal matrix 
\[ (\Sigma_\lambda)_{ii} = \max \{ \Sigma_{ii} - \lambda, 0 \} \] This is true because $X - Z = \lambda UV^T \in \delta \|Z\|_*$. Thus the prox function can be written as 
\[ \mbox{prox}_t(X) = S_{\lambda t}(X) = U \Sigma_\lambda V^T \] and the generalized gradient update step is
\[ X^+ = S_{\lambda t} (X + t(P_{\Omega}(A) - P_{\Omega}(X))) \]
since $\|\nabla g(Y) - \nabla g(X) \|_F = \|P_{\Omega}(A) - P_{\Omega}(X)\|_F \leq \|Y - X\|_F$ the Lipschitz constant is $L=1$ thus the step size can be picked as $t = 1$ leading to
\[ X^+ = S_{\lambda}(P_\Omega (A) + X - P_{\Omega}(X)) = S_\lambda (P_\Omega (A ) + P_\Omega^{\perp} (X)) \]
where $P_\Omega(X) + P_\Omega^{\perp} (X) = X$
This is called the \textbf{soft-impute} algorithm for matrix completion.

\textbf{Discussions}
\begin{enumerate}
	\item \textit{Why is this method called "generalized gradient descent"?} Note that the function to be minimized is $f = g + h$ and if we let $h = 0$ its equivalent to gradient descent, if we set $g = 0$ its proximal minimization algorithm and if we set $h = I_C$ where $I_C$ denotes projection onto a set then it becomes projected gradient descent. 
	\item Since all the three methods are specific cases of this they have $O(1/k)$ convergence rate.
\end{enumerate}

\subsection{Projected Gradient Descent}
The problem of minimizing over a closed convex ($ min_{x \in C} g(x) $) set can be shown to be equivalent to $ min_x g(x) + I_C(x) $ where 
\[ I_C(x) = \left \{ \begin{array}{l l}
0 & x \in C \\
\infty & x \notin C
\end{array} 
\right .
\]
The prox function for this problem is
\begin{eqnarray}
\nonumber prox_t(x) &=& \arg \min_z \frac{1}{2t}\|x-z\|^2 + I_C(z) \\
\nonumber &=& \arg \min_{z \in C} \|x - z\|^2 \\
&=& P_C(x) \quad \mbox{ projection operator on $C$}
\end{eqnarray}
Thus the update step is \[ x^+ = P_C(x - t\nabla(x)) \] which implies that it first performs the usual gradient update and then its projected onto the constrained set as shown in the figure \ref{fig:projected} below:-
\begin{figure}[h]
\begin{center}

	\includegraphics[scale=0.4]{projected.png}
	\caption{The gradient step may take the solution out of the set $C$ shown shaded. It is then projected back on the set. \label{fig:projected}}
	\end{center}
\end{figure}

Some sets that are easy to project on are:-
\begin{itemize}
	\item Affine images $C = \{ Ax + b : x \in \mathbb{R}^n \}$
	\item Solution set of linear system $C = \{ x \in \mathbb{R}^n : Ax = b \}$
	\item Nonnegative orthant $ C = \{ x \ in \mathbb{R}^n : x \succeq 0 \}$
	\item Norm balls $C = \{ x \in \mathbb{R}^n : \|x\|_p \leq 1 \}$, for $p = 1,2,\infty$
\end{itemize}
Note that it may be hard to project on arbitrary polyhedron.

\subsection{Proximal Minimization}
If $g = 0 $ then the objective is just \[ \min_x h(x) \] and can be solved by using just the prox update \[ x^+ = \arg \min_z \frac{1}{2t} \|x -z \|^2 + h(z) \] Note that if the prox function can not be solved in closed form then this method is not implementable. 

\textbf{Discussion}
\begin{enumerate}
	\item Generalized gradient descent assumes that the prox can be evaluated exactly. 
	\item There is in general no convergence guarantee if the prox fucntion cannot be calculated exactly.
	\item There are some exceptions to this(eg. partial proximal minimization )
\end{enumerate}

\section{To be Covered in Next Lecture}
In the next lecture we would deal with cutting edge first order methods that can achieve an optimal rate of $O(1/k^2)$

\section*{References}
\beginrefs
\bibentry{LSU11}{\sc E. Candes}, 
``Lecture Notes for Math 301,''
{\it Stanford University},
Winter 2010-2011.
\bibentry{LSU11}{\sc L. Vandenberghe}, 
``Lecture Notes for EE 236C,''
{\it UCLA},
Spring 2010-2011.

\endrefs

% **** THIS ENDS THE EXAMPLES. DON'T DELETE THE FOLLOWING LINE:

\end{document}





