\section{Gradient Descent}

Gradient descent is the grandfather of first order methods.
It simply starts at an initial point and then repeatedly takes a step opposite to the
gradient direction of the function at the current point. The gradient descent algorithm to minimize a function $f(x)$
is as follows:
\begin{tabbing}
\hspace*{.15in} \= \hspace*{.25in} \= \hspace*{.25in} \= \hspace*{.25in} \= \hspace*{.25in} \=\kill
\>{\bf for} $k=0,1,2,\dots$ {\bf do } \\
\>\> $g_k \leftarrow \nabla f(x_k)$ \\
\>\> $x_{k+1} \leftarrow x_k - t_k g_k$ \\
\>{\bf end for} \\
\end{tabbing}


Figure~\ref{fig:grad} shows gradient descent in action. The figure shows a function in two variables, $a$ and $b$.
The contours in the plot are \emph{level sets}, where the value of the function is constant along the contour.
Darker colors indicate lower values of the function. Note that the function has two minima: a local minimum to the right and a global minimum to the left.

\begin{figure}[ht]
    \centering
    \includegraphics[scale=0.5]{grad}
    \caption{Demonstration of gradient descent.}
    \label{fig:grad}
\end{figure}

The figure shows how the algorithm proceeds starting from $x_0$. At each
point the gradient (solid arrow) is computed and a step is taken in the opposite direction (dotted arrow), until the algorithm stops at some point
near the local minimum.

From the description of the algorithm and the figure, we can see that there are several major choices to be made in implementing gradient descent:
the initial point $x_0$, the step size $t_k$, and the stopping condition.
We will not discuss choice of $t_k$ at this time.

\subsection{Where Do We Start?}

Ideally, one would choose $x_0$ to be close to the minimum.
In many situations, however, it is very difficult to make such a guess; we often simply choose zero.

The demonstration in Figure~\ref{fig:grad} demonstrates that for some functions, the choice of initial point may be critical.
For example, our choice of $x_0$ resulted in finding a local minimum.
Had we started at a point like $y_0$, we would have reached the global minimum, the actual correct answer.
We also might be extremely unlucky and start precisely at $z_0$.
At $z_0$ the gradient is 0, because this point is minimum with respect to $a$ and maximum with respect to $b$
(see how the color changes along the vertical and horizontal directions).
Such a point is called a \emph{saddle point}, as it resembles the saddle of a horse.
If we start at $z_0$ the algorithm will immediately terminate.

A common workaround for this problem is to run the algorithm several times from widely dispersed starting points, and then pick the best result.

\subsection{When Do We Stop?}

When we stop iterating depends on the problem setting.
Ideally, we stop when $f(x_k)$ is ``close enough'' to the minimum.
What is ``close enough'' depends on the situation at hand.
% However, there are reasons to stop before that happens, a technique known as \emph{early stopping}:


\subsubsection*{Early Stopping}
\textbf{Held-out Data:} In machine learning and statistics, we sometimes want to employ \emph{early stopping} to avoid overfitting.
We often wish to optimize a sample average based on given data.
Specifically, if $x$ denotes the parameters we are optimizing and $i$ denotes
a data point from a distribution $p(i)$, then we usually try to minimize
$f(x) = \E_{p(i)}[f_i(x)]$
where $f_i$ is the portion of the function corresponding to data point $i$.
If we have an \iid sample $I \sim p(i)$, we can then simply optimize the sample mean
$\hat{f}(x) = \sum_{i \in I} f_i(x) / \abs{I}$.

For example, in linear regression we try to minimize the squared error loss function $f(x) = \E_{a,b} \abs{a \cdot x - b}^2$,
where $a \in \R^p$ is a feature vector, $b \in \R$ is the true output prediction and $x \in \R^p$ is the weight vector we are optimizing.
$f_i(x)$ is then simply $\abs{a_i \cdot x - b_i}^2$.

Fully optimizing this sample mean might overfit the data, resulting in poor generalizability to new data.
A common solution to this problem is to evaluate the objective function on a held-out validation sample, and stop when the objective begins to increase on that sample, a sign of overfitting.
This process is demonstrated in Figure~\ref{fig:gradoverfit}.
This way, early stopping acts as a sort of regularization.

\begin{figure}[ht]
    \centering
    \includegraphics[scale=0.4]{gradoverfit}
    \caption{%
        Early stopping for regularization.
        The figure shows the value of the objective function on training data (blue) and held-out data (red) vs the number of steps.
        The black line indicates where to stop, when the objective function begins to increase on held-out data.%
    }
    \label{fig:gradoverfit}
\end{figure}

\textbf{When it is unnecessary to proceed:} In some other situations, the problem setting limits the effective (or desired) precision of the solution.
For example, if we know that randomness of the data limits our confidence in the solution to two decimal places, there is no point in spending significant amounts of computation optimizing $x$ to seven decimal places.

\subsubsection*{Using Convergence Bounds}
Sometimes we have no reason for early stopping, and wish to ensure that $f(x_k)$ is close enough to the minimum in a principled way.
In this case, to estimate the minimum number of steps, we resort to convergence bounds, which take the form
\begin{equation*}
    K \geq K_f \; (f(x_0) - f(x^*)) \; g\left( \tfrac{1}{\varepsilon} \right),
\end{equation*}
where $K$ is the number of steps,
$K_f$ is a parameter that can be estimated from the properties of the function, 
$g$ is function that also depends on the properties of the function,
$\varepsilon$ is the error tolerance,
and $f(x^*)$ is the true minimum, which can be estimated from duality.
More details on that will be revealed later in the course.

\subsection{Other Issues}
In addition to local minima and the (rare) possibility of getting stuck at a saddle point, there are other issues that should be taken into consideration.

For example, if the step size $t_k$ remains large, it may lead to an oscillatory behavior that does not converge.  

Another issue is that, depending on the function and starting point, gradient descent could continue indefinitely because there is no minimum.
Consider for example minimizing $e^x$: there is no finite $x$ for which $\frac{d}{dx}e^x = 0$.
Other functions could have such asymptotic minima but also a global minimum of lower value; gradient descent, depending on its starting point, might forever chase the asymptote, unaware of the true answer elsewhere in the search space.
