\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{proof}
\usepackage{mathpartir}
\usepackage{url}
\usepackage{listings}
\usepackage{color}
\usepackage{tikz}
\definecolor{eclipseBlue}{RGB}{42,0.0,255}
\definecolor{eclipseGreen}{RGB}{63,127,95}

\lstset {
  basicstyle=\small\ttfamily,
  captionpos=b,
  tabsize=2,
  columns=fixed,
  breaklines=true,
  frame=l,
  numbers=left,
  numberstyle=\small\ttfamily,
  morekeywords= {
    EQUAL, GREATER, LESS, NONE, SOME, abstraction, abstype, and, andalso, array, as, before, bool, case, char, datatype, do, else, end, eqtype, exception, exn, false, fn, fun, functor, handle, if, in, include, infix, infixr, int, let, list, local, nil, nonfix, not, o, of, op, open, option, orelse, overload, print, raise, real, rec, ref, sharing, sig, signature, string, struct, structure, substring, then, true, type, unit, val, vector, where, while, with, withtype, word
  },
  morestring=[b]",
  morecomment=[s]{(*}{*)},
  stringstyle=\color{black},
  identifierstyle=\color{eclipseBlue},
  keywordstyle=\color{red},
  commentstyle=\color{eclipseGreen}
} 

\newtheorem{theorem}{Theorem}

\newcommand{\texttilde}{\url{~}}
\newcommand{\step}{\stackrel{1}{\Longrightarrow}}
\newcommand{\stepk}[1]{\stackrel{#1}{\Longrightarrow}}

\title{Lecture 6 - Asymptotic Complexity of Work and Span}
\author{David M Kahn}
\date{Summer 2022}

\begin{document}

\maketitle

\section{Asymptotic Complexity}

\textit{Complexity} can be thought of as a 
measure of how difficult something is to compute.
Typically, this would be measured in terms of 
the number of steps it takes, but in general it 
might be used to refer to the consumption of 
any sort of resource, like space or energy. 
In this course, we will be analyzing the 
complexity of particular programs.

The adjective \textit{asymptotic} describes 
the quality of complexity we normally care about: 
how it behaves in the limit. We like to think about 
how the complexity of a program scales with the size of 
its input (or occasionally the size of other features)
so that we have a loose sense of our code's efficiency. 
Faster growing complexities 
will always \textit{eventually} outpace slower 
ones, and are thus less efficient \textit{in the limit}.
Knowing how the complexity scales can also inform
us how many more 
resources to set aside to apply our code to 
larger problems. 

\subsection{Big O}

In order to classify the different growth 
rates of complexity, we use a special 
notation called \textit{big O}, along with 
its relatives\footnote{
  This is also called Bachmann-Landau notation.
}. Each of these is used to
pick out sets of numerical functions
that have growth rates within certain
parameterized bounds. For example, we 
call the functions in $O(n)$ \textit{linearly bounded},
or sometimes just \textit{linear}.

Big O itself gives us a sort of upper bound 
on function growth rates, and will be our 
main focus in this course. Sometimes people 
describe big O as bounding the worst case cost - 
this is not really true. Big O is simply a 
way to describe functions with growth rates 
under some bound, and can be used to describe
worst case, average case, best case, or any 
other sort of cost you care to measure.

It is convenient and common when talking about 
big O to explicitly 
include the formal parameter of 
functions, so we would write 
$f(x)$ instead of $f$. With this 
convention in mind, the definition 
of $O(f(x))$ is the \textit{set} of 
\textit{functions} that are 
\textit{upper-bounded} by a scalar of $f(x)$ 
on \textit{large enough} inputs. Or 
more formally, $g(x) \in O(f(x))$ \footnote{
  We read the symbol $\in$ as ``is a member of the set'',
  or more simply, ``is in".
} 
if: 
\begin{itemize}
  \item for some choice of $n_0$ and $c$
  \item for every $n \geq n_0$
  \item $|c * f(n)| \geq |g(n)|$
\end{itemize}
It does not matter which $n_0$ and $c$ you choose, 
as long as \textit{any} choice works. Also, 
we may ignore the absolute values 
in the final condition if we are just working 
with positive functions, which is usually the
case.

We can also define big O for functions of multiple 
variables, like $O(f(m,n))$. The way we extend our 
old definition is to have \textit{both} $m$ and $n$ 
be at least $n_0$, so we consider how 
the function grows as \textit{all} parameters get big.\footnote{
Be aware that sometimes people 
extend big O to multiple 
parameters differently than this definition. 
However, this is probably the most common choice.
}

Here are some examples of big O classifications
and non-classifications:

\begin{itemize}
  \item $n \in O(n)$ but $n \not \in O(1)$
  \item $500n + 37\sqrt n - 2.5 \in O(n)$ and 
  $n \in O(500n + 37\sqrt n - 2.5)$
  \item $n \in O(n^2)$ but $n^2 \not \in O(n)$
  \item $log_2(n) \in O(\sqrt n)$ but
  $\sqrt n \not \in O(log_2(n))$
  \item $2^n \in O(3^n)$ but $3^n \not \in O(2^n)$
  \item $5 \in O(1)$ and $1 \in O(5)$
  \item $\frac 1 n \in O(1)$ but $1 \not \in O(\frac 1 n)$
  \item 
    $\begin{cases}
      2^n & n \leq 1000000
      \\
      5 & \mathit{else}
    \end{cases} \in O(1)$ even though $2^n \not \in O(1)$
\end{itemize}

There are many common abuses of big O notation 
used by those that work with it, so it is 
good to be aware of them. The common idea 
is that people write $O(f(n))$ to stand in 
for some function in the set. Thus, you 
will see notation like the following:

\begin{itemize}
  \item $f(n) = O(g(n))$ meaning $f(n) \in O(g(n))$
  \item $n^2 + 5n + O(1)$ meaning $n^2 + 5n + f(n)$ 
  for some $f(n) \in O(1)$
\end{itemize}

With such notation in mind, here are some properties of big O:
\begin{itemize}
  \item $O(f(n)) = O(c*f(n))$ for $c \neq 0$
  \item $O(f(n)) + O(g(n)) = O(f(n) + g(n)) $\\
  $= max(O(f(n)),O(g(n))) = O(max(f(n),g(n)))$
  \item $O(f(n)) * O(g(n)) = O(f(n) * g(n))$
  \item $b^{O(f(n))}$ may \textit{not} be equal to $O(b^{f(n)})$
  \item $O(log_a(n)) = O(log_b(n))$ for any valid bases $a,b$
    \footnote{We may therefore just write $O(log(n))$ without ambiguity.}
  \item $O(log(n^k)) = O(log(n))$ for any constant $k$
  \item $O(n^a) \subsetneq O(n^b)$ for $b>a \geq 0$
  \item $O(a^n) \subsetneq O(b^n)$ for $b>a\geq 1$
  \item $O(n^a) \subsetneq O(b^n)$ for $b > 1$
  \item $O(log(n)) \subsetneq O(n^a)$ for $a > 1$
  \item $O(\sum_{i=1}^n \frac 1 i) = O(log(n))$
  \item $O(n!) = O(e^{-n}n^{n+\frac 1 2})$
  \item $O(\sum_{i=1}^n i^k) = O(n^{k+1})$
\end{itemize}

There are other asymptotic notations 
related to big O, including the lower 
bound version big $\Omega$
($f \in \Omega (g)$ iff $g \in O(f)$) and 
the tight bound version big $\Theta$ ($f \in \Theta(g)$ iff 
$f \in O(g)$ and $f \in \Omega(g)$). There are
also even more, like little o, but we will not 
go into those here.

As a final note about these asymptotics:
The growth rates characterized by big O are 
not the absolute metric of 
efficiency, just one that we commonly consider.
For example, there exist matrix multiplication 
algorithms that run in $O(n^{2.373})$ steps\cite{alman2021refined},
which is a strictly better asymptotic bound than the 
$\Theta(n^3)$ steps taken by the naive algorithm.
However, the ``better'' algorithm only 
becomes more efficient in 
terms of the absolute number of steps taken 
when run on matrices so utterly gigantic that 
humans could not actually use them. This sort 
of situation is possible because we can 
pick \textit{any} $c$ and $n_0$ to show 
that a function is asymptotically bounded with 
big O, even outrageously large $c$ and $n_0$.
These kinds of algorithms are
are called \textit{galactic algorithms}, and 
serve as one way to show us that 
big O is not everything.

\section{Work}

\textit{Work} is one of the most common 
resources to measure with big O. The \textit{work}
of a sequential computation is how many steps it takes 
to perform, which in our 
case we can consider to be the number of 
reductions $\step$. 

When considering the 
asymptotic complexity of the work an SML program 
performs, it 
is not actually important to know the exact 
number of such reductions, since scaling 
them by a constant would give the same answer.
Thus, we often will simply make the 
observation that some constant number 
of reductions must occur if our code 
only performs operations like 
pattern matching and arithmetic, and 
does not use potentially-expensive 
operations like function calls; 
we do not need to explicitly calculate 
what that constant is.

There are many ways to bound the 
asymptotic work of a program. One 
of the most common is finding a  
\textit{recurrence relation}. This 
is a collection of (in)equalities 
that recursively defines how 
much work is taken as a function 
of the input. Finding a recurrence 
relation is an especially natural 
approach for 
functional programs, since 
the work recursion will usually 
correspond directly to the 
program's recursion.

Consider the following program for 
calculating integer exponentiation, 
where we will consider the input size 
to be the magnitude of the input integer
$k$: \footnote{
  as opposed to the number of bits 
  making up $k$, which would be 
  logarithmic in $k$'s magnitude.
}

\begin{lstlisting}
(*
 * badpow : int * int -> int
 * REQUIRES: k >= 0
 * ENSURES: badpow (n, k) ==> n^k (where 0^0 is defined as 1)
 *)
fun badpow (n:int, k:int) : int = 
  if k = 0
  then 1 
  else if k mod 2 = 0 
  then badpow (n, k div 2) * badpow (n, k div 2) 
  else n * badpow (n, k div 2) * badpow (n, k div 2) 
\end{lstlisting}
This code gives us the following recurrence for 
the amount of work $W_{badpow}(k)$:\footnote{
  Note that in this case the recurrence is 
  independent of the other input $n$, so 
  we can exclude it. However, this may not 
  always be the case for code with multiple
  inputs.
}
\begin{align*}
  W_{badpow}(0) = & c_0
  \\
  W_{badpow}(k) = & c_1 + 2 * W_{badpow}(\lfloor k / 2 \rfloor)
  & \textrm{for odd $k > 0$}
  \\
  W_{badpow}(k) = & c_2 + 2 * W_{badpow}(\lfloor k / 2 \rfloor)
  & \textrm{for even $k > 0$}
\end{align*}

This recurrence comes from noticing:
\begin{itemize}
  \item The operation of \texttt{badpow} on 0 
just directly calculuates an answer with 
no extra function calls, thus taking 
some constant amount of work $c_0$. 

  \item The operation of \texttt{badpow} 
  otherwise does some constant amount of calculation
  (differing between $c_1$ for odd $k$ or $c_2$ 
  for even $k$) 
  and then also makes 2 recursive calls with 
  the exponent parameter divided by 2. (Because 
  it is integer division it also rounds down.)
\end{itemize}

Once we have this recurrence there are many 
ways to go about solving it. For instance,
one could guess what the answer is and 
prove it by induction, or one could 
apply the master theorem\cite{bentley1980general}, 
or use any other kind of method 
that might come to mind. However, 
what we will teach you in this course is 
a nice approach called the \textit{tree method}.

The idea behind the tree method is to first 
think of the structure of the recursive calls 
like a tree\footnote{
  This can mean the tree is just a 
  line, where there is only 1 recursive call 
  at each level of recursion.
}. In our example, an input of
$k$ makes 2 recursive calls with inputs of 
size $\lfloor k/2 \rfloor$, each of which
makes 2 recursive calls with inputs of 
size $\lfloor \lfloor k/2 \rfloor / 2 \rfloor$, 
all the way down until $k=0$, in which 
case no recursive calls are made. 

Here is what the top of such a tree would look like:

\begin{center}
  \begin{tikzpicture}[scale=0.2]
  \tikzstyle{every node}+=[inner sep=0pt]
  \draw [black] (35.6,-16.7) circle (4);
  \draw (35.6,-16.7) node {$k$};
  \draw [black] (28.2,-25.8) circle (4);
  \draw (28.2,-25.8) node {$\lfloor k/2\rfloor$};
  \draw [black] (42.7,-25.8) circle (4);
  \draw (42.7,-25.8) node {$\lfloor k/2 \rfloor$};
  \draw [black] (20.6,-35.6) circle (4);
  \draw (20.6,-35.6) node {$\lfloor \lfloor k/2 \rfloor /2 \rfloor$};
  \draw [black] (31.1,-35.6) circle (4);
  \draw (31.1,-35.6) node {$\lfloor \lfloor k/2 \rfloor /2 \rfloor$};
  \draw [black] (40.2,-35.6) circle (4);
  \draw (40.2,-35.6) node {$\lfloor \lfloor k/2 \rfloor /2 \rfloor$};
  \draw [black] (51.2,-35.6) circle (4);
  \draw (51.2,-35.6) node {$\lfloor \lfloor k/2 \rfloor /2 \rfloor$};
  \draw [black] (22.44,-33.23) -- (26.36,-28.17);
\draw [black] (29.05,-28.68) -- (30.25,-32.72);
\draw [black] (40.94,-32.69) -- (41.96,-28.71);
\draw [black] (44.67,-28.07) -- (49.23,-33.33);
\draw [black] (30.09,-23.47) -- (33.71,-19.03);
\draw [black] (40.85,-23.43) -- (37.45,-19.07);
\draw [black] (22.44,-33.23) -- (26.36,-28.17);
\draw [black] (21.03,-38.57) -- (21.57,-42.33);
\fill [black] (21.57,-42.33) -- (21.95,-41.47) -- (20.96,-41.61);
\draw [black] (31.93,-38.48) -- (33.07,-42.42);
\fill [black] (33.07,-42.42) -- (33.33,-41.51) -- (32.37,-41.79);
\draw [black] (40.95,-38.51) -- (41.95,-42.39);
\fill [black] (41.95,-42.39) -- (42.24,-41.5) -- (41.27,-41.75);
\draw [black] (52.1,-38.5) -- (55.4,-42.9);
\fill [black] (55.4,-42.9) -- (55.32,-41.96) -- (54.52,-42.56);
\draw [black] (50.77,-38.57) -- (50.23,-42.33);
\fill [black] (50.23,-42.33) -- (50.84,-41.61) -- (49.85,-41.47);
\draw [black] (39.62,-38.54) -- (38.88,-42.36);
\fill [black] (38.88,-42.36) -- (39.52,-41.67) -- (38.54,-41.47);
\draw [black] (30.47,-38.53) -- (29.63,-42.37);
\fill [black] (29.63,-42.37) -- (30.29,-41.69) -- (29.32,-41.48);
\draw [black] (19.12,-38.21) -- (16.58,-42.69);
\fill [black] (16.58,-42.69) -- (17.41,-42.24) -- (16.54,-41.75);
  \end{tikzpicture}
  \end{center}

Once we have this tree, we want to think about 
three quantities:
\begin{itemize}
  \item the maximum depth of the tree 
  \item the number of nodes in a row at a given depth 
  \item the amount of work ``at a given node''
\end{itemize}

\paragraph{depth} To calculate the first of these, the depth, we need to know 
how many times one can (integer) divide $k$ by 2 
until one gets 0. The answer happens to 
be $\lfloor log_2(k) \rfloor + 1$ (or 0 if $k=0$, 
but we are interested in the behaviour of large $k$). 
There are a variety of ways to verify this, like induction.

Of course, since we are only interested in upper bounds,
we can also overapproximate the depth and still 
get a valid answer\footnote{
  But be careful! If you overapproximate too much, 
  the answer may not be as tight as it could be, 
  even if it does remain valid.
}. This can make the determination of depth 
easier.
Since $k/2 \geq \lfloor k /2 \rfloor$ (for non-negative $k$), we can just count 
how many times we need to divide by 2 to get to 1, and 
ignore the integer rounding. 
That is more directly calculable by solving 
for $d$ in $\frac k {2^d} = 1$. The solution is 
$d = log_2(k)$, which gives an upper bound 
on the number of recursive calls to get from
input $k$ to input 1. 
Then we need one more round of recursion 
to get from an input of 1 to an input of 0, so 
the tree depth is no greater than $log_2(k)+1$.

\paragraph{nodes in a row} The number of 
nodes at a given row is fairly easy to calculate.
For a tree with fixed branching degree $b$, 
the number of nodes at row $i$ (1-indexed from the 
root) is $b^{i-1}$. In our case, $b=2$, since exactly 
2 recursive calls are made in each non-base case.
For trees without fixed branching degree, 
the situation is a little harder.


\paragraph{work per node} Since each node 
corresponds to a level of recursion, 
this means the amount of work ``at a node'' is the 
amount of work our recurrence gives \textit{not} 
counting recursive uses, since the work in the recursive 
use would be ``at'' a node 1 layer deeper in the tree. 
This means each leaf has $c_0$ work, and otherwise 
each node has $c_1$ or $c_2$ work depending on if 
the input is odd or even.

Again, however, there is an opportunity to make 
our lives easier by overapproximation. Let
the constant $c_3$ be at least $max(c_0,c_1,c_2)$.
This provides an upper bound on the work done per 
node, and thus is a safe replacement to use 
when calculating the big O upper bound.
(One could even just call it $O(1)$, but 
lets not do that here.) This  
corresponds to using the recurrence below:
\begin{align*}
  W_{badpow}(0) \leq & c_3
  \\
  W_{badpow}(k) \leq & c_3 + 2 * W_{badpow}(\lfloor k / 2 \rfloor)
  & \textrm{for } k > 0
\end{align*}

\paragraph{putting it together} Now that we 
have (upper bounds on) these 3 quantities,
all that remains is to add up the work over 
the whole tree. The easiest way to do this is 
by first finding the work accross each row, and then adding 
the row totals vertically up the tree. For an input 
of size $n$, if 
$d(n)$ bounds the recursion tree depth, $r(n,i)$ bounds 
the number of nodes at row $i$, and 
$w(n,i)$ bounds the work done per node at row $i$,
then this sum is given by:
$$\sum_{i=1}^{d(n)}w(n,i) * r(n,i)$$

For our example, this works out to:
$$\sum_{i=1}^{log_2(k)+1} 2^{i-1} * c_3 = c_3 * (2^{log_2(k)+1} - 1)
= c_3 * (2 * k - 1) \in O(k)$$

Thus, the implementation of \texttt{badpow} takes 
work linear in the magnitude of the input $k$.
This (as its name suggests) is not the most 
efficient implementation. Compare it to 
\texttt{fastpow} below, which can be found to 
take $O(log(k))$ time using the tree method.

\begin{lstlisting}
(*
 * fastpow : int * int -> int
 * REQUIRES: k >= 0
 * ENSURES: fastpow (n, k) ==> n^k (where 0^0 is defined as 1)
 *)
fun fastpow (n:int, k:int) : int = 
  if k = 0
  then 1 
  else 
  let 
    val subsoln = fastpow (n, k div 2) 
    val squared = subsoln * subsoln
  in 
    if k mod 2 = 0 
    then squared 
    else n * squared
  end  
\end{lstlisting}

\section{Span}

Span is similar to work, but instead counts 
the number of steps of a \textit{parallel} 
computation. Otherwise, reasoning about 
span is quite similar to reasoning 
about work.

Our reductions $\step$ do not 
describe parallel computation, so we need
new, parallel operational 
semantics. To do this, we will 
define $\step_p$, the parallel reduction 
step. It behaves exactly like the 
sequential reduction step $\step$ 
everywhere\footnote{
  So some operations still do occur 
  sequentially, even for parallel reduction.
} except for tuples\footnote{
  Tuples include arguments to binary 
  operators, since, e.g., \texttt{a + b} is 
  just shorthand for the application 
  \texttt{op+ (a,b)}.
}. We can then reduce tuples in the 
following parallel way, exemplified by 
the 2-tuple: 

\texttt{(}$e_1$\texttt{,}$e_2$\texttt{)}
$\step_p$\texttt{(}$e_1'$\texttt{,}$e_2'$\texttt{)}
if 
\begin{itemize}
  \item $e_1 \step_p e_1'$ and $e_2 \step_p e_2'$
  \item $e_1 \step_p e_1'$ and $e_2=e_2'$ is a value 
  \item $e_1=e_1'$ is a value and $e_2 \step_p e_2'$
\end{itemize}

Without considering exceptions,
$\step$ and $\step_p$ induce extensionally 
equivalent behaviour. If one evaluates, 
so does the other, and if one loops forever,
so does the other. Exceptions still
can make them behave differently though. But all 
this is not so important, because 
span only cares about the 
number of steps to reduce to a value.

With our new parallel reduction rules,
we can define the span of the
execution of $e$ as $k$ if $e \stepk{k}_p v$ 
for a value $v$. Like before with work, it
is never critical to actually calculate 
an exact span for asymptotic reasoning, 
only to recognize when it 
is e.g. constant. Thus, we will not 
make much use of these new reduction rules
directly, even though they 
do underpin our reasoning.

The main effect of working with parallel reduction 
is that the cost of tuples does not \textit{add} 
the cost of each element, but takes the \textit{max} 
cost. This is because sequential work had to reduce 
each element one-by-one, whereas our new 
parallel rules allow the reduction of all 
elements at once. Sometimes 
this may not effect the asymptotics,
since, e.g., $O(f(n) + g(n)) = O(max(f(n),g(n)))$.
But, if these operations are iterated, like 
through a recurrence, an asymptotic difference can appear. 

Consider \texttt{badpow} from our work analysis 
above. The recurrences it gives for 
\textit{span} are the following:
\begin{align*}
  S_{badpow}(0) = & c_4
  \\
  S_{badpow}(k) = & c_5 + max(S_{badpow}(\lfloor k / 2 \rfloor), S_{badpow}(\lfloor k / 2 \rfloor))
  & \textrm{for odd $k > 0$}
  \\
  S_{badpow}(k) = & c_6 + max(S_{badpow}(\lfloor k / 2 \rfloor), S_{badpow}(\lfloor k / 2 \rfloor))
  & \textrm{for even $k > 0$}
\end{align*}

This time around we can simplify the $max$ 
and upper-bound our constants to get:
\begin{align*}
  S_{badpow}(0) \leq & c_7
  \\
  S_{badpow}(k) \leq & c_7 + S_{badpow}(\lfloor k / 2 \rfloor)
  & & \textrm{for } k > 0
\end{align*}

For this recurrence, our tree is actually a  
line, and its depth is no greater than 
$log_2(k)+1$ for the same reason as before. 
The asymptotic span is thus characterized by
the upper bound:

$$\sum_{i=1}^{log_2(k)+1} 1 * c_7= c_7 * (log_2(k)+1) \in O(log(k))$$

Thus, this implementation of \texttt{badpow} 
does have better span than 
work, and parallelism would speed it up (asymptotically).
Compare this to the naive implementation 
below, which has linear work \textit{and} linear span.

\begin{lstlisting}
(*
 * pow : int * int -> int
 * REQUIRES: k >= 0
 * ENSURES: pow (n, k) ==> n^k (where 0^0 is defined as 1)
 *)
fun pow (n:int, k:int) : int = 
  (case k of
      0 => 1 
    | _ => n * pow (n, k - 1)
  )
\end{lstlisting}


\bibliographystyle{IEEEtran}
\bibliography{sources}


\end{document}