\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{proof}
\usepackage{mathpartir}
\usepackage{url}
\usepackage{listings}
\usepackage{color}
\definecolor{eclipseBlue}{RGB}{42,0.0,255}
\definecolor{eclipseGreen}{RGB}{63,127,95}

\lstset {
  basicstyle=\small\ttfamily,
  captionpos=b,
  tabsize=2,
  columns=fixed,
  breaklines=true,
  frame=l,
  numbers=left,
  numberstyle=\small\ttfamily,
  morekeywords= {
    EQUAL, GREATER, LESS, NONE, SOME, abstraction, abstype, and, andalso, array, as, before, bool, case, char, datatype, do, else, end, eqtype, exception, exn, false, fn, fun, functor, handle, if, in, include, infix, infixr, int, let, list, local, nil, nonfix, not, o, of, op, open, option, orelse, overload, print, raise, real, rec, ref, sharing, sig, signature, string, struct, structure, substring, then, true, type, unit, val, vector, where, while, with, withtype, word
  },
  morestring=[b]",
  morecomment=[s]{(*}{*)},
  stringstyle=\color{black},
  identifierstyle=\color{eclipseBlue},
  keywordstyle=\color{red},
  commentstyle=\color{eclipseGreen}
} 

\newtheorem{theorem}{Theorem}

\newcommand{\texttilde}{\url{~}}
\newcommand{\step}{\stackrel{1}{\Longrightarrow}}
\newcommand{\stepk}[1]{\stackrel{#1}{\Longrightarrow}}

\title{Lecture 4 - Lists and Structural Induction}
\author{David M Kahn
  \footnote{Adapted from Michael Erdmann's Spring 2022 notes, 
  in turn adapted from a draft of Frank Pfenning}
}
\date{Summer 2022}

\begin{document}

\maketitle

\section{Lists}

Lists in SML are a built-in data type for storing 
arbitrary finite sequences of one type of data. Most languages 
have some sort of data type like this, like Java's Arrays
and Python's lists. However, each language treats 
this sort of data type a little differently. For SML, 
this means:

\begin{itemize}
  \item Everything in the list must be the same type. This 
  keeps the type sytem clean and elegant.\footnote{This 
  retriction to a single type
  is not actually that restrictive, as we will see 
  later in the course.}
  
  \item There is no arbitrary access; no indexing 
  into the list. In many languages you can write something 
  like \verb"lst[3]" to access the 4th element of the 
  list \verb"lst", but not in SML. This is because index systems are 
  more complicated than we want our basic data types 
  to be. Instead, SML opts for a more 
  naive data structure:
  the \textit{linked list}. This simplicity will turn 
  out to be beneficial when it allows us 
  to make use of \textit{structural induction}.
\end{itemize}

\paragraph{Type} $t$ \verb"list" for any element type $t$
(there is no type that is just \verb"list" without a 
type parameter for the element)

\paragraph{Values} 
\begin{itemize}
  \item \verb"[]" the empty list; it may also be called ``nil".
  \item \verb"v1::v2::v3::"\dots\verb"::vn::[]" a list 
  of $n$ other values ``cons''-ed together; this may also be written 
  as \verb"[v1, v2, v3,"\dots\verb",vn]"
\end{itemize}

\paragraph{Expressions}
\begin{itemize}
  \item Make: The expressions 
  that make lists include all list values, any 
  use of the \textit{cons} operation \verb"::" which 
  adds an element to the front of another list, or 
  the append operation \verb"@" which sticks two 
  lists together

  \item Use: Casing is the preferred way 
  to break up lists, like 
  \\
  \verb"case lst of [] => "$e_1$\verb"| hd::tl => "$e_2$, since \verb"[]" and \verb"::" 
  are also patterns. (\verb"@" is not a pattern)
\end{itemize}

\paragraph{Typing}
\begin{itemize}
  \item Make: 
  \begin{itemize}
    \item \verb"[]:"$t$ \verb"list" for any type $t$
    \item $e_1$\verb"::"$e_2$\verb":"$t$ \verb"list" if $e_1$\verb":"$t$ 
    and $e_2$\verb":"$t$ \verb"list"
    \item $e_1$\verb" @ "$e_2$\verb":"$t$ \verb"list" if $e_1$ and  
    and $e_2$ are both typed $t$ \verb"list"
  \end{itemize}

  \item Use: Lists can be used by breaking them up with 
  case statements in the normal way. For example, 
  \verb"case "$e_1$\verb" of [] => "$e_2$\verb"| hd::tl => "$e_3$\verb":"$t$
  if \begin{itemize}
    \item $e_1$\verb":"$t'$ \verb"list"
    \item $e_2$\verb":"$t$
    \item $e_3$\verb":"$t$ assuming \verb"hd:"$t'$ and 
    \verb"tl:"$t'$ \verb"list"
  \end{itemize}
\end{itemize}

\paragraph{Reduction} left-to-right for binary operators, as usual

\paragraph{Special Notes} 
Unlike most binary operators which are 
left associative, cons \verb"::" is right 
associative. So \verb"a::b::c" implicitly gets 
parenthesized as \verb"a::(b::c)". 
If you pay attention to the typing 
of cons, this is the only associativity that makes sense. 

\section{Structural Induction}

Structural induction is a special kind of 
induction that is performed on inductively 
structured data, to prove that a property 
holds for all such data. 
For data to be inductively structured,
it must have at least one 
base value (like nil \verb"[]"), and also some number of
ways of constructing new values out of 
old ones (like cons \verb"::") -- 
we call the collection of both of
these the data type's \textit{(inductive) constructors}. 
Because lists have such a set of constructors, this 
means that lists are a great target for 
structural induction. Later we will 
see that we can structurally induct over other 
data types too.

We can actually derive structural induction  
from our usual induction over natural numbers\footnote{
  We can also go the other direction, and derive 
  induction over natural numbers from structural 
  induction. To go in this direction, note that 
  natural numbers may be structured with a base 
  value of 0 and an inductive constructor 
  of incrementation by 1 (successor).
}. 
This can be done by having the number we induct on count 
the number of constructors used in the data type (plus one).
Thus, for lists, 
the base case of 0 corresponds to the case for the empty list, 
and the inductive step going from $n$ to $n+1$ corresponds 
to the step going from a list \textit{tl} of length $n$ to 
a list \textit{hd}\verb"::"\textit{tl} of length $n+1$.
For lists, this is just induction on the list's length!

However, it is useful to shortcut this process, and simply 
consider inductive cases for each constructor directly.
Each base value forms a base case, and each way of making 
new values forms a different inductive step\footnote{
  If there are only base values and no way of 
  building bigger values from smaller, then 
  structural induction is proof by cases! 
}. For the 
type $t$ \verb"list", 
this leaves us with the inductive principle:
\\
\\
for some predicate P, if 
\begin{itemize}
  \item P(\verb"[]") holds
  \item P(\textit{hd}\verb"::"\textit{tl}) holds assuming 
  P(\textit{tl}) for values \textit{hd}\verb":"$t$ and \textit{tl}\verb":"$t$ \verb"list"
\end{itemize}
then P(\textit{lst}) holds for all values \textit{lst}\verb":"$t$ \verb"list"
\\
\\

There also exist the same variants we had before for 
induction: strong induction and induction on a generalized 
predicate.

\subsection{Example}

Here is a nice proof example for showing how to use 
structural induction alongside other proof principles.

First, let us assume that the following is the 
implementation of \verb"@", and assume that this 
implementation is total\footnote{
  In this course, we will either tell you that 
  a function is total, or you will have to prove it 
  yourself.
}. Consider proving its totality 
yourself as an exercise!

\begin{lstlisting}
(*
 * @ : int list * int list -> int list
 * REQUIRES: true
 * ENSURES: a @ b ==> a list containing all and only 
 *  the elements of a followed by those b, maintaining order
 *)
infix @
fun (lst1:int list) @ (lst2:int list) : int list = 
  (case lst1 of
      [] => lst2 
    | hd::tl => hd :: (tl @ lst2)
  )
\end{lstlisting}


\begin{theorem}
  \verb"@" is associative, i.e., for \verb"int list" values 
  $a,b,c$

  $$ (a\;  \verb"@" \; b) \;\verb"@"\; c \cong a \; \verb"@"\; (b\; \verb"@" \; c)$$
\end{theorem}
\begin{proof}
  We prove by structural induction over $a$.
  
  \paragraph{Base Case} $a = \verb"[]"$

  For this case, we want to show 
  $([] \;  @ \; b) \; @ \; c \cong [] \;  @ \; (b \; @ \; c)$.

  First we show that $([] \;  @ \; b) \; @ \; c \cong v_1$ for 
  some value $v_1$ by showing the reduction 
  $([] \;  @ \; b) \; @ \; c \; \Longrightarrow \; v$,
  since extensional equivalence is closed under reduction.
  \begin{align*} 
    (\mathtt{[]} \;  @ \; b) \; @ \; c  \;\Longrightarrow\; 
    & b\; @ \; c & \textrm{clause 1}
    \\
    \;\Longrightarrow\; & v_1 \; \textrm{(for some value $v_1$)}
    & \textrm{totality of @}
  \end{align*}

  Then we show that $[] \;  @ \; (b \; @ \; c) \cong v_1$ as 
  well, again via reduction.

  \begin{align*} 
    [] \;  @ \; (b \; @ \; c) \; \Longrightarrow \; 
    & [] \; @ \; v_1 \; \textrm{(for the same $v_1$ as above)}
    & \textrm{totality of @}
    \\
    \Longrightarrow \; & v_1 & \textrm{clause 1}
  \end{align*}

  Finally, because extensional equivalence is an equivalence relation,
  these two facts mean that 
  $([] \;  @ \; b) \; @ \; c \cong [] \;  @ \; (b \; @ \; c)$,
  completing the case.

  \paragraph{Inductive Step} $a = \mathit{hd}::\mathit{tl}$
  for \verb"int list" values $\mathit{hd},\mathit{tl}$
  
  For this case, we want to show 
  $((\mathit{hd}::\mathit{tl})\;  @ \; b) \; @ \; c \cong (\mathit{hd}::\mathit{tl}) \;  @ \; (b \; @ \; c)$,
  assuming that 
  $(\mathit{tl}\;  @ \; b) \; @ \; c \cong \mathit{tl} \;  @ \; (b \; @ \; c)$.

  First, we show that $((\mathit{hd}::\mathit{tl})\;  @ \; b) \; @ \; c
  \cong \mathit{hd}::((\mathit{tl}\;  @ \; b) \; @ \; c)$ via 
  a sequence of reductions and extensional equivalences. Note 
  that the extensional equivalence of the last line follows 
  due the reduction that was ensured by totality.
  \begin{align*}
     ((\mathit{hd}::\mathit{tl})\;  @ \; b) \; @ \; c \; \Longrightarrow \; 
    & (\mathit{hd}::(\mathit{tl}\;  @ \; b)) \; @ \; c
    & \textrm{clause 2}
    \\
    \Longrightarrow \; 
    & (\mathit{hd}::v_2) \; @ \; c \; \textrm{(for some value $v_2$)}
    & \textrm{totality of @}
    \\
    \Longrightarrow \; 
    & \mathit{hd}::(v_2 \; @ \; c) \;
    & \textrm{clause 2}
    \\
    \cong \; 
    & \mathit{hd}::((\mathit{tl}\;  @ \; b) \; @ \; c) \;
    & \mathit{tl}\;  @ \; b \cong v_2
  \end{align*}

  Then, we show that $(\mathit{hd}::\mathit{tl})\;  @ \; (b \; @ \; c)
  \cong \mathit{hd}::((\mathit{tl}\;  @ \; b) \; @ \; c)$ 
  similarly as before.
  \begin{align*}
    (\mathit{hd}::\mathit{tl})\;  @ \; (b \; @ \; c) \; \Longrightarrow \; 
   & (\mathit{hd}::\mathit{tl})\;  @ \; v_1 \;  \textrm{(for some value $v_1$)}
   & \textrm{totality of @}
   \\
   \Longrightarrow \; 
   & \mathit{hd}::(\mathit{tl}\;  @ \; v_1) 
   & \textrm{clause 2}
   \\
   \cong \;
   & \mathit{hd}::(\mathit{tl}\;  @ \; (b \; @ \; c)) 
   & b \; @ \; c \cong v_1
 \end{align*}

 Finally, using the inductive hypothesis that 
 $(\mathit{tl}\;  @ \; b) \; @ \; c \cong \mathit{tl} \;  @ \; (b \; @ \; c)$, 
 we can relate the above two conclusions as
 $\mathit{hd}::((\mathit{tl}\;  @ \; b) \; @ \; c)\cong \mathit{hd}::(\mathit{tl}\;  @ \; (b \; @ \; c)) $.
 Since extensional equivalence is an equivalence relation, 
 this means we can conclude
 $((\mathit{hd}::\mathit{tl})\;  @ \; b) \cong (\mathit{hd}::\mathit{tl})\;  @ \; (b \; @ \; c)$,
 completing the case.


\end{proof}

Note that we needed to use the totality of \verb"@" 
in the above proof. Without it, we have no way of knowing that
expressions like $b \; @ \; c$ actually are valuable,
and we need to turn them into values to use our 
reduction rules. We also made repeated use of 
referential transparency to swap around extensionally 
equivalent subexpressions.


\subsection{A Mathematical Digression}

To be more aware of what we are doing mathematically 
when we induct, it is key to realize that all forms
of induction rely on the structure they are inducting 
over being the \textit{least fixed point} of its 
constructors. For natural numbers, this means 
we take the the smallest set containing 
0 and closed under adding 1, which is the standard 
collection of natural numbers. For integer lists, this means 
we take the the smallest 
set of values containing the empty list and closed under cons-ing 
another \verb"int" onto the front, which is the 
standard collection of finite integer lists. 

Note that if we added ``infinity" to the above set of 
natural numbers, the set is still closed under its 
constructors -- infinity plus 1 is still infinity.
Likewise, adding infinite 
(\verb"v1::v2::v3::v4::"\dots) 
or looped (\verb"lst = v::lst") ``lists" to 
our set of list values would still give a set
closed under the list constructors. 
However, both these sets are \textit{larger} than 
the previous sets named, since they are strict supersets,
and so are \textit{not} 
what we consider when we induct. 

Such a focus when inducting over SML lists 
is appropriate because the type of lists 
is an \textit{inductive} type, 
as are most types that you will see. This means that
the values of that type are only considered to be those 
in the smallest set closed under its constructors, which are the same 
sets that induction acts on. As a result, 
those infinite and looped ``lists" are not actually 
lists, and it is perfectly OK that our proof technique of 
induction misses them.

Thus, we can say that 
the length function for a list given below is total,
even though it would never terminate on 
an infinite or looped ``list''. 

\begin{lstlisting}
  (*
   * len: int list -> int
   * REQUIRES: true
   * ENSURES: len lst evaluates to the length of lst
   *)
  fun len lst = 
    (case lst of 
        [] => 0
      | _::tl => 1 + len tl
    )
\end{lstlisting}

Do not take it for granted that every type you 
see in the wild is inductive. For example, in OCaml, a cousin of SML,
one can make looped lists. This can make it difficult
to reason fully about some languages, since induction 
would not be sufficient. However, for right now with SML, 
we are only using inductive types.

Some languages also have features 
to allow treating the set 
of values for a type as 
the \textit{greatest} fixed point of its constructors --  
such types are called \textit{co}inductive types. These correspond 
to a dual proof technique to induction called 
\textit{co}induction. You need not worry about this technique
right now, but it is out there if you wanted to go beyond 
the course.


\section{Tail Recursion}

\textit{Tail recursion} is an 
important optimization tool in the arsenal of  
a functional programmer. This is 
a way of ordering recursive calls 
so that the machine executing 
the code does not need to 
remember to do any additional 
computation after the recursive call 
returns. This has a number of consequences 
for the efficiency of the code.

\subsection{Definitions}

\paragraph{Tail Call}
Within the body of a function f, 
a function call is a tail call 
if f does not inspect or compute 
with the result of that call; 
instead that call is the “last” 
computation f does, and f returns 
the call's result.

\paragraph{Tail Recursion}
A function is tail recursive if 
every call it makes to a recursive 
function is a tail call, and each of 
those called recursive functions can 
themselves be considered tail recursive.
\\
\\
A prototypical template (but not the only 
possible template) for a tail recursive 
function is the following, where \verb"e1" and 
\verb"e2" are arbitrary expressions (that do not 
make additional function calls), and 
\verb"p1" and \verb"p2" are appropriate patterns.

\begin{lstlisting}
  fun f x =
    (case x of 
        p1 => e1
      | p2 => f (e2)
    )
\end{lstlisting}
Notice how our reduction rules would cause \verb"e2" to 
be computed \textit{before} applying \verb"f", so 
the call to \verb"f" is a tail call.

\subsection{Machine View}

While we do not normally care to think about 
the implementation details of the machine 
running our code, and it is not strictly-speaking 
necessary for the course, doing so for tail recursion 
can be enlightening.

The machine running our code must track the 
environment to resolve variables, and also 
something called the \textit{return address} 
which tells the machine where in the code to return 
with any computed values. These are usually 
stored in an area of memory called the \textit{call stack}. 
Every time a function is called, a new \textit{frame} is usually 
added to call stack to store any new local variables 
that call of the function uses, as well as the return address.
When a function call would return a value, it simply stores that value 
in a return register, 
pops the code pointer back to its return 
address, deallocates its frame, and restores the environment 
stored in the previous frame.

However, for tail recursive functions, the machine can 
make an optimization. By definition, no computation is 
done with the return of the tail calls inside a tail recursive function
-- instead the value returned from the tail call is just itself 
returned by the function. This means the machine can forget the 
environment it would normally store in a stack frame, since 
no variables are needed if no computation will be done. This 
also means that the machine only needs to remember 
the single return address of original function caller -- 
when the tail recursive function returns, by definition there is no 
computation remaining to 
do in any level of its recursion, so the next computation to 
occur will be outside of the tail recursive function, wherever it 
was originally called. Together, these two facts mean 
there is nothing 
that needs to be stored in the call stack frames, so the machine 
can save time and memory by just never allocating them. Instead, 
the machine can treat the tail recursion like an imperative loop,
which is a code pattern it can execute efficiently.



\subsection{Example}
The following implementation \verb"len"
of the length function for lists is \textit{not} 
tail recursive, because \verb"len" is 
not called as a tail call -- after 
the recursive call to 
\verb"len" returns, the code performs 
the additional computation of adding
one.

\begin{lstlisting}
  (*
   * len : int list -> int
   * REQUIRES: true
   * ENSURES: len lst ==> the length of lst
   *)
  fun len (lst:int list) : int = 
    (case lst of
        [] => 0
      | _::tl => 1 + len tl
    )
\end{lstlisting}

However, the following implementation \verb"lenT"
\textit{is} tail recursive. Note how it 
achieves tail recursion by introducing 
an accumulator and making use of a 
helper function. This is a common 
pattern for achieving tail recursion.

\begin{lstlisting}
  (*
   * lenHelp : int list * int -> int
   * REQUIRES: true
   * ENSURES lenHelp (lst, a) evaluates to a + length of lst
   *)
  fun lenHelp (lst:int list, acc:int) : int =
    (case lst of 
        [] => acc 
      | _::tl => lenHelp (tl, acc + 1)
    )

  (*
   * lenT : int list -> int
   * REQUIRES: true
   * ENSURES: lenT lst ==> the length of lst
   *)
  fun lenT (lst:int list) : int = 
    lenHelp (lst, 0)
\end{lstlisting}


We can visually see the effect of 
tail recursion by observing the 
execution trace of each implementation. 

For the non-tail recursive \verb"len", 
the execution trace bulges out in the middle,
since it must remember to increment for  
every element in the list. Recording all of 
these increments
requires an amount of extra memory proportional 
to the length of the input list\footnote{
  Specifically, each increment needs to 
  be pointed to by a return address, and each 
  return address needs its own stack frame.
}.
Just look how much of each line is devoted 
to something other than the list!

\begin{lstlisting}
  len [1, 2, 3, 4, 5, 6, 7, 8]
==> 1 + len [2, 3, 4, 5, 6, 7, 8]
==> 1 + (1 + len [3, 4, 5, 6, 7, 8])
==> 1 + (1 + (1 + len [4, 5, 6, 7, 8]))
==> 1 + (1 + (1 + (1 + len [5, 6, 7, 8])))
==> 1 + (1 + (1 + (1 + (1 + len [6, 7, 8]))))
==> 1 + (1 + (1 + (1 + (1 + (1 + len [7, 8])))))
==> 1 + (1 + (1 + (1 + (1 + (1 + (1 + len [8]))))))
==> 1 + (1 + (1 + (1 + (1 + (1 + (1 + (1 + len [])))))))
==> 1 + (1 + (1 + (1 + (1 + (1 + (1 + (1 + 0)))))))
==> 1 + (1 + (1 + (1 + (1 + (1 + (1 + 1))))))
==> 1 + (1 + (1 + (1 + (1 + (1 + 2)))))
==> 1 + (1 + (1 + (1 + (1 + 3))))
==> 1 + (1 + (1 + (1 + 4)))
==> 1 + (1 + (1 + 5))
==> 1 + (1 + 6)
==> 1 + 7 
==> 8
\end{lstlisting}

However, the tail recursive \verb"lenT"
orders its computations to perform the incrementation 
earlier using an accumulator. Notice that 
this time, the amount of each line 
devoted to something other than the list 
remains bounded by a constant, and each 
line never makes reference to anything 
from a previous recursive call's function body.

\begin{lstlisting}
  lenT [1, 2, 3, 4, 5, 6, 7, 8]
==> lenHelp ([1, 2, 3, 4, 5, 6, 7, 8], 0)
==> lenHelp ([2, 3, 4, 5, 6, 7, 8], 0 + 1)
==> lenHelp ([2, 3, 4, 5, 6, 7, 8], 1)
==> lenHelp ([3, 4, 5, 6, 7, 8], 1 + 1)
==> lenHelp ([3, 4, 5, 6, 7, 8], 2)
==> lenHelp ([4, 5, 6, 7, 8], 2 + 1)
==> lenHelp ([4, 5, 6, 7, 8], 3)
==> lenHelp ([5, 6, 7, 8], 3 + 1)
==> lenHelp ([5, 6, 7, 8], 4)
==> lenHelp ([6, 7, 8], 4 + 1)
==> lenHelp ([6, 7, 8], 5)
==> lenHelp ([7, 8], 5 + 1)
==> lenHelp ([7, 8], 6)
==> lenHelp ([8], 6 + 1)
==> lenHelp ([8], 7)
==> lenHelp ([], 7 + 1)
==> lenHelp ([], 8)
==> 8
\end{lstlisting}



\end{document}