\lecturetitle{2}{Compression 1}{29 Aug 97}{Ben Zhao}

\section{Introduction to Compression}

\begin{itemize} 
  \item Information Theory

  \item Lossless Compression
    \begin{enumerate}
    \item Huffman Coding
    \item Shannon-Fano Coding
    \item Arithmetic Coding
    \item LZ78 and LZW (GIF, UNIX Compress...)
    \item LZ77 (gzip, ZIP, ...)
    \item PPM (prediction by partial matching)
    \end{enumerate}

  \item Lossy Compression
    \begin{enumerate}
    \item Transform Coding (Blocks: JPEG, MPEG, Wavelets...)
    \item Vector Quantization
    \end{enumerate}

\end{itemize}

\subsection{Information Theory}

How much can we compress any given data?
It
Depends on how much ``information'' the data contains.

\subsubsection*{Entropy} 

Shannon borrowed the the notion of {\em entropy} from statistical
physics to capture the information contained in an information source.
For a set of possible messages $S$, it is defined as,
 \[ H(S) = \sum_{i}p_{i}\log_{2}\frac{1}{p_{i}} \]
where $p_{i}$ is the probability of $i^{th}$ message that can be
sent by the source.  The entropy is
a measure of the average number of bits needed to send a message,
therefore larger entropies represent more information.  Perhaps
counter intuitively, the more random a set of messages (the more even
the probabilities) the more information they contain.
The terms
$ \log_{2}\frac{1}{p_{i}} $ represent the number of bits used
to code message $i$ (the information contained in $S_i$).  Therefore messages with higher probability will
use fewer bits.

\subsubsection*{Examples:}

\begin{eqnarray*}
 p_{i} &= & \{ 0.25, 0.25, 0.25, 0.125, 0.125 \}\\[.05in]
 H & = & 3 \times 0.25 \times \log_{2} 4 + 2 \times 0.125 \times \log_{2} 8 \\
   &  = & 1.5 + 0.75 \\
   & = & 2.25\\[.1in]
 p_{i} &=& \{ 0.5, 0.125, 0.125, 0.125, 0.125 \}\\[.05in]
 H &=& 0.5 \times \log_{2} 2 + 4 \times 0.125 \times \log_{2} 8\\
  &=& 0.5 + 1.5\\
  &=& 2\\[.1in]
 p_{i} &=& \{ 0.75, 0.0625, 0.0625, 0.0625, 0.0625 \} \\[.05in]
 H & &= 0.75 \times \log_{2}(\frac{4}{3}) + 4 \times 0.0625 \times \log_{2} 16\\
 & =& 0.3 + 1\\
 & =& 1.3
\end{eqnarray*}
Note that the more uneven the distribution, the lower the Entropy.

\subsection{The English Language}

We might be interested in how much information the English Language
contains.  This could be used as a bound on how much we can compress
English, and could also allows us to compare the density (information
content) of different languages.

\begin{table}
\centering
\begin{tabular}{|l|l|} \hline\hline
				&	{\em bits/char} \\ \hline\hline
bits $\lceil \log (96) \rceil$	&	7 \\ \hline
entropy				&	4.5 \\ \hline
Huffman Code (avg.)		&	4.7 \\ \hline
Entropy (Groups of 8)		&	2.4 \\ \hline
Asymptotically approaches:	&	1.3 \\ \hline\hline
Compress			&	3.7 \\ \hline
Gzip				&	2.7 \\ \hline
BOA				&	2.0 \\ \hline
\end{tabular}
\caption{\label{entropy} Information Content of the English Language}
\end{table}

One way to measure the information content is in terms of the average
number of bits per character.  Table~\ref{entropy} shows a few ways to
measure the information of English in terms of bits-per-character.  If
we assume equal probabilities for all characters, a separate code for
each character, and that there are 96 printable characters (the number
on a standard keyboard) then each character would take $\lceil \log 96
\rceil = 7$ bits.  The entropy assuming even probabilities is
$\log 96 = 6.6$ bits/char.  If we give the characters a probability
distribution (based on a corpus of English text) the entropy is
reduced to about 4.5 bits/char.  If we assume a separate code for each
character (for which the Huffman code is optimal) the number is
slightly larger 4.7 bits/char.

Note that so far we have not taken any advantage of relationships
among adjacent or nearby characters.  If you break text into blocks
of 8 characters, measure the entropy of those blocks (based on measuring
their frequency in an English corpus) you get an entropy of about 19 bits.
When we divide this by the fact we are coding 8 characters at a time,
the entropy (bits) per character is 2.4.
If we group larger and larger blocks people have estimated that the
entropy would approach 1.3 (or lower).  It is impossible to actually
measure this because there are too many possible strings to run
statistics on, and no corpus large enough.

This value 1.3 bits/char is an estimate of the information content of
the English language. 
Assuming it is approximately correct, this bounds how much we can
expect to compress English text if we want lossless compression.
Table~\ref{entropy} also shows the compression
rate of various compressors.  All these, however, are general purpose
and not designed specifically for the English language.  The last one,
BOA, is the current state-of-the-art for general-purpose compressors.
To reach the 1.3 bits/char the compressor would surely have to ``know''
about English grammar, standard idioms, etc..

\begin{table}
\centering
\begin{tabular}{|l|l|l|l|} \hline \hline
Date		&  bpc	&	scheme	&	authors \\ \hline\hline
May 1977	&  3.94	&	LZ77	&	Ziv, Lempel \\ \hline
1984		&  3.32 & 	LZMW	&	Miller and Wegman \\ \hline
1987		&  3.30 &	LZH	&	Brent \\ \hline
1987 		&  3.24 &	MTF	&	Moffat \\ \hline
1987		&  3.18	&	LZB	&	Bell \\ \hline
.		&  2.71	&	GZIP	&	. \\ \hline
1988		&  2.48 &	PPMC	&	Moffat \\ \hline
.		&  2.47	&	SAKDC	&	Williams \\ \hline
.		&  2.47	&	PPMD	&	Howard \\ \hline
Nov 1993	&  2.34	&	PPMC	&	Moffat \\ \hline
Oct 1994	&  2.34 &	PPM$^{*}$ &	Cleary, Teahan, Witten \\ \hline
18 Nov 1994	&  2.33 &	PPMD	&	Moffat \\ \hline
1995		&  2.29 &	BW	&	Burrows, Wheeler \\ \hline
31 Jan 1995	&  2.27 &	PPMD	&	Teahan \\ \hline
1997    	&  1.99 &	BOA	&	\\ \hline\hline
\end{tabular}
\caption{\label{Calgary} Lossless compression ratios for text compression on
Calgary Corpus}
\end{table}

A more complete set of compression ratios for the Calgary corpus for a
variety of compressors is shown in Table~\ref{Calgary}.  The Calgary
corpus is a standard benchmark for measuring compression ratios and
mostly consists of English text.  In particular it consists of 2
books, 5 papers, 1 bibliography, 1 collection of news articles, 3
programs, 1 terminal session, 2 object files, 1 geophysical data, and
1 bit-map b/w image.  The table shows how the state of the art has improved
over the years.

\section{Basic Compression Algorithms}

We will first discuss codes that assign a unique bit-string for coding
each symbol (Huffman codes and Shannon Fano codes) and then
discuss Arithmetic coding which ``blends'' the codes from different
symbols.

\subsection{Prefix Codes}
  \begin{itemize}
  \item each symbol (message) is a leaf in a binary tree
  \item the code is encoded in the path from root to the leaf (left = 0, right = 1)
  \item {\em Property}: No code is a prefix of another code
  \end{itemize}

The average length of a prefix code assuming a probability
distribution on the symbols is defined as $l_a = \sum_i p_i l_i$ where
$p_i$ is the probability of the $i^{th}$ symbol and $l_i$ is the
length of its code (the depth of its leaf).

\subsection{Huffman Codes}
\index{Huffman Coding}

Huffman codes are an optimal set of prefix codes.  They are optimal
in the sense that no other prefix code will have a shorter average
length.

To build a Huffman code tree:
  \begin{itemize}
  \item start with a forest of trees each representing a symbol and with
  weight $w_i = p_{i}$
  \item Repeat until single tree
    \begin{itemize}
    \item select two trees with lowest weights ($w_{1}$ and $w_{2}$)
    \item combine into single tree with weight $w_{1} + w_{2}$
    \end{itemize}
  \end{itemize}

Properties of Huffman Coding:

  \begin{enumerate}
  \item 
The length of the code for symbol $i$ is approximately
$\log_2 (\frac{1}{p_{i}})$.
(In the original lecture I said that 
$\lceil \log_2 (\frac{1}{p_{i}}) \rceil$
is an upper bound on the length of each code.
This is actually not true, as pointed out by a student.  Consider
the probabilities $.01, .30, .34, .35$.  A Huffman coding
will give the probability .30 a length 3, 
which is $> \lceil \log_2 (1/.3) \rceil = 2$.
It is the 
last time I trust ``Data and Image Compression'' by Gilbert Held.)

  \item 
The average length $l_a$ of a Huffman code matches the entropy if 
$p_{i} = \frac{1}{2^{j}} $ where $j$ is any
integer, for all $i$.
  \end{enumerate}

\subsection{Shannon-Fano Coding}

Also a prefix code, but tree is built from the top down, as opposed to being
built from bottom-up as in the Huffman Coding.  First sort $S$ by
probabilities, then partition list of messages into two partitions, such that
the subsums of the probabilities in each partition is as close to 50\% as
possible. Each partition is assigned a branch with a 1 or a 0. Repeat process
until tree is fully built and all messages are separated into leaves.


  \begin{verbatim}
  make_codes(S)
      if |S| = 1 then S[0]
      else
         sort S by probabilities
         split S into S1 and S2 such that W(S1) approx. W(S2) 
         return node(make_code(S1), make_code(S2))
  \end{verbatim}
where {\tt W(S)} represents the sum of the probabilities in the set {\tt S}.
Unlike Huffman codes, Shannon-Fano codes do not always give optimal prefix
codes.

\subsection{Arithmetic Coding}
\index{Arithmetic Coding}

Given the probability of each symbol, we start with a probability range of 1.
Then we separate the range of 1 into discrete partitions according to the
probabilities of each symbol.  So if $P(a) = 0.6$ and $P(b) = 0.4$, then 60\%
of the range is one partition for the symbol $a$. As each symbol is read from
the input, the range is further refined as the lower and upper bounds of the
range are modified by the probability of the new symbol.  If we use $L_{i}$
and $U_{i}$ to denote the lower and upper bounds, and $l_{i}$ and $u_{i}$ to
denote the range of probability for that single symbol (i.e. $l_{a} = 0$ and
$u_{a} = 0.6$), then we have: 
\[ L_{i} = L_{i-1} + (U_{i-1} - L_{i-1}) \times l_{i} \] and
\[ U_{i} = L_{i-1} + (U_{i-1} - L_{i-1}) \times u_{i} \]


{\em Properties}:

  \begin{itemize}
  \item might not be able to emit code until seen full message (in cases
  where the range straddles 0.5, the first bit cannot be output until the
  range falls to one side of 0.5)
  \item asymptotically approaches entropy
  \item can require arbitrary precision arithmetic to implement (although
        we will see a nice trick in the next class)
  \item often used together with other methods (i.e. as a back-end)
  \end{itemize}
A solution to the straddling and precision problems is to separate
the input into fixed size blocks (e.g. 100 symbols), and 
output compressed code for each block

\subsection{Deriving Probabilities}

For Huffman, Shannon-Fano and Arithmetic coding we need probabilities.
How do we derive them.

  \begin{itemize}
  \item Static vs. Dynamic
    \begin{itemize}
    \item Static for a single document: We could measure the probabilities
           over the document we plan to send and use those.
           In this case we have to send the probabilities (or generated
           codes) to the decoder.
    \item Static for a language: We could use a pre-measured set of
           probabilities.
           In this case the decoder could already know the probabilities.
    \item Dynamic: We could update the probabilities as we are sending
           a document based on what we have seen.
           In this case the decoder can also calculate the probabilities 
           on the
           fly as it decodes the message (we don't have to send them).
    \end{itemize}

  \item Single symbols vs. Groups of Symbols
    \begin{itemize}
       \item We could measure the probability of each symbol independent
             of its context.
       \item We could measure the probability based on a context (i.e. previous symbols).  This
             is discussed further in the next class and will usually
             give much better compression.
    \end{itemize}
  \end{itemize}

\subsubsection*{PPM: Prediction by partial matching}

This is one of general-purpose algorithms that gets the best compression
ratios (although it is certainly not the fastest).
It is based on generating static probabilities over the current document
using the symbols and their context.
The algorithm can be outlined as follows.

  \begin{itemize}
  \item Try finding all multiple occurrences (matches) of strings of length
  $n$ in the message ($n$ is a preset constant).
  \item For leftovers, try to find multiple occurrences of strings of $n-1$.
  \item repeat until all possible matches are found, 
        and assign a probability to
        each string according to \# of occurrences.
  \item Send the probabilities along with the arithmetic coding of the
        message that uses the probabilities.
  \end{itemize}

\section{Lempel-Ziv Overview}

The Lempel-Ziv algorithms compress by building a table of previously
seen strings.  They do not use probabilities (a string is
either in the table or it is not).
There are two main variants based on two papers they wrote, one in
1977 (LZ77) and the other in 1978 (LZ78).

\begin{itemize}
\item
LZ78 (Dictionary Based)
  \begin{itemize}
  \item LZW:  Lempel-Ziv-Welch
  \item LZC:  Lempel-Ziv Compress
  \item LZMW:  (Unix Compress, GIF)
  \end{itemize}

\item
LZ77 (Sliding Window)
  \begin{itemize}
  \item Window instead of dictionary
  \item LZSS:  LZ-Storer-Szymanski (gzip, ZIP, many others)
  \end{itemize}
\end{itemize}

Traditionally, LZ77 is better but slower.

\subsection{LZ78 and LZW}
\index{LZ78 and LZW}

Basically starts with a string lookup table (dictionary) consisting of
the standard characters, and generates a dynamic table of new string
patterns of increasing length.  The decompression algorithm depends on
it being able to generate the same table as the compressor.

\subsubsection*{Encode:}

\begin{verbatim}
  w = NIL
  while (K = readchar()) {
    if wK in dictionary
      w = wK
    else {
        output code for w
        add wK to dictionary
        w = K }
  }
  output code for w
\end{verbatim}

\begin{table}
\begin{minipage}{.48\textwidth}
\centering
\begin{tabular}{|l|l|l|l|} \hline 
K	&	w	&	wK	&	output \\ \hline\hline
a	&		&	a	&	\\ \hline
b	&	a	&	ab(256)	&	a \\ \hline
c	&	b	&	bc(257)	&	b \\ \hline
a	&	c	&	ca(258)	&	c \\ \hline
b	&	a	&	ab	&	\\ \hline
c	&	ab	&	abc(259) &	256 \\ \hline
a	&	c	&	ca	&	\\ \hline
-	&	ca	&	-	&	258 \\ \hline
\end{tabular}
\caption{\label{encode1} LZ78 Encoding on ``abcabca''}
\end{minipage}
\begin{minipage}{.48\textwidth}
\centering
\begin{tabular}{|l|l|l|l|} \hline 
K	&	w	&	wK	&	output \\ \hline\hline
a	&		&	a	&	\\ \hline
a	&	a	&	aa(256) &	a \\ \hline
a	&	a	&	aa	&	\\ \hline
a	&	aa	&	aaa(257) &	256 \\ \hline
a	&	a	&	aa	&	\\ \hline
a	&	aa	&	aaa	&	\\ \hline
-	&	aaa	&		&	257 \\ \hline
\end{tabular}
\caption{\label{encode2} LZ78 Encoding on ``aaaaaa''}
\end{minipage}
\end{table}

Key is noting that the code for for a new string {\tt wK} is not
output until the second time it is seen.  This makes it possible for
the decoder to know what the code represents.  If in the {\tt else}
clause we output the new code for {\tt wK} instead of {\tt w} the
message would be impossible to decode.


Tables~\ref{encode1} and~\ref{encode2} show two examples of the encoder.
Note that the second example shows a contingency that must be accounted for,
when the input has a long string of the same character.  The decoder must be
able to recognize this and know that code 256 is ``aa''.  This is handled
by the {\tt else} clause below.

\subsubsection*{Decode:}

\begin{verbatim}
  w = lookup(readcode())
  output w
  while ( c = readcode()) {
    if C in dictionary
        wn = lookup(C)
    else
        wn = stringcat(w, w[0])
    output wn
    K = wn[0]
    add wK to dictionary
    w = wn
  }
\end{verbatim}

\begin{table}
\centering
\begin{tabular}{|l|l|l|l|} \hline 
input & w (at start) &	output (wn)	&	add to dict (w wn[0])\\ \hline\hline
a     &              &  a               &               \\ \hline
b     & a	     &	b		&	ab(256) \\ \hline
c     & b	     &	c		&	bc(257) \\ \hline
256   & c 	     &	ab		&	ca(258) \\ \hline
258   & ab	     &	ca		&	abc(259) \\ \hline
\end{tabular}

\vspace{.15in}

\begin{tabular}{|l|l|l|l|} \hline 
input & w (at start) &	output (wn)	&	add to dict (w wn[0])\\ \hline\hline
a     &              &  a               &               \\ \hline
256   & a	     &	aa		&	aa(256) \\ \hline
257   & aa	     &	aaa     	&	aaa(257) \\ \hline
\end{tabular}
\caption{\label{decode1} LZ78 Decoding on ``a b c 256 258'' and on
``a 256 257''}
\end{table}

Tables~\ref{decode1} shows two examples of decoding.

\subsubsection*{Properties:}
  \begin{enumerate}
  \item Dynamic: (e.g. works well with tar files which have many formats),
  adaptable.
  \item No need to send the dictionary (In LZW need to send clear signals to
  clear dictionary/string table)
  \item Can use Tries to store the dictionary/lookup tables, making lookups
  effectively executable in unit time.
  \end{enumerate}


What happens when dictionary gets too large?
  \begin{enumerate}
  \item Throw dictionary away when reaching a certain size (GIF)
  \item Throw dictionary away when not effective (Unix Compress)
  \item Throw Least Recently Used entry away when reaches a certain size
  (BTLZ - British Telecom Standard)
  \end{enumerate}

\subsection{LZ77 (Sliding Windows)}
\index{LZ77 (Sliding Windows)}

This scheme involves a finite lookahead window, and the $X$ most recently
characters encoded as the dictionary. look in the lookahead window to find
the longest string that starts at the beginning of the buffer and has already
occurred in the dictionary.  Then output:  1. position of prev. occurrence in
window, 2. length of string, 3. next char.  Then advance windows by length +
1.


Example:

\begin{verbatim}
  1 2 3 4 5 6 7 8 9 10 11  12 13 14 15
  a a c a a c a b c a  b || a  b  a  c <--- Data Source
  <-----Dictionary-----> || <--L.B.-->
   (previously encoded)    (lookahead buffer)
   Lookahead buffer encoded as <10, 3, c>
\end{verbatim}


{\em LZSS Variant}:

Don't output next char, instead there are two output formats:

(0, position, length)

(1, char)

Typically only output first format if length $\geq 3$.  Use Huffman coding to
code the outputs.

{\em Optimization}: Use Trie to quickly search for match in window (gzip).


Example:   Window size = 32
\begin{verbatim}
Input : aaaaaa
Output: (1, a)  (0, 31, 5)
Input : abcabca
Output: (1, a) (1, b) (1, c) (0, 29, 3) (1, a)
\end{verbatim}


