% MODIFY THE FOLLOWING TO INCLUDE THE LECTURE #, TOPIC, DATE, and YOUR NAME
\lecturetitle{25}{Guest lecture Eric Brewer on HotBot}{26 Nov 97}{Carleton Miyamoto}

% INCLUDE AN OUTLINE OF THE CLASS TOPICS
\begin{itemize} 
  \item Comment on Assignment (Guy)
  \item Goals of HotBot
  \item Internal Architecture
    \begin{enumerate}
    \item Semantics
    \item Hardware
    \item Data Base Model
    \item Score
    \item Filters
    \end{enumerate}
  \item Optimizations
    \begin{enumerate}
    \item Caching
    \item Compression
    \item Parallelism
    \item Misc.
    \end{enumerate}
  \item Crawling
  \item Summary
  \item References
\end{itemize}

\section{Comment on Assignment (Guy)}

\subsubsection*{Problem 3}

Problem 3 in the last assignment could be done by combining the affine gap and
space efficiency examples given in class and in the readings.  There, however,
were a few caveats which needed to be addressed.

First, the algorithm must keep separate pointers for the E and S entries. This
is necessary because the path could have come from an E or S crossing.  The
F entries can be generated on the fly.

% gapcrs.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(2574,2349)(1939,-3148)
\thicklines
\put(1951,-2086){\line( 1, 0){2550}}
\put(2026,-961){\line( 5,-3){375}}
\put(2401,-1186){\line( 0,-1){300}}
\put(2401,-1486){\line( 2,-1){300}}
\put(2701,-1636){\line( 5,-4){375}}
\put(3076,-1936){\line( 3,-5){225}}
\put(3301,-2311){\line( 3,-2){450}}
\put(3751,-2611){\line( 0,-1){300}}
\put(3751,-2911){\line( 5,-1){677.885}}
\put(2776,-2536){\line( 6, 5){450}}
\put(1951,-3136){\framebox(2550,2325){}}
\put(3526,-1636){\line(-6,-5){450}}
\put(3526,-2086){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Middle Row}}}
\put(2251,-1111){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Path}}}
\put(3301,-1561){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Gap starts}}}
\put(2401,-2686){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Gap ends}}}
\end{picture}
\caption{A gap may cross the middle row}
\label{fig:gapcrs}
\end{figure}
% gapcrs.fig Figure END

Second, gaps which cross the middle row must be dealt with carefully
(Fig. \ref{fig:gapcrs}).  If not the algorithm could double charge for that
gap.  A solution is to store both pointers back to the middle row for both E
and S.

% esmtx.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(2637,2895)(1939,-3148)
\thicklines
\put(2026,-3061){\framebox(2400,300){}}
\put(1951,-1861){\line( 1, 0){2550}}
\put(1951,-2086){\line( 1, 0){2550}}
\put(2026,-736){\framebox(2400,300){}}
\put(1951,-3136){\framebox(2550,2325){}}
\put(4276,-586){\line(-1,-3){472.500}}
\multiput(3826,-2011)(-3.75000,7.50000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3751,-1861){\line( 3,-2){225}}
\put(3976,-2011){\line(-1, 0){150}}
\put(2476,-361){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Pointers for last row}}}
\put(3901,-586){\line(-3,-5){866.912}}
\put(3001,-2011){\line( 0, 1){150}}
\multiput(3001,-1861)(6.00000,-6.00000){26}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3151,-2011){\line(-1, 0){150}}
\put(3001,-2011){\line( 0, 1){ 75}}
\put(2626,-586){\line(-1,-5){285.577}}
\multiput(2326,-2011)(-3.75000,7.50000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(2251,-1861){\line( 3,-1){225}}
\multiput(2476,-1936)(-7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(4576,-2011){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Middle Row}}}
\put(2851,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Last Row}}}
\end{picture}
\caption{Keep two arrays for each of S and E}
\label{fig:esmtx}
\end{figure}
% esmtx.fig Figure END

Only four arrays of size O(m) are required in a space efficient algorithm (see
Fig. \ref{fig:esmtx}).  They hold E, S, and the pointers pointing back to the
location in the middle row from where E and S came.

\section{Goals of HotBot}

\subsubsection*{History}

HotBot originated in the Berkeley NOW project.  An application was needed that
would show the benefits of clustering.  A search engine was decided upon as
the killer app. because its requirements closely matched what cluster computing
could provide.

\subsubsection*{Availability}

In order to achieve 24x7 uptime, a mission critical infrastructure is
necessary.  Fault tolerance and redundancy are cornerstones in achieving this
goal.  The multiple individual machines in clusters provide a natural method.

\subsubsection*{Scalability}

Additionally, a search engine must expand its service as client requests grow.
Since growth in the WEB, however, is occurring at a staggering rate,
big, expensive machines, quickly would become overloaded, with replacement the
only upgrade option.  So, incremental scalability could provide a large
economic advantage over the longer term.

Not only the hardware, but the software design must take all this into
account.  Fine grained parallelism and multi-processor support should adapt
well in this environment.

Both the availability and scalability requirements for a search engine could be
addressed by the NOW clusters.

\section{Internal Architecture}

\subsection{Semantics}

The semantics of a search engine are simple.  Although the search engine itself
must be highly available, that data it contains does not.  The WEB can always
be used as a last recourse to retrieve any requested information.

Additionally, a very weak consistency model can be maintained. Known
as BASE~\cite{Fox97}, it allows stale information to persist for a
limited time.

\subsection{Hardware}

Currently 330 UltraSPARC CPUs power HotBot.  The system is very homogeneous,
which allows a node to take over for an arbitrary one when necessary.  These
are connected to a vast number of RAID disk subsystems which contain over 0.5TB
of data.

If a single node goes down, the data associated with that node becomes
temporarily unavailable.  This is okay due to the weak consistency model.
Also, this constitutes only a small fraction of the total data.  Note that the
system as a whole, however, continues to process requests despite this.

In comparison to a single, large, powerful SMP system, HotBot's commodity
parts are faster and cost less.  For example, in order to expand the service,
additional, ordinary systems can be bought, adding their power to the current
set.  SMP's, however, are not additive in this respect, nor do they provide
any form of fault tolerance in system up-time.  At the very least,
replication is necessary, requiring additional, expensive hardware.

\subsection{Data Base Model}

The search engine algorithms can be described in relational data base (RDBMS)
terms.  There exists a relation w (a row in a table) for each unique word in
each document found in the WEB: w[document ID, score, position information].
The entry keeps track of the document in which the word was found (using a
unique ID).  The score helps in matching the word to a search and is discussed
below.  The position information helps in phrase searching.  There are
currently about 10M words in HotBot's database with an average of about 1K
lines per word.  HotBot has to deal with some words which occur in almost every
document and with many words that occur very infrequently.

There is also a relation D for all documents on the WEB where:
D[ID, URL, etc ... ].  This identifies each document with a unique ID, as well
as information to retrieve the document later.

When the RDBMS needs to be updated with fresh data, an atomic swap operation
swaps a new data base to replace the old between queries.  This allows hot
updates to occur without any interruption of service.

% db.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(2949,3099)(4264,-3373)
\thicklines
\put(4351,-2461){\line( 0,-1){150}}
\put(4426,-2461){\line( 0,-1){150}}
\put(6676,-2761){\line( 1, 0){225}}
\put(6751,-2761){\line( 0,-1){150}}
\put(6826,-2761){\line( 0,-1){150}}
\put(6901,-3136){\line(-1,-1){225}}
\put(6676,-3361){\line( 0, 1){225}}
\put(6676,-3136){\line( 1,-1){225}}
\put(6676,-2536){\line( 0, 1){225}}
\put(6676,-2311){\line( 1,-1){225}}
\put(6901,-2536){\line( 0, 1){225}}
\put(6901,-2311){\line(-1,-1){225}}
\put(7201,-2536){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Equi-join}}}
\put(7201,-2911){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}List}}}
\put(7201,-3286){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Semi-join}}}
\put(4576,-3361){\line( 0, 1){225}}
\put(4576,-3136){\line( 1,-1){225}}
\put(4801,-3361){\line( 0, 1){225}}
\put(4801,-3136){\line(-1,-1){225}}
\put(4726,-3136){\line( 0, 1){375}}
\put(5401,-1561){\line(-1,-1){225}}
\put(5176,-1786){\line( 0, 1){225}}
\put(5176,-1561){\line( 1,-1){225}}
\put(4276,-2461){\line( 1, 0){225}}
\put(6301,-511){\line( 0, 1){225}}
\put(6301,-286){\line( 1,-1){225}}
\put(6526,-511){\line( 0, 1){225}}
\put(6526,-286){\line(-1,-1){225}}
\put(6676,-436){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id = id}}}
\put(5551,-1486){\line( 1, 1){300}}
\put(6001,-886){\line( 5, 4){375}}
\put(6451,-586){\line( 2,-1){750}}
\put(4747,-2336){\line( 6, 5){531.148}}
\put(5326,-1861){\line( 4,-3){600}}
\put(4651,-2611){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id, f}}}
\put(4951,-3286){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id = id}}}
\put(5551,-1711){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id = id}}}
\put(5551,-1111){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Select top 10}}}
\put(7201,-1186){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Documents}}}
\put(6001,-2611){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}filter}}}
\put(6676,-2086){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Key:}}}
\end{picture}
\caption{An example of a search query}
\label{fig:dbqueries}
\end{figure}
% db.fig Figure END

When a search query is received by HotBot, it goes through a series of
relational joins.  For example, if the query ``A B filter:C'' is received
(i.e. search for A and B, then apply a boolean filter on the output based on
C), Fig. \ref{fig:dbqueries} shows the relational operations.  The first
equi-join (lower left) is doing the search for A (think of it as an ``and''
operation).  It then creates a list of the result, sorted by score (f).  The
semi-join then executes the boolean filter on the list.  The top ten scores
are selected and a final equi-join is done to get the associated documents for
the result page to return to the client.

\subsection{Score}

The score (f) determines when a word matches a query and how closely.  Higher
scores (weights) cause the document to appear higher on HotBot's priority
lists to the RDBMS joins and on the result page returned to the client.
The value is calculated as a function of a few different attributes, which
includes:
\begin{enumerate}
\item Number of times a word appears in the document
\item Document length
\item If the word appears in the document title
\end{enumerate}
This function is then broken down into equal area slices, and a ``score'' from
0 to 255 is assigned to each slice.  See Fig. \ref{fig:freq} for an example
of how the function and slices could appear.

% freq.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(5787,2514)(1501,-3688)
\thicklines
\multiput(3076,-3061)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3226,-3136){\line( 1, 0){ 75}}
\put(3301,-3136){\line( 1, 0){ 75}}
\multiput(3376,-3136)(7.50000,3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3526,-3061){\line( 0, 1){150}}
\put(3526,-2911){\line( 0, 1){150}}
\multiput(3526,-2761)(3.75000,7.50000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3601,-2611){\line( 0, 1){150}}
\multiput(3601,-2461)(6.25000,6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3676,-2386){\line( 0, 1){ 75}}
\put(3676,-2311){\line( 1, 3){ 75}}
\put(3751,-2086){\line( 0, 1){ 75}}
\multiput(3751,-2011)(3.75000,7.50000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(3826,-1861)(7.50000,3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(3976,-1786){\line( 1, 0){225}}
\multiput(4201,-1786)(6.25000,-6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(4276,-1861)(6.25000,-6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(4351,-1936)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(4501,-2011)(6.25000,-6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(4576,-2086)(6.00000,-6.00000){26}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(4726,-2236){\line( 1, 0){150}}
\multiput(4876,-2236)(6.00000,-6.00000){26}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(5026,-2386){\line( 1, 0){150}}
\multiput(5176,-2386)(6.25000,-6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(5251,-2461)(6.25000,-6.25000){13}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(5326,-2536)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(5476,-2611)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(5626,-2686)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\multiput(5776,-2761)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(5926,-2836){\line( 2,-1){300}}
\put(6226,-2986){\line( 0, 1){  0}}
\put(6226,-2986){\line( 1, 0){ 75}}
\put(6301,-2986){\line( 3,-1){225}}
\put(6526,-3061){\line( 1, 0){ 75}}
\multiput(6601,-3061)(7.50000,-3.75000){21}{\makebox(6.6667,10.0000){\SetFigFont{7}{8.4}{rm}.}}
\put(6751,-3136){\line( 1, 0){ 75}}
\put(6826,-3136){\line( 0, 1){  0}}
\put(6826,-3136){\line( 1, 0){300}}
\multiput(3451,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\multiput(3751,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\multiput(3976,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\multiput(4201,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\put(3001,-1186){\line( 0,-1){2100}}
\put(3001,-3286){\line( 1, 0){4275}}
\multiput(4501,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\put(4651,-3661){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Score = f}}}
\multiput(4951,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\multiput(5776,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\multiput(6676,-1261)(0.00000,-13.50000){151}{\makebox(6.6667,10.0000){\SetFigFont{10}{12}{rm}.}}
\put(1501,-2236){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Frequency of word}}}
\end{picture}
\caption{Determining the score}
\label{fig:freq}
\end{figure}
% freq.fig Figure END

\subsection{Filters}

As shown in an example above, filters allow the query to express boolean
matches.  In fact, filters are implemented as meta-words, where w contains all
documents that would match the specific filter.  As a result, a filter can
be thought of as the addition of this meta-word on the search line.  Filters
are use-full in expressing searches on features of pages that cannot be
expressed otherwise (such as does the page contain an image).  Note, however,
that less-than and greater-than operators cannot be easily addressed by this
mechanism.  Dates, however, are important and do require some type of
less-than, greater-than functionality.  HotBot uses a trick which expands a
date query into log(n) boolean queries, where ``n'' is the number of bits in
the date specification.

\section{Optimizations}

In order to achieve better throughput of query transactions, optimizations
help to streamline the search process.

\subsection{Caching}

Instead of just caching documents, HotBot also caches search results.  In fact,
the ``next 10'' button is actually another full search that completes quickly
because of the cache.

\subsection{Compression}

Extensive compression helps to minimize the footprint of the data.  Score and
position information are split with IDs stored in both lists.  Compression
then allows the position information, which is used infrequently, to occupy
minimal space.  A type of delta-encoding is used (similar to Huffman) that
uses a variable number of bits.

In HotBot, because of the large amount of data, nothing is
uncompressed, even when loaded directly into memory, until necessary.
The reasoning is that the additional page faults that would be
generated by uncompressing when loading into memory outweighs the time
to uncompress the data more often~\cite{Douglis93,Banks}.

\subsection{Data Base}

HotBot uses a technique called join selectivity to speed its RDBMS operations.
It orders operations which have the highest probability of resulting in a
short list first.  This reduces the work of future RDBMS operations.  For
example, applying very specific boolean filters first could reduce list sizes
for later equi-joins.

In addition, HotBot spreads the RDBMS data across many nodes in the cluster (by
document ID in the D relation) at a relatively fine grain.  Single node failures become manageable and the amount of data affected minimal.  Also, when
updating, the fine grains allow atomic swaps of the data to be simple and
require little additional space.

\subsection{Parallelism}

When optimizing for parallelism, ``balance'' is the key word.  If all nodes
are working to their fullest, and there are no single bottlenecks, this allows
for the highest degree of parallelism.  HotBot attempts to address this by:
\begin{enumerate}
\item Randomizing data across the nodes
\item Use a large number of disks to hide seek latency
\end{enumerate}
The randomizing reduces the risk of hot spots forming within the cluster
(similar to the arguments in~\cite{Wald94}).

% par.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(2499,3624)(3889,-7498)
\thicklines
\put(3976,-6586){\line( 0,-1){150}}
\put(4051,-6586){\line( 0,-1){150}}
\put(4201,-7486){\line( 0, 1){225}}
\put(4201,-7261){\line( 1,-1){225}}
\put(4426,-7486){\line( 0, 1){225}}
\put(4426,-7261){\line(-1,-1){225}}
\put(4351,-7261){\line( 0, 1){375}}
\put(4276,-6736){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id, f}}}
\put(4576,-7411){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id = id}}}
\put(5476,-4111){\line( 0, 1){225}}
\put(5476,-3886){\line( 1,-1){225}}
\put(5701,-4111){\line( 0, 1){225}}
\put(5701,-3886){\line(-1,-1){225}}
\put(5176,-4486){\line( 5, 4){375}}
\put(5626,-4186){\line( 2,-1){750}}
\put(6376,-4786){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Documents}}}
\put(3901,-6586){\line( 1, 0){225}}
\put(5851,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}id = id}}}
\put(4726,-4711){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Select top 10}}}
\put(5176,-5386){\line( 0,-1){525}}
\put(5251,-5386){\line( 3,-5){311.029}}
\put(5101,-5386){\line(-1,-3){150}}
\put(5326,-5311){\line( 1,-1){450}}
\put(5026,-5311){\line(-1,-1){600}}
\put(4351,-6286){\line( 0,-1){225}}
\put(5176,-4786){\line( 0,-1){225}}
\put(5026,-5311){\makebox(0,0)[lb]{\smash{\SetFigFont{25}{30.0}{rm}U}}}
\put(3901,-6211){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Select top 10}}}
\end{picture}
\caption{Parallelization of queries}
\label{fig:parqueries}
\end{figure}
% par.fig Figure END

HotBot also executes each query in parallel.  Because each node is responsible
for part of the total database, it first issues the query to each node
(Fig. \ref{fig:parqueries}).  The nodes then send the highest scoring subset
of their results back to a ``master'' node.  This then takes the union of the
answers, returns the top scores among all of the nodes responding, then does
the final equi-join to associate the answers with the documents.  Note that
every query utilizes every node.

% fan.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(3502,2083)(1250,-3086)
\thicklines
\put(2251,-2911){\circle{336}}
\put(3151,-2911){\circle{336}}
\put(1426,-2911){\circle{336}}
\put(4576,-2911){\circle{336}}
\put(2401,-1711){\line( 0, 1){450}}
\put(2326,-2236){\line(-5,-3){750}}
\put(2401,-2311){\line(-1,-5){ 75}}
\put(2401,-2011){\circle{336}}
\put(2551,-2236){\line( 1,-1){450}}
\put(4501,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}N}}}
\put(2626,-2161){\line( 3,-1){1777.500}}
\put(2626,-1936){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Master}}}
\put(3601,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}..............}}}
\put(1951,-1111){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}TCP connection}}}
\put(1426,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}1}}}
\put(2251,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}2}}}
\put(3151,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}3}}}
\end{picture}
\caption{Naive fanout}
\label{fig:fanout}
\end{figure}
% fan.fig Figure END

Another problem is how the fanout from the original master node, that
initially receives the query, happens as the queries are being distributed
(Fig. \ref{fig:fanout}).  A naive solution would be to send the queries in
sequence starting from the first node.  This, however, could lead to hot spots
or head of line blocking (queries are artificially blocked on a resource due
to a previous query) occurring in the fanout chain.

% fanopt.fig Figure START
\begin{figure}
\setlength{\unitlength}{0.00083300in}%
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(4313,3945)(514,-4948)
\thicklines
\put(2251,-2911){\circle{336}}
\put(1426,-2911){\circle{336}}
\put(4576,-2911){\circle{336}}
\put(976,-3961){\circle{336}}
\put(1876,-3961){\circle{336}}
\put(2833,-3961){\circle{336}}
\put(4651,-3961){\circle{336}}
\put(3151,-2911){\circle{336}}
\put(2401,-1711){\line( 0, 1){450}}
\put(2326,-2236){\line(-5,-3){750}}
\put(2401,-2311){\line(-1,-5){ 75}}
\put(2551,-2236){\line( 1,-1){450}}
\put(2626,-2161){\line( 3,-1){1777.500}}
\put(1351,-3136){\line(-1,-2){300}}
\put(1501,-3136){\line( 3,-5){363.971}}
\put(1576,-3136){\line( 2,-1){1200}}
\put(901,-4186){\line(-3,-5){397.059}}
\put(2401,-2011){\circle{336}}
\put(976,-4186){\line( 1,-5){150}}
\put(4651,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(1126,-4186){\line( 5,-6){510.246}}
\put(2701,-4186){\line(-3,-4){450}}
\put(2776,-4186){\line( 0,-1){675}}
\put(2926,-4186){\line( 2,-3){450}}
\put(4576,-3136){\line( 1,-6){111.487}}
\put(2626,-1936){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}Master}}}
\put(3601,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}..............}}}
\put(1951,-1111){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}TCP connection}}}
\put(3451,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}..................}}}
\put(1426,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(2251,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(3151,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(4576,-2986){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(976,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(1876,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\put(2851,-4036){\makebox(0,0)[lb]{\smash{\SetFigFont{12}{14.4}{rm}?}}}
\end{picture}
\caption{Optimized fanout (each ? is a random node)}
\label{fig:optfanout}
\end{figure}
% fanopt.fig Figure END

HotBot attempts to avoid problems by using two techniques
(Fig. \ref{fig:optfanout}):
\begin{enumerate}
\item Fanout in multiple levels (a fat tree structure)
\item At each stage, fanout to random nodes
\end{enumerate}

In the future, HotBot could take advantage of knowledge of the physical
network layout.  A more optimized fanout may know where the limiting bisection
bandwidths between cut-sets of the network are located and incur less traffic
over those connections.

\subsection{Misc.}

Some additional optimizations are done to speed up each query:
\begin{enumerate}
\item Precomputation: As much information as possible is computed off-line.
For example, the scores can be computed and stored as the pages are read in
from the WEB.  Also, filters (i.e. meta-words) can be created and managed
before queries requesting the information arrive.
\item Partitions of the nodes are updated at varying times.  This gives a
better illusion that the data is up-to-date.  Some data, such as news groups
are updated more frequently.
\item HotBot will dynamically skip words that are too expensive to compute.
This usually occurs with words that appear in many documents, such as ``a''
or ``the''.
\end{enumerate}

\section{Crawling}

HotBot has a separate partition that crawls the WEB at about 10M pages a day.
It uses ``popular'' sites (such as possibly Yahoo) as the roots of these
searches.  These updates eventually enter the HotBot database at various
times.

\section{Summary}

The benefits gained from clustering matched the expectations placed on search
engines well.  Their availability, scalability, and price/performance ratio
cannot be achieved even by more expensive SMP servers.

\begin{thebibliography}{4}

\bibitem[1]{Fox97}
A. Fox, S. Gribble, Y. Chawathe, E. Brewer, and P. Gauthier,
{\em Cluster-Based Scalable Network Services},
(To appear at SOSP-16, Oct. 1997).

\bibitem[2]{Douglis93}
Fred Douglis. 
{\em The Compression Cache: Using On-line Compression to Extend
Physical Memory.}
USENIX Proceedings, Winter 1993.

\bibitem[3]{Banks}
D. Banks, M Stemm. 
{\em Investigating Virtual Memory Compression on Portable
Architectures.}
\verb|http://HTTP.CS.Berkeley.EDU/~stemm/classes/cs262.ps.gz|

\bibitem[4]{Wald94}
Waldspurger and Weihl. 
{\em Lottery Scheduling: Flexible Proportional-Share
Resource Management.}  
Proceedings of the First Symposium of Operating System
Design and Implementation, November 1994.

\end{thebibliography}

