\documentclass[11pt]{article}
\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{url}
\usepackage{verbatim}
\usepackage{moreverb}
\let\verbatiminput=\verbatimtabinput
\def\verbatimtabsize{4\relax} 
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

\title{BigML Assignment 4: Naive Bayes and Phrases on Hadoop}
\author{Due Tuesday, February 28, 2012 via Blackboard}
\date{Out February 14, 2012}
\begin{document}
\maketitle

\section{Important Note}
As usual, {\bf you are expected to use Java for this assignment}.  

{\bf This assignment is worth a total of 200 points} because there are conceptually two assignments here: the Hadoop implementation of Naive Bayes (Assignment 2) and the Hadoop implementation of Phrase finding (Assignment 3).  You may use your ``one missed assignment'' credit (if you have not used it already) for one of the two parts, but not both.

Alona Fyshe (afyshe@cs.cmu.edu) is the contact TA for this homework.  Please post clarification questions to the Google Group:  \\
\url{machine-learning-with-large-datasets-10-605-in-spring-2012}


\section{Introduction}
For this assignment, you will re-implement Phrase finding and Naive Bayes (Assignments 2 and 3) for the Hadoop Mapreduce framework.  You are free (and encouraged) to reuse your code from those assignments.

\section{Getting started}
\subsection{Installing Hadoop}
We've put together a guide to installing Hadoop on your computer:
\url{http://www.cs.cmu.edu/~afyshe/Mapreducenotes.pdf}

\subsection{Using AWS}
We will be distributing AWS credits soon.  Once you receive your accounts, you can proceed with this section.  Before the accounts are assigned, you can work on the assignment using a local copy of Hadoop.  In fact, developing and debugging your Mapreduce code using a local Hadoop installation is much easier and more time efficient.

Here are a few hints for running jobs on AWS:
\subsubsection{Submitting a Jar job}
The easiest way to submit jobs to AWS is via the commandline.  You need to install the elastic mr command line interface - there are many versions for many different languages.  I used the ruby one and found it to be painless to install/use

\url{http://aws.amazon.com/code/Elastic-MapReduce/2264}

You can also start a job with the AWS console's web interface  - go to the elastic mapreduce tab in your AWS console, click create new jobflow, choose the custom JAR jobtype and the rest is pretty self-explanatory.  I find this takes longer than the command line, but if you�re more comfortable with a GUI, that�s fine.

\subsubsection{Viewing job progress}
Tutorial for viewing the jobtracker on your local machine (via proxy)

\url{http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingtheHadoopUserInterface.html}

(You can also ssh into the machine using the command line interface, and then use the lynx commands in the login preamble to view the job tracker)

\section{The Data}

We are using the data from Assignments 2 \& 3.  They are still on AFS, and they are also in this public s3 bucket:
\url{s3://bigml-shared/phrases} and  \url{s3://bigml-shared/webdocs}

The data format is the same as for Assignments 2 \& 3, with one small change:  The phrase data has format:
\begin{verbatim}
<text>\t<year>\t<text>\t<year>\t<count>
\end{verbatim}
Where text is the unigram/bigram, count is the number of times the text occurred in books during the decade indicated by year.  This change is a byproduct of changing a SequenceFile to a TextFile.  If you ignore the first two fields of the tab-separated line, the format is the same as you saw for Assignment 3.


\section{One Small Change}
In the interest of freshness, this time, use the bigrams/unigrams from the 1960s as your corpus, and those from 1970-1990 as the background corpus.

\section{Deliverables}

Submit a compressed archive (zip, tar, etc) of your code.  With it, include a makefile so that 
\begin{enumerate}

\item The command \emph{make nb}  will start a hadoop job \emph{locally}, assuming that there is data \emph{on hdfs} at the following two locations:
\begin{verbatim}
/user/demo/dbpedia/abstract.tiny.train
/user/demo/dbpedia/abstract.tiny.test
\end{verbatim}
  Your code should
  train and test a classifier using those files and your Hadoop implementation of NaiveBayes.  The final output should be written to stdout, with the same format as Assignment 2.
  \item The command \emph{make phrases}  will start your hadoop pipeline \emph{locally}, assuming that there is data \emph{on hdfs} in the following locations
\begin{verbatim}
/user/demo/phrases/unigram_select_60_90_lower_apple/
/user/demo/phrases/bigram_select_60_90_lower_apple/
\end{verbatim}
Your code will then compile counts from scratch (i.e. perform all of the steps of the pipeline) and identify interesting phrases.  The top 20 phrases sorted by \emph{total score} should be written to stdout as in Assignment 3.
\end{enumerate}

For this assignment, please also include the controller and syslog files {\bf from AWS} for each of the mapreduce jobs in both of your pipelines. The controller and syslog files can be downloaded from the AWS console.  Simply go to the Elastic Mapreduce tab.  Click your job in the list, and then the debug button, the syslog link, and save the resulting file.  With your code and a makefile, please include a pdf document with answers to the questions below.
  
\paragraph{Part A: Naive Bayes}
\begin{enumerate}
 \item Compare the single machine implementation of Naive Bayes you submitted in Assignment 2 and your new Hadoop version. Show  the speed and accuracy you got for the two big data sets (abstract and links).
 \item Tell us about your experience using Hadoop for this portion of the assignment.  Was it easy to port your code over from the Assignment 2?  Is there anything that would have made the process easier?
\end{enumerate}
For the Naive Bayes pipeline, include the controller and syslog files created when you process the full abstracts dataset.  Create a folder called \texttt{naivebayes}.  In it, put the controller and syslog files from each of your Mapreduces, named syslogN and controllerN for $N={1..n}$ where $n$ is the number of Mapreduce jobs in your pipeline. Create a README file briefly describing each of the steps of your pipeline, numbered in the same order as the controller/syslog files.

\paragraph{Part B: Phrase Finding}
\begin{enumerate}
\item What are the top 20 phrases (sorted by  \emph{total score}) from the \emph{full data set only} (again, with the 1960s as the foreground corpus this time).
 \item What do your phrase finding results tell you about the 1960s?  Are there any interesting phrases?
 \item Tell us about your experience using Hadoop for this portion of the assignment.  Was it easy to port your code over from the Assignment 3?  Is there anything that would have made the process easier?
 \end{enumerate}
 For the Phrases pipeline, include the controller and syslog files created when you process the full dataset.   Create a folder called \texttt{phrases}.  In it, put the controller and syslog files from each of your mapreduces, named syslogN and controllerN for $N={1..n}$ where $n$ is the number of mapreduce jobs in your pipeline. Create a README file briefly describing each of the steps of your pipeline, numbered in the same order as the controller/syslog files.
 
 


%\paragraph{Naive Bayes}
% For the Naive Bayes pipeline, include the logs created when you process the full abstracts dataset.  Create a folder called \texttt{naivebayes}.  In it, put the controller and syslog files from each of your Mapreduces, named syslogN and controllerN for $N={1..n}$ where $n$ is the number of Mapreduce jobs in your pipeline. Create a README file briefly describing each of the steps of your pipeline, numbered in the same order as the controller/syslog files.
%
%\paragraph{Phrases}
%For the Phrases pipeline, include the logs created when you process the full dataset.   Create a folder called \texttt{phrases}.  In it, put the controller and syslog files from each of your mapreduces, named syslogN and controllerN for $N={1..n}$ where $n$ is the number of mapreduce jobs in your pipeline. Create a README file briefly describing each of the steps of your pipeline, numbered in the same order as the controller/syslog files.


\section{Marking breakdown}
\subsection{Naive Bayes}
\begin{itemize}
\item Code correctness and makefile functionality {\bf [75 points]}.
\item Part A: Question 1 {\bf [10 points]} 
\item  Part A: Question 2 {\bf [5 points]} 
\item Controller/Syslog files  {\bf [10 points]} 
\end{itemize}


\subsection{Phrase Finding}
\begin{itemize}
\item Code correctness and makefile functionality {\bf [70 points]}.
\item Part B: Question  1 {\bf [10 points]} 
\item  Part B: Question 2  {\bf [5 points]} 
\item  Part B: Question 3 {\bf [5 points]} 
\item Controller/Syslog files  {\bf [10 points]} 
\end{itemize}

\section{Hints/Good to know}
You may consider using the GenericOptionsParser for passing command line arguments - it parses all arguments that have the form \texttt{-D name=value} and turns them into name/value pairs in your Configuration object.
\end{document}  