


\begin{problem}{Programming: PCA and GMM}
\textbf{Introduction}
% Welcome to the programming component of this assignment! 
% \\\\
This assignment includes an autograder for you to grade your answers on your machine. This can be run with the command: \begin{verbatim}
    python3.6 autograder.py
\end{verbatim}
The code for this assignment consists of several Python files, some of which you will need to read and understand in order to complete the assignment, and some of which you can ignore. You can download and unzip all the code, data, and supporting files from \verb |hw4_programming.zip|.
\\\\
\textbf{Files you will edit}\\
\verb |kmeans.py|
Your code to implement K-means clustering algorithm.\\\\
\verb |gmm.py| Your code to implement clustering with Gaussian mixture models.
\\\\
\textbf{Files you might want to look at}\\
\verb |util.py|
Convenience methods to generate various plots that will be needed in this assignment.\\\\
\verb |test_cases/Q*/*.py| These are the unit tests that the autograder runs. Ideally, you would be writing these unit tests yourself, but we are saving you a bit of time and allowing the autograder to check these things. You should definitely be looking at these to see what is and is not being tested. These test cases also generate plots related to the task; the plots are saved in a directory named \verb|figures|. The autograder on Gradescope may run a different version of these unit tests.
\\\\
\textbf{Files you can safely ignore}\\
\verb|autograder.py| Autograder infrastructure code.
\\\\
\textbf{Files to Edit and Submit:} You will fill in portions of \verb|kmeans.py| and \verb|gmm.py| during the assignment. You should submit this file containing your code and comments to the Programming component on Gradescope. Please do NOT change the other files in this distribution or submit any of our original files other than these files. Please do not change the names of any provided functions or classes within the code, or you will wreak havoc on the autograder.
\\\\
\textbf{Report:} Many of the sections in this programming assignment will contain questions that are not autograded. You will place the requested results in the appropriate locations within the PDF of the Written component of this assignment.
\\\\
\textbf{Evaluation:} Your assignment will be assessed based on your code, the output of the autograder, and the required contents of in the Written component.\\\\
\textbf{Academic Dishonesty:} We will be checking your code against other submissions in the class for logical redundancy. If you copy someone else's code and submit it with minor changes, we will know. These cheat detectors are quite hard to fool, so please don't try. We trust you all to submit your own work only; please don't let us down. If you do, we will pursue the strongest consequences available to us.
\\\\
\textbf{Getting Help:} You are not alone! If you find yourself stuck on something, contact the course staff for help. Office hours, recitation, and Piazza are there for your support; please use them. If you can't make our office hours, let us know and we will schedule more. We want these assignments to be rewarding and instructional, not frustrating and demoralizing. But, we don't know when or how to help unless you ask.
\newpage
\begin{question}[9]Question 1: K-means
\\
In \verb|kmeans.py|, implement the kmeans function to perform K-means clustering algorithm on the MNIST dataset. See function docstring for details.
\\\\
As you'll see in the code, we initialize the cluster centers to the first K points in the dataset. While this isn't an ideal initialization, it helps remove any randomness in the autograder.
\\\\
You may run the following command to run some unit tests on your Q1 implementation:
\begin{verbatim}
    python3.6 autograder.py -q Q1
\end{verbatim}
The autograder will also render your 784 dimensional K-means centers as images. It will save these plots as as png files in a new directory named \verb|figures|. You are required to include these some of these figures as part of the written component of this assignment. See the written component for details about which images to include.
\end{question}
\begin{question}[9]Question 2: Gaussian Mixture Models
\\
In this question, we'll be testing our GMM skills on an MNIST dataset with just the zero and one digits. Of course, since this in an unsupervised clustering task, we don't have any of the class labels for this dataset. To make our probabilistic clusting easier to visualize, we'll preprocess the MNIST dataset with PCA to compress it down to a 2-dimensional feature space.
\\\\
\textbf{PCA}\\
In \verb|gmm.py|, implement the pca function to use PCA to project data onto the first K principal components. See function docstring for details.
\\\\
You may use either \verb|numpy.linalg.eig| or \verb|numpy.linalg.svd| to compute your eigenvectors. We recommend \verb|numpy.linalg.svd| as it returns sorted arrays. You may NOT use high-level PCA solvers, for instance \verb|sklearn.decomposition.PCA|.
\\\\
\textbf{NOTE:} Eigenvectors are not unique. They can be scaled and still be correct eigenvectors. To make the autograder happy, you must pass your eigenvector matrix through \verb|gmm.consistent_scale_eigenvectors| before using them to transform your data.
\\\\
You may run the following command to run some unit tests on your Q2 PCA implementation:
\begin{verbatim}
python3.6 autograder.py -t test_cases/Q2/test1.py
python3.6 autograder.py -t test_cases/Q2/test2.py
\end{verbatim}
\textbf{EM for GMM}\\
We've already given you the high level structure for the EM algorithm for Gaussian mixture models in \verb|gmm.learn_gmm()|, which alternates between: E) updating the conditional class probabilities given the data and the parameters and M) updating the parameters given the data and the conditional class probabilities. While you shouldn't edit the core content of this \verb|learn_gmm()| function, you may want to uncomment some of the printing and plotting statements to help you better understand what is happening at each iteration.
\\\\
We've also provided a function to compute PDF of a multivariate Gaussian, \verb|gmm.gaussian_pdf()|, and a function to compute the log likelihood of the GMM, \verb|gmm.log_likelihood()|, that we are trying to maximize:
$$l(\mathbf{\theta}; \mathbf{x^{(1)}}, \mathbf{x^{(2)}}, \cdots, \mathbf{x^{(N)}}) = \sum_{i = 1}^N \log p (\mathbf{x^{(i)}} | \mathbf{\theta}) = \sum_{i = 1}^N \log \sum_{k = 1} ^ K \pi_k \mathcal{N} (\mathbf{x^{(i)}}; \mathbf{\mu_k}, \mathbf{\Sigma_k})$$
where $\pi_k = p(z_k^{(i)} = 1), \forall i$.\\\\
\textbf{E-step}\\
Implement the \verb|update_Z()| function in \verb|gmm.py|. See the function docstring for details.
\\\\
For a fixed set of Gaussian mixture model parameters, $\mathbf{\theta^{(t)}}$, update the probability that each point, $\mathbf{x^{(i)}}$, belongs to the cluster $k$:
$$p(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}}) \longleftarrow \frac{\pi_k^{(i)}\mathcal{N}(\mathbf{x^{(i)}};\mathbf{\mu_k^{(t)}},\mathbf{\Sigma_k^{(t)}})}{\sum_{j = 1} ^ K \pi_j^{(t)}\mathcal{N}(\mathbf{x^{(i)}};\mathbf{\mu_j^{(t)}},\mathbf{\Sigma_j^{(t)}})}, \forall i, k$$
\textbf{M-step}\\
Implement the \verb|update_parameters()| function in \verb|gmm.py|. See the function docstring for details.
\begin{align*}
    \pi_k^{(t+1)} &\longleftarrow \frac{\sum_{i = 1} ^ N P(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}})}{N}, \forall k\\
    \mathbf{\mu_k^{(t+1)}} &\longleftarrow \frac{\sum_{i = 1} ^ N P(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}})\mathbf{x^{(i)}}}{\sum_{i = 1}^N P(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}})}, \forall k \\
    \mathbf{\Sigma_k^{t+1}} &\longleftarrow \frac{\sum_{i = 1}^N P(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}})(\mathbf{x^{(i)}} - \mathbf{\mu_k^{(t+1)}})(\mathbf{x^{(i)}} - \mathbf{\mu_k^{(t+1)}})^T}{\sum_{i = 1}^N P(z_k^{(i)} = 1 | \mathbf{x^{(i)}}, \mathbf{\theta^{(t)}})}, \forall k
\end{align*}
You may run the following command to run some unit tests on your Q2 implementation:
\begin{verbatim}
python3.6 autograder.py -q Q2
\end{verbatim}
The autograder will also save various plots of PCA and GMM results as png files in a new directory named \verb|figures|. You are required to include these some of these figures as part of the written component of this assignment. See the written component for details about which plots to include.
\\\\
\textbf{Submission}\\
Complete all questions as specified in the above instructions. Then upload \verb|kmeans.py| and \verb|gmm.py| to Gradescope. Your submission should finish running within 20 minutes, after which it will time out on Gradescope.
\\\\
Don't forget to include any request results in the PDF of the Written component, which is to be submitted on Gradescope as well.
\\\\
You may submit to Gradescope as many times as you like. You may also run the autograder on your own machine to speed up the development process. Just note that the autograder on Gradescope will be slightly different than the local autograder. The autograder can be invoked on your own machine using the command:
\begin{verbatim}
    python3.6 autograder.py
\end{verbatim}
Note that running the autograder locally will not register your grades with us. Remember to submit your code when you want to register your grades for this assignment.
\\\\
The autograder on Gradescope might take a while but don't worry: \textbf{so long as you submit before the deadline, it's not late}.
\end{question}
\newpage 
The following questions should be completed after you work through the programming portion of this assignment.

\begin{question}[12] PCA

Include the plots of the toy dataset before and after running PCA with K=2.

\solutionspace{8cm}{16.5cm}{PCA before and after:}{\FourPCAToy}

Include the plots of the MNIST zeros and ones dataset after running PCA with K=2.

\solutionspace{8cm}{8cm}{PCA MNIST:}{\FourPCAMNIST}

\end{question}

\newpage

\begin{question}[8] GMM Toy Datasets

Include the plots after learning the GMM parameters for K=2 on toy dataset one and two.

\solutionspace{8cm}{8cm}{GMM Toy 1, K=2:}{\FourGMMToyOne}
\quad
\solutionspace{8cm}{8cm}{GMM Toy 2, K=2:}{\FourGMMToyTwo}

\end{question}

\begin{question}[8] GMM MNIST Zeros and Ones

Include the plots after learning the GMM parameters for K=2 and K=5 on the MNIST zeros and ones dataset.

\solutionspace{8cm}{8cm}{GMM MNIST, K=2:}{\FourGMMMNISTKTwo}
\quad
\solutionspace{8cm}{8cm}{GMM MNIST, K=5:}{\FourGMMMNISTKFive}

\end{question}

\end{problem}

\begin{problem}{Programming: K-means}

The following questions should be completed after you work through the programming portion of this assignment.

\begin{question}[6] K=2

Include the images of the cluster centers after running k-means with two clusters.

\solutionspace{5cm}{16.66cm}{Centers K=2:}{\ThreeKTwo}

\end{question}

\begin{question}[6] K=5

Include the images of the cluster centers after running k-means with five clusters.

\solutionspace{5cm}{16.66cm}{Centers K=5:}{\ThreeKFive}

\end{question}

\begin{question}[6] K=10

Include the images of the cluster centers after running k-means with ten clusters.

\solutionspace{5cm}{16.66cm}{Centers K=10:}{\ThreeKTen}

\end{question}

\end{problem}