\begin{problem}{Kernel PCA}


\begin{question}[4] PCA in high-dimensional feature spaces

Recall that for standard PCA, the principal components for a zero-centered dataset $\mathbf{X} \in \mathbb{R}^{N\times D}$ are found by eigen decomposition of its covariance matrix $\mathbf{C}$. (Note that $N$ is the number of samples, and $D$ is the number of features). 

Let $\mathbf{x}_i \in \mathbb{R}^{D}$ be the $i^{th}$ row of $\mathbf{X}$ as a column vector, and the covariance matrix can be expressed as:
\begin{gather*}
\mathbf{C} = \frac{1}{N}\sum_i^N \xv_i\xv_i^\mathrm{T}.    
\end{gather*}
 
The $j^{th}$ eigenvector (principal component) $\mathbf{u}_j$ and its corresponding eigenvalue $\lambda_j$ can be found by solving,
 \begin{align*}
    \mathbf{C}\mathbf{u}_j = \lambda_j\mathbf{u}_j
\end{align*}
Now let us consider a mapping $\phi: \mathbb{R}^D \rightarrow \mathbb{R}^M$, which projects each original data point onto a higher dimensional space ($M > D$). This space is called \textbf{feature space}. It may be possible to obtain better dimensionality reduction when PCA is applied in the (nonlinear) feature space.

First assume that the new data points after projection are also zero-centered, meaning $$\sum_i^N \phi(\xv_i) = \mathbf{0}$$ and they have $\mathbf{S}$ as the covariance matrix
$$
\mathbf{S} = \frac1{N}\sum^N_{i=1}\phi(\xv_i)\phi(\xv_i)^\top.
$$  If we do standard PCA directly in the feature space by solving the following eigen decomposition problem, $$\mathbf{S}\mathbf{v}_j = \lambda_j\mathbf{v}_j,$$
what could potentially be problematic? Please provide a brief explanation.

Hint: Think about the size of $\mathbf{S}$.  \label{eigen}

\solutionspace{5cm}{15cm}{}{\TwoA}
\end{question}


In the subsequent problems, we will derive an alternate solution using kernels where the PCA in feature space can be achieved by eigendecomposition of an $N\times N$ matrix, instead of the $M \times M$ covariance matrix $S$.

\begin{question}[6]

Show that the $j^{th}$ principal component $\mathbf{v}_j$ can be expressed as a linear combination of transformed data points. That is, there exists an $N$-dimensional vector $\mathbf{w}_j  = (w_{j1},\hdots,w_{jn},\hdots,w_{jN})^\mathrm{T}$ such that:
\[\mathbf{v}_j = \sum_{i=1}^Nw_{ji}\phiv(\xv_i) = \phiv(\mathbf{X})^\top \mathbf{w}_j\] where $\phiv(\mathbf{X}) = [\phiv(\mathbf{x}_1), \phiv(\mathbf{x}_2), ... ,\phiv(\mathbf{x}_N)]^T$ is the $N \times M$ transformed data matrix. \label{linear_combo}

Note: We only care about the eigenvalues that are non-zero.

\solutionspace{8cm}{15cm}{}{\TwoB}
\end{question}

Now you will show that the weight vector $\mathbf{w}_j, \forall j$ can be found by solving a  $N \times N$ eigen decomposition problem. The idea here is to replace the covariance matrix $\mathbf{S}$ with the kernel matrix 
$$\mathbf{K} \in \mathbb{R}^{N\times N}, \mathbf{K}_{i,j}=k(\xv_i,\xv_j) = \phiv(\xv_i)^\mathrm{T}\phiv(\xv_j),$$ 
to avoid working in the feature space directly. \label{weight}

\begin{question}[8]

Prove that any $\mathbf{w}_j$, of which the corresponding eigenvalue $\lambda_j$ is non-zero, can be obtained by solving, 
$$\mathbf{K}\mathbf{w}_j = N\lambda_j\mathbf{w}_j$$
Hint: start from the original eigen decomposition  problem in terms of ${\bf S}$ and ${\bf v}_j$ and use the results from the previous questions to arrive at the above equation. Also, you may assume $\mathbf{K}$ is invertible.

\solutionspace{8cm}{15cm}{}{\TwoC}
\end{question}

\begin{question}[6]

Notice that recovering the PC vector $\mathbf{v}_j$ from the weight vector $\mathbf{w}_j$ requires writing out the high-dimensional feature representation $\phi(\mathbf{X})$, which is problematic. In kernel PCA we avoid computing the PC vectors, and instead directly work with projections of data points onto the PC components. Show that the projection of any transformed point $\phiv(\mathbf{x})$ onto the PC $\mathbf{v}_j$ can be computed using the kernel and $\mathbf{w}_j$ only as:
$$
\phiv(\mathbf{x})^\top \mathbf{v}_j = [k(\mathbf{x},\mathbf{x}_1) \  k(\mathbf{x},\mathbf{x}_2) \dots k(\mathbf{x},\mathbf{x}_N)] \mathbf{w}_j
$$

\solutionspace{8cm}{15cm}{}{\TwoD}
\end{question}
\end{problem}