
\newcommand{\eps}{\epsilon}
% AdaBoost commands
\newcommand{\trainerr}[1]{\hat{\epsilon}_S\!\left(#1\right)}
\newcommand{\generr}[1]{\epsilon\!\left(#1\right)}
\newcommand{\D}{\mathcal{D}}
\newcommand{\margin}{\text{margin}}
% \newcommand{\sign}{\text{sign}}
\newcommand{\PrS}{\hat{\Pr_{(x^{(i)}, y^{(i)}) \sim S}}}
\newcommand{\PrSinline}{\hat{\Pr}_{(x^{(i)}, y^{(i)}) \sim S}}  % inline PrS


\subsection{Random Forests}

\begin{enumerate}
\item What are some downsides of decision trees, and how can we explain this in the context of the bias-variance tradeoff?

\begin{soln}
learned greedily, can overfit if depth isn't controlled, low bias but high variance 
\end{soln}

$$\text{Random Forests = Sample Bagging + Split-Feature Randomization}$$

\item What is \textbf{sample bagging}?

\vspace{0.5in}
\begin{soln}
Bagging stands for bootstrap aggregating. A bootstrapped dataset has the same number of rows as the original dataset, but its rows are drawn from the original dataset with replacement. Models are trained on each individual dataset, but since the training datasets are not as similar to each other, the learned models tend to be different and more variable as well. 
Aggregating refers to the model predictions being combined to form the final prediction. 
\end{soln}

\item What is \textbf{split-feature randomization}?

\vspace{0.5in}
\begin{soln}
In the splits for each decision tree, instead of choosing the split feature from the set of all features, we limit the feature set to a randomly chosen subset of all the features. This further reduces correlation between the learned decision trees as the ``best'' feature may not always be available to split on.
\end{soln}

\item How do these techniques affect the bias and variance of an individual tree?
\vspace{5em}
\begin{soln}
    Both increase bias and decrease variance by limiting the information available to train on, compared to a decision tree trained on the full original dataset.
\end{soln}

\item How do these techniques affect the bias and variance of an ensemble of trees?

\begin{soln}
    The increase in bias carries over from the individual trees. The ensemble has lower variance because we are aggregating predictions over a set of trees that are not fully correlated. Both techniques are designed to make the trees less correlated with one another, as reducing covariance between trees reduces variance of the average over trees.

    Consider random variables $X_1, X_2$, each with variance $\sigma^2$.
    The variance of their average is given by
    \begin{align*}
        \text{Var}(\frac{1}{2} X_1 + \frac{1}{2} X_2)
        &= \text{Var}(\frac{1}{2} X_1) + \text{Var}(\frac{1}{2} X_2) + 2 \text{Cov} (\frac{1}{2} X_1, \frac{1}{2} X_2) \\
        &= \frac{1}{4} \text{Var}(X_1) + \frac{1}{4} \text{Var}(X_2) + \frac{1}{2} \text{Cov} (X_1, X_2) \\
        &= \frac{1}{4} \sigma^2 + \frac{1}{4} \sigma^2 + \frac{1}{2} \sigma^2 \frac{\text{Cov} (X_1, X_2)}{\sigma \cdot \sigma} \\
        &= \frac{1}{4} \sigma^2 + \frac{1}{4} \sigma^2 + \frac{1}{2} \rho \sigma^2 \\
        &= \sigma^2 \frac{1}{2} (1 + \rho)
    \end{align*}
    where $\rho$ is the correlation between $X_1, X_2$.

    If $\rho = 1$ (fully correlated trees), then we achieve no variance reduction: the average has variance $\sigma^2$. In general, reducing $\rho$ makes the trees less correlated and reduces variance.

    Note that our techniques are generally not extreme enough to generate anticorrelated trees, where tree predictions would oppose each other. The more anticorrelated our predictions are, the closer our model gets to always predicting $0$, which does reduce variance, but at the cost of performance on our task.
\end{soln}

\newpage

\item For each data point $\xv^{(i)}$, define $t^{(-i)}$ to be the set of decision trees that $\xv^{(i)}$ was not used to train. Use each tree in $t^{(-i)}$ to make a prediction for $\xv^{(i)}$, and use these predictions to make an aggregated prediction $\overline{t^{(-i)}}(\xv^{(i)})$ (i.e. for classification take the majority vote). Then, we can define the \textit{out-of-bag} error as follows:
$$E_{OOB} = \frac{1}{N}\sum_{i=1}^N\mathbbm{1}\left(\overline{t^{(-i)}}(\xv^{(i)}) \neq y^{(i)}\right)$$

Why can we use $E_{OOB}$ for hyperparameter optimization even though it was calculated using training points we used to learn the decision trees with? 

\vspace{1cm}

\begin{soln}
While every point was used to train a certain set of decision trees, the calculation of $E_{OOB}$ takes advantage of the fact that the nature of bootstrapped datasets means that there will generally be a reasonably large proportion of them that do not contain any particular point. 

Therefore, for the decision trees that were trained on these datasets, the training point is equivalent to a test point as the tree has never seen it before. Since each tree is trained independently, there will never be a scenario in which a tree is both trained on a data point and also evaluated on it. 
\end{soln}

\item \textbf{Random Forest Example:} Suppose we train a random forest with two decision trees on the following dataset, using the provided bootstrap samples. Assume that for ties, we predict $Y=1$.

\begin{tabular}{|c|cccc|c|}
     \hline
     All & $X_0$ & $X_1$ & $X_2$ & $X_3$ & $Y$  \\
     \hline
     1 & 1 & 0 & 0 & 0 & 1 \\
     2 & 0 & 0 & 1 & 0 & 1 \\
     3 & 0 & 0 & 0 & 1 & 1 \\
     4 & 0 & 0 & 0 & 0 & 0 \\
     5 & 0 & 1 & 0 & 1 & 1 \\
     \hline
\end{tabular}

\begin{tabular}{|c|cccc|c|}
     \hline
     Sample 1 & $X_0$ & $X_1$ & $X_2$ & $X_3$ & $Y$  \\
     \hline
     1 & 1 & 0 & 0 & 0 & 1 \\
     4 & 0 & 0 & 0 & 0 & 0 \\
     5 & 0 & 1 & 0 & 1 & 1 \\
     \hline
\end{tabular}
\begin{tabular}{|c|cccc|c|}
     \hline
     Sample 2 & $X_0$ & $X_1$ & $X_2$ & $X_3$ & $Y$  \\
     \hline
     3 & 0 & 0 & 0 & 1 & 1 \\
     4 & 0 & 0 & 0 & 0 & 0 \\
     5 & 0 & 1 & 0 & 1 & 1 \\
     \hline
\end{tabular}

\begin{enumerate}
    \item Suppose we train our first tree on Sample 1 and the split feature randomization chooses $\{X_1, X_2\}$ for the feature candidates at the root. What feature will we split on at the root?
    \vspace{1em}
    \begin{soln}
        $X_1$
    \end{soln}

    \newpage
    
    \item Suppose we then recurse on the left child (with feature value $0$) of the root and split feature randomization chooses $\{X_0, X_2\}$ for the feature indices. What feature will we split on? 
    \vspace{2em}
    \begin{soln}
        $X_0$
    \end{soln}    

    \item Suppose we train our second tree on Sample 2 and the split feature randomization chooses $\{X_2, X_3\}$ for the feature candidates at the root. What feature will we split on at the root?
    \vspace{2em}
    \begin{soln}
        $X_3$
    \end{soln}

    \item What is the training error of the ensemble?
    \begin{soln}
        $1/5$, as only point $2$ is incorrect.
    
        Point 1: tree 1 predicts 1, tree 2 predicts 0, so prediction is 1

        Point 2: tree 1 predicts 0, tree 2 predicts 0, so prediction is 0

        Point 3: tree 1 predicts 0, tree 2 predicts 1, so prediction is 1

        Point 4: tree 1 predicts 0, tree 2 predicts 0, so prediction is 0

        Point 5: tree 1 predicts 1, tree 2 predicts 1, so prediction is 1
    \end{soln}
    \vspace{3em}

    \item What is the out of bag error of the ensemble?
    \begin{soln}
        $4/5$, as only point $5$ is correct
    
        Point 1: only tree 2 is involved, prediction is 0

        Point 2: both trees are involved, prediction is 0

        Point 3: only tree 1 is involved, prediction is 0
    
        Points 4 and 5: majority vote over 0 trees predicts 1
    \end{soln}
\end{enumerate}

\end{enumerate}

