Regression
Build a function that maps $X$ to a real value $Y$.
- Linear regression
- Nonlinear/Nonparametric regression (Tree regression, Kernel, etc.)
Linear Reression
- For linear regression minimizing least-square loss is equivalent to maximizing likelihood, with the assumption that the error term is zero-mean gaussian distributed.
- Closed-form solution:
- $\theta=(X^TX)^{-1}X^Ty$
- But really this is not good. Inversion is expensive and exact solution is very sensitive to outliners. not good. we need some regularization.
- Lasso:
- Or
Tree Regression
- Each terminal node is a real value
Kernel Regression
- Locally weighted average using kernel as the weighting function