Build a function that maps $X$ to a real value $Y$.
- Linear regression
- Nonlinear/Nonparametric regression (Tree regression, Kernel, etc.)
- For linear regression minimizing least-square loss is equivalent to maximizing likelihood, with the assumption that the error term is zero-mean gaussian distributed.
- Closed-form solution:
- But really this is not good. Inversion is expensive and exact solution is very sensitive to outliners. not good. we need some regularization.
- Each terminal node is a real value
- Locally weighted average using kernel as the weighting function