Jan 31:
Function optimization using
first and
second
order gradient methods
Goal: Review gradient descent approaches.
A nice chapter on function optimization techniques:
Numerical Recipes in C, chapter 10
(2nd or 3rd edition, 2nd edition is electronically available for free
under Obsolete Versions):
Minimization or Maximization of Functions,
This material from any other numerical methods book is also fine.
Resources:
Matlab fminunc,
Numerical Recipes,
GSL,
AMPL,
NEOS,
software list 1,
Useful
software guide,
gradient method,
line search,
conjugate gradient,
conjugate gradient v2,
quasi-Newton/variable metric methods,
Newton's method,
Levenberg Marquardt,
Reduced dimensionality second order methods.
Other lectures:
Stanford MSandE 311;
U. Stuttgart: Toussaint
Papers:
Optimization Methods for Large-Scale Machine Learning;
Identifying and attacking the saddle point problem in
high-dimensional non-convex optimization