Rémi Flamary

Optimization for Machine Learning M1

Courses

• Introduction to numerical optimization [PDF]
  • Optimization problem formulation and principles
  • Properties of optimization problems
  • Machine learning as an optimization problem
• Constrained Optimization and Standard Optimization problems [PDF]
  • Constraints, Lagrangian and KKT
  • Linear Program (LP)
  • Quadratic Program (QP)
  • Other Classical problems (MIP,QCQP,SOCP,SDP)
• Smooth Optimization [PDF]
  • Gradient descent
  • Newton, quasi-Newton and Limited memory
• Non-smooth Optimization [PDF]
  • Proximal operator and proximal methods
  • Conditional gradient
• Conclusion
  • Other approaches
  • Optimization problem decision tree
  • References and toolboxes

Practical sessions

• Standard problems [PDF][manhattan.npz]
• Smooth optimization [PDF][digits.npz]
• Non-smooth optimization

Practical sessions will require a working Python environnement with te libraries Numpy/Scipy and Matplotlib installed. You can get such an environnement for Windows/Linux/MacOSX on Anaconda.

Here is a list of nice references for the Pythoon :

• Tutorial Numpy
• Doc Numpy
• Doc Scipy
• Doc ScMatplotlib/pylab