DDA4300/CME307/MS&E311 Optimization in Data Science and Machine Learning Winter 2021-2023

| Announcements (Check it!) | General Information | Course Outline | Handouts | Assignments |

---

Course Outline

• Part I: Math Reviews and Math. Prog. Models
  • Math. Prog. Models and Applications:
    Reinforced Learning and Markov Decision/Game Process, SVM/Logistic Regression, Wasserstein Barycenter, Neural-Network Regression, Compressed-Sensing, Facility Locations, Online Resource Allocation, Fisher's Pricing Model, Arrow-Debreu's Equilibrium, Combinatorial Auctions and Prediction Markets, Sensor Network Localization, etc.
    (Lecture Slide Note 1, L&Y Chapts 1, 2.1-2, 6.1-2, 7.2, 11.3, 11.6)
  • Mathematical Preliminaries
    (Lecture Slide Note 2, L&Y Appdendices A, B and C, Chapt 1)
• Part II: Math. Prog. Theories
  • Elements of Convex Analysis and Conic Linear Programming (CLP)
    (Lecture Slide Note 3, L&Y Appendex B, Chapters 3.1-2 and 6.1-4)
  • CLP Duality Theorems and Applications
    (Lecture Slide Note 4, L&Y Chapters 3.1-2 and 6.3-4)
  • Support-Size and Rank of CLP Solutions and Applications
    (Lecture Slide Notes 5, L&Y Chapters 3.1-2 and 6.4-5)
  • Optimality Conditions for Linearly-Constrained Nonlinear Optimization
    (Lecture Slide Notes 6, L&Y Chapters 7.1-3 and 11.1-8)
  • Optimality Conditions for Generally-Constrained Optimization
    (Lecture Slide Notes 7, L&Y Chapters 11.1-8)
  • Lagrangian Duality Theory and Applications
    (Lecture Slide Note 8, L&Y Chapters 11.7-8, 14.1-2)
• Part III: Math. Prog. Algorithms
  • Zero-Order and First-Order Optimization Algorithms
    (Lecture Slide Note 9, L&Y Chapters 7.6-7, 8.1-5)
  • More First-Order Optimization Algorithm
    (Lecture Slide Note 10, L&Y Chapters 4.2, 8.4-5, 9.1-7, 12.3-6)
  • Dual/Lagrangian Methods for Constrained Optimization
    (Lecture Slide Note 11, L&Y Chapter 14.1-6)
  • Second Order Optimization Algorithms I
    (Lecture Slide Note 12, L&Y Chapter 8.6-7, 9.1-5, 10.1-4)
  • Second Order Optimization Algorithms II: Interior-Point Algorithms
    (Lecture Slide Note 13, L&Y Chapters 5.4-7 and 6.6)
  • Randomized Block Coordinate and Stochastic (Sub-)Gradient Methods
    (Lecture Slide Note 14, L&Y Chapter 8.8 and 3.5)
  • Other Emerging Methods (time permitting)

---