DDA4300/CME307/MS&E311 Optimization in Data Science and Machine Learning Winter 2021-2023

| Announcements (Check it!) | General Information | Course Outline | Handouts | Assignments |

• Class Times and Locations
  • Class Room/time: CUHKSZ ChengDao 102; 10:30-11:50 on Monday and Wednesday (Beijing Time) every week, February 8 to April 19, 2023
    
  • Tutorial/Problem Session (HW Session, Midterm Exam, Project Session/Presentation, Make-Up Lectures): CUHKSZ ChengDao 102; 18:00-19:20 every Friday
    

• Prerequisites : Stanford University Math 113, 115 (or equivalent).

• Text: David Luenberger and Yinyu Ye, Linear and Nonlinear Programming, 5th Edition. See Blackboard for supplemental materials.

Course Staff

• Professor: Yinyu Ye
  • Office Hour Place: Room 511b, Dao Yuan Building
  • Office hours: by appointment or Wednesdays 2:00pm - 3:00pm (Check Blackborad for any possible change)
  • Email: yinyu-ye@stanford.edu or yeyinyu@cuhk.edu.cn
  • Phone:
• Course Assistant:
  • zongsenyang@link.cuhk.edu.cn, Office Hour: Tuesday 3pm-4pm, Location: 4th Floor, Zhixin Building
  • chenhaowang@link.cuhk.edu.cn, Office Hour: Friday 3pm-4pm, DY324
  • Yushun Zhang, yushunzhang@link.cuhk.edu.cn, Office hour: Thursday 3pm-4pm, DY324
• Staff Support: ?
  • Office: ?
  • Email: ?
  • Phone: ?

Grading and Exams

• Homework Exercises: 4 homework assignments will be graded - 60%.
• Computational Project: A team project (up to 5 people) due April 26, -- 40%
• Blackboard Participation: You will be awarded with up to 3% extra credit if you answer other students' questions on the course Blackboard. Students who take the course for pass/nopass just need to complete the project.

About Optimization

CME307/MS&E311 emphasizes high level pictues of (convex or nonconvex) Optimization/game, including classical duality and fix-point theories, KKT conditions, efficient algorithms and recent progresses in Linear and Nonlinear Optimization/Game---one of the central mathematical decision models in Data Science, Machine Learning, Reinforcement Learning, Business Analytics, and Operations Management . CME307MS&E311 can be viewed as a continuation course of MS&E211. Although there are some minor overlaps with MS&E 211, CME307/MS&E311 covers more advanced application models, in-depth optimization theories, and state of the art algorithms which were not able to be covered by MS&E211. CME307/MS&E311 also extensively covers Nonlinear and Nonconvex Optimization/Game problems, complementing to MS&E310 (Linear Optimization) and other "Convex Optimization" courses.

The field of optimization is concerned with the study of maximization and minimization of mathematical functions. Very often the arguments of (i.e., variables in) these functions are subject to side conditions or constraints. By virtue of its great utility in such diverse areas as applied science, engineering, economics, finance, medicine, data analysis, machine learning and statistics, optimization holds an important place in both the practical world and the scientific world. Indeed, as far back as the Eighteenth Century, the famous Swiss mathematician and physicist Leonhard Euler (1707-1783) proclaimed that ... nothing at all takes place in the Universe in which some rule of maximum or minimum does not appear. The subject is so pervasive that we even find some optimization terms in our everyday language.

Optimization often goes by the name Mathematical Programming (MP). The latter name tends to be used in conjunction with finite-dimensional optimization problems, which in fact are what we shall be studying here. The word "Programming" should not be confused with computer programming which in fact it antedates. As originally used, the term refers to the timing and magnitude of actions to be carried out so as to achieve a goal in the best possible way. Highlights of topics of this year are: Decision Models/Applications -- Online Pricing and Resource Allocation, Markov Decision Process and Reinforcement Learning, Data Classification via Wasserstein Barycenter, Sensor-Network Localization, Neural-Learning Verification, Distributionally Robust Decisioning and Learning, Economic/Game Equilibrium, Financial Techniques and Risk Management, Sparse and Low Rank Regression, Conic Linear Optimization;

Theories -- Alternative-Theorems, Optimality Conditions, Conic and Lagrangian Duality, Complexity Analyses;

Algorithms/Methods -- Steepest Descent, Multiplicative-Descent, Accelerated Descent, Block-Coordinate Descent, Stochastic Gradient Descent, Newton Descent, Path-Following Descent, Descent-First and Feasible-Second, Interior-Point Methods, Lagrangian Relaxations, ADMM methods, Optimization with random samplings and column generation, and other fast/heuristic Algorithms for NONCONVEX optimization with certain provable guarantee..., which you would learn during the process of the course.

Course Requirements

What background is needed for CME307/MS&E311? This is a graduate-level Core course in the ICME and MS&E Department.  No prior optimization background is required (although it should do no harm to have some).  In this sense, it is an introductory course, but it is not intended to be an elementary course. Students who have taken courses such as MS&E 211 (or MS&E 310) will see some repetition of material. This is unavoidable, but CME 307 and MS&E 311 are intended to be more theoretical than MS&E 211.

Students in this course will be expected to possess a firm background in the following mathematical subjects: multivariate differential calculus; fundamental concepts of analysis; linear algebra and matrix theory. Familiarity with computers and computer programming are also be useful, since various algorithm implementation projects can be substituted for the final exam. Above all, it is essential to have a tolerance for mathematical discourse plus an ability to follow - and devise one's own - mathematical proofs. .