IE598: Reinforcement Learning & Learning-based Control

Course Information

• When: Tuesday/Thursday 2:00pm--3:20pm

• Where: 1024 Lincoln Hall

• Who: Dr. Yingying Li

• Office Hours: Thursdays, 4pm-5pm (Tentative) at CSL 347 (1308 W Main St, Urbana)

Tentative Schedule

• Part 1: When and How to Model Problems as Markov Decision Processes (MDP) and Optimal Control (OC)

In this part, we will introduce the basic formulation of MDP and OC and their variations. We will discuss how to model and reformulate problems in these forms.

• Part 2: How to Solve MDP and OC When Models are Known and Small.

In this part, we will learn an important and commonly used tool to handle both MDP and OC: Dynamic Programming (DP). 

We will start with the simplest setting: finite-horizon formulation, then cover the infinite-horizon formulation, and briefly discuss the partial observation case.

We will also introduce popular algorithms inspired by DP, such as value iteration, policy iteration, rollout algorithms, and optimal control methods such as LQR and model predictive control

• Part 3: When Models are Unknown: Reinforcement Learning and Learning-based Control

We will discuss major directions of algorithm designs for RL and LBC: including Q learning, policy gradient, actor-critic, certainty equivalence, system identification, model-based RL, etc. We will also introduce an important tool for RL in both algorithmic and theoretical senses: multi-armed bandit.

• Part 4: When Models are Too Large: Approximations and Neural Networks

This part will introduce state/action/function approximations for RL and LBC to tackle the curse-of-dimensionality challenge, such as DQN, Fitted Q, quantization, Interpolation, LSTD, etc.

• Part 5: State-of-the-art and Research Topics: Guest Lectures

This part will introduce some advanced research topics on RL and LBC. Some sessions will have invited speakers discussing state-of-the-art research progress. Some tentative topics include: safe RL, robust RL, online control, multi-agent RL, imitation and representation learning, diffusion models, RLHF, MCTS, applications to robotics, etc.

Textbooks

• Dimitri Bertsekas. Dynamic Programming and Optimal Control: Volume I and II

Other Useful Resources

• Dimitri Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming.

• Andrew Barto and Richard S. Sutton. Reinforcement Learning: An Introduction

• MIT 6.7950 Reinforcement Learning: Foundations and Methods

• Stanford CS234: Reinforcement Learning

Grading

• Assignments: 30%. (There will be three assignments, each contributes 10%.)

• Midterm Exam: 30%.

• Final Project: 40%.

Assignments

• Will be posted on Canvas

• Please submit your answers to Gradescope.

• One late submission is allowed. 

Midterm

• Open book, but no computers or cell-phones or any electronic devices.

Final Project

• You can find all the details in the syllabus.pdf on Canvas.

• Basically, 10% for project proposal and proposal discussion, 15% for in-class presentation, and 15% for a final project report.

• Either literature review of existing papers or research projects such as applying RL or LBC to your research.

• In-class presentations will take place before the Reading Day.