Theoretical Foundations of Reinforcement Learning. 3 credits, 3 contact hours

Catalog description

Prerequisites: DS 675 or DS 699 or instructor approval. This course offers a graduate-level introduction to the theory behind reinforcement learning (RL). Students will study the mathematical framework of Markov Decision Processes (MDPs) and use it to analyze algorithms for both planning and learning in sequential decision-making problems. The course is organized into two parts. The first part covers planning, where the environment model is assumed known. Topics include dynamic programming (value iteration, policy iteration), online planning, function approximation in approximate policy iteration, and the computational limits of planning under different structural assumptions. The second part shifts to the learning setting, where the agent must interact with an unknown environment. Topics include sample complexity and regret, the optimism principle, and exploration algorithms for tabular and linear MDPs, and recent frameworks for exploration with function approximation such as estimation-to-decision and maximize-to-explore. Students will complete proof-based assignments and read and present research papers in the topic.