Lecture Slides

Complete Lecture Slides (PDF - 2.9MB)

LEC # TOPICS
Finite Horizon Problems
Lecture 1 (PDF)
  • Introduction to Dynamic Programming
  • Examples of Dynamic Programming
  • Significance of Feedback
Lecture 2 (PDF)
  • The Basic Problem
  • Principle of Optimality
  • The General Dynamic Programming Algorithm
  • State Augmentation
Lecture 3 (PDF)
  • Deterministic Finite-State Problem
  • Backward Shortest Path Algorithm
  • Forward Shortest Path Algorithm
  • Alternative Shortest Path Algorithms
Lecture 4 (PDF)
  • Examples of Stochastic Dynamic Programming Problems
  • Linear-Quadratic Problems
  • Inventory Control
Lecture 5 (PDF)
  • Stopping Problems
  • Scheduling Problems
  • Minimax Control
Lecture 6 (PDF)
  • Problems with Imperfect State Info
  • Reduction to the Perfect State Info Cas
  • Linear Quadratic Problems
  • Separation of Estimation and Control
Lecture 7 (PDF)
  • Imperfect State Information
  • Sufficient Statistics
  • Conditional State Distribution as a Sufficient Statistic
  • Finite-State Analysis
Lecture 8 (PDF)
  • Suboptimal Control
  • Cost Approximation Methods: Classification
  • Certainty Equivalent Control
  • Limited Lookahead Policies
  • Performance Bounds
  • Problem Approximation Approach
  • Parametric Cost-To-Go Approximation
Lecture 9 (PDF)
  • Rollout Algorithms
  • Cost Improvement Property
  • Discrete Deterministic Problems
  • Approximations to Rollout Algorithms
  • Model Predictive Control (MPS)
  • Discretization of Continuous Time
  • Discretization of Continuous Space
  • Other Suboptimal Approaches
Simple Infinite Horizon Problems
Lecture 10 (PDF)
  • Infinite Horizon Problems
  • Stochastic Shortest Path (SSP) Problems
  • Bellman's Equation
  • Dynamic Programming – Value Iteration
  • Discounted Problems as a Special Case of SSP
Lecture 11 (PDF)
  • Review of Stochastic Shortest Path Problems
  • Computation Methods for SSP
  • Computational Methods for Discounted Problems
Lecture 12 (PDF)
  • Average Cost Per Stage Problems
  • Connection With Stochastic Shortest Path Problems
  • Bellman's Equation
  • Value Iteration, Policy Iteration
Lecture 13 (PDF)
  • Control of Continuous-Time Markov Chains: Semi-Markov Problems
  • Problem Formulation: Equivalence to Discrete-Time Problems
  • Discounted Problems
  • Average Cost Problems
Advanced Infinite Horizon Problems
Lecture 14 (PDF)
  • Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods
Lecture 15 (PDF)
  • Review of Basic Theory of Discounted Problems
  • Monotonicity of Contraction Properties
  • Contraction Mappings in Dynamic Programming
  • Discounted Problems: Countable State Space with Unbounded Costs
  • Generalized Discounted Dynamic Programming
  • An Introduction to Abstract Dynamic Programming
Lecture 16 (PDF)
  • Review of Computational Theory of Discounted Problems
  • Value Iteration (VI)
  • Policy Iteration (PI)
  • Optimistic PI
  • Computational Methods for Generalized Discounted Dynamic Programming
  • Asynchronous Algorithms
Lecture 17 (PDF)
  • Undiscounted Problems
  • Stochastic Shortest Path Problems
  • Proper and Improper Policies
  • Analysis and Computational Methods for SSP
  • Pathologies of SSP
  • SSP Under Weak Conditions
Lecture 18 (PDF)
  • Undiscounted Total Cost Problems
  • Positive and Negative Cost Problems
  • Deterministic Optimal Cost Problems
  • Adaptive (Linear Quadratic) Dynamic Programming
  • Affine Monotomic and Risk Sensitive Problems
Lecture 19 (PDF)
  • Introduction to approximate Dynamic Programming
  • Approximation in Policy Space
  • Approximation in Value Space, Rollout / Simulation-based Single Policy Iteration
  • Approximation in Value Space Using Problem Approximation
Lecture 20 (PDF)
  • Discounted Problems
  • Approximate (fitted) VI
  • Approximate PI
  • The Projected Equation
  • Contraction Properties: Error Bounds
  • Matrix Form of the Projected Equation
  • Simulation-based Implementation
  • LSTD, LSPE, and TD Methods
Lecture 21 (PDF)
  • Review of Approximate Policy Iteration
  • Projected Equation Methods for Policy Evaluation
  • Simulation-Based Implementation Issues, Multistep Projected Equation Methods
  • Bias-Variance Tradeoff
  • Exploration-Enhanced Implementations, Oscillations
Lecture 22 (PDF)
  • Aggregation as an Approximation Methodology
  • Aggregate Problem
  • Simulation-based Aggregation
  • Q-Learning
Lecture 23 (PDF)
  • Additional Topics in Advanced Dynamic Programming
  • Stochastic Shortest Path Problems
  • Average Cost Problems
  • Generalizations
  • Basis Function Adaptation
  • Gradient-based Approximation in Policy Space
  • An Overview