I am going to attempt to keep this page upto date... please make changes as you see fit:

Week 1

Overview of MDPs

*Definitions (state space, Markov property, closed sets, MDP's)

*Agent Action Based Observations

*Value Function for MDPs

*Observable, partially Observable, and Non-Observable

*Histories, Horizons, Rewards and Plan's

Week 2

*Bellman's Principal of Optimality

*Value Iteration

**Implementation of Value Iteration

*Howard's Optimal Stationary Policy

*Successive Approximation

*Policy Iteration

Week 3

*Heuristic Searches

**See Papers for papers regarding these topics..

*Goal Based MDPs

*RTDP

A

Week 4

AO

LAO

*Labeled RTDP

*HDP

Week 5

*Factored MDP's

*Definitions and Comparisons to Flat MDPs

** Robot Example with different arc's etc.

*BDDs

**Definition and Construction

Week 6

*ADDs

**Addition, Subtraction, Multiplication, MAX, MIN

**Reduction

**Using to model Flat MDPs

*SPUDD

**Small Example and PseudoCode

*Planit Suite Intro (Big Picture)

**DIET, HELL, LIMBO, BNB, PET, PlanIt and interactions

Week 7

*More PlanIt

**Using the preference elicitor, create a plan. Demo of coffee/robot domain.

**DIET and PlanIt code analysis. What is significant in each as far as programming.

**[https://soapbox.iraproject.org/Blog/tabid/71/EntryID/11/Default.aspx Using Eclipse and linking with SVN]

*More PlanIt

**Looking at the algorithms

**Discussion and dissection of the code within the planner

Week 8

*SPRING BREAK

Week 9

*Linear Programming

**MAX and MIN functions

**Constraints

**Conical Forms

*More Linear Programming

**Use for Flat MDPs

**Back-Projection

**Use with DBNs

Week 10

SLAO

**Presentation by Renee

**Pseudo-Code, Examples and Discussion

*sRTDP

**Presentation by Nick

**Pseudo-Code, Examples and Discussion

Week 11

*Crash on Paper

*More Linear Programming

**Factored MDP uses

**MIN and MAX etc.

Week 12

*Reinforcement Learning

**Supervised / Unsupervised Learning

**Model Free Learning

**Q-Learning

*More Reinforcement Learning

**Model-Based Learning

**Differences with Q-Learning

Week 13

*More Reinforcement Learning

**More Model-Free refinements and algorithms

*Semi-MDPs

**Temporal Abstraction

**Episodes

**Semi-MDPs and Options

Week 14

*Discussion of End of Semester Report