> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Markov Decision Processes
> MDPs, Bellman equations, and dynamic programming for sequential decision making.
We started looking at different agent behavior architectures starting from the planning agents where the *model* of the environment is known and with *no interaction* with it, the agent improves its policy, using this model as well as problem solving and logical reasoning skills.
We now look at agents that can plan by:
1. *Interacting* with the environment by receiving *reward* signals from it during each interaction.
2. Knowing the model (dynamics) of the environment, they have an internal *objective* function that they try to optimize based on the *experience* they accumulate.
The problem as will see, will be described via a set of four equations called Bellman expectation and Bellman optimality equations that connect the values (utility) with each state or action with the policy (strategy) of the agent. These equations can be solved by Dynamic Programming algorithms to produce the optimal policy that the agent must adopt.
Computationally we will go through approaches that solve the MDP as efficiently as possible - namely, the value and policy iteration algorithms.
## Topics
States, actions, transitions, rewards, and value functions.
Computing value functions using Bellman expectation equations.
Optimal value functions and the Bellman optimality equations.
Finding optimal policies through evaluation and improvement.
## Resources
Apart from the notes here that are largely based on [David Silver's (Deep Mind) course material](https://www.davidsilver.uk/teaching/) and [video lectures](https://www.youtube.com/watch?v=2pWv7GOvuf0\&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ), you can consult these additional resources:
* [Sutton & Barto's Reinforcement Learning Book](http://incompleteideas.net/book/RLbook2020.pdf) - David Silver's slides and video lectures are based on this book. The code in Python is [here](https://github.com/ShangtongZhang/reinforcement-learning-an-introduction).
* [Deep Reinforcement Learning in Python](https://www.amazon.com/Deep-Reinforcement-Learning-Python-Hands-dp-0135172381/dp/0135172381/) - written by Google researchers.
Many of the algorithms presented here like policy and value iteration have been developed in older repos such as [rlcode](https://github.com/rlcode/reinforcement-learning) and [dennybritz](https://github.com/dennybritz/reinforcement-learning). This site is being migrated to be compatible with Farama and their [Gymnasium](https://gymnasium.farama.org/) tooling.
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/book/mdps/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).