> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # Proximal Policy Optimization (PPO) > PPO is an actor-critic policy gradient algorithm that constrains each update to stay close to the previous policy, stabilizing training by preventing destructively large gradient steps. PPO actor-critic loop: query q enters the Policy Model, which generates output o evaluated by a frozen Reference Model (KL penalty), a frozen Reward Model, and a trainable Value Model; combined reward r and value estimate v feed into GAE to produce advantage A; dashed arcs show actor and critic gradient paths back to their respective models

PPO actor-critic loop: query q enters the Policy Model, which generates output o evaluated by a frozen Reference Model (KL penalty), a frozen Reward Model, and a trainable Value Model; combined reward r and value estimate v feed into GAE to produce advantage A; dashed arcs show actor and critic gradient paths back to their respective models

*Editable Mermaid source: [`images/ppo-loop.mermaid.md`](images/ppo-loop.mermaid.md)*