Proximal Policy Optimization (PPO) - aegean.ai

PPO actor-critic loop: query q enters the Policy Model, which generates output o evaluated by a frozen Reference Model (KL penalty), a frozen Reward Model, and a trainable Value Model; combined reward r and value estimate v feed into GAE to produce advantage A; dashed arcs show actor and critic gradient paths back to their respective models

Editable Mermaid source: images/ppo-loop.mermaid.md

Analytical Derivations

Further reading

Huang et al. (2022) — The 37 Implementation Details of Proximal Policy Optimization — a detailed walkthrough of every engineering choice needed to reproduce PPO results in practice

Found an error on this page? Report an issue.