> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # Grouped Relative Policy Optimization > GRPO removes the value critic of PPO by sampling G outputs per prompt and computing advantages from group-normalized rewards, reducing memory and compute while retaining stable policy updates. GRPO training loop: query q enters the Policy Model, which samples G outputs o_1 to o_G; a frozen Reward Model scores each output into r_1 to r_G; Group Computation normalizes the scores into advantages A_1 to A_G; a frozen Reference Model provides a per-token KL penalty fed back to the Policy Model

GRPO training loop: query q enters the Policy Model, which samples G outputs o_1 to o_G; a frozen Reward Model scores each output into r_1 to r_G; Group Computation normalizes the scores into advantages A_1 to A_G; a frozen Reference Model provides a per-token KL penalty fed back to the Policy Model

*Editable Mermaid source: [`images/grpo-loop.mermaid.md`](images/grpo-loop.mermaid.md)*