GRPO vs PPO for LLM Fine-Tuning: A Worked Advantage Example
GRPO (Group Relative Policy Optimization) and PPO (Proximal Policy Optimization) are the two reinforcement-learning algorithms behind almost every RLHF and RLVR fine-tuning pipeline in production today — PPO in the original InstructGPT/ChatGPT recipe, GRPO in DeepSeekMath and DeepSeek-R1. Most explanations describe the difference in words: "GRPO drops the critic." Fewer show what that actually changes in the numbers. Below, the same four sampled completions to the same prompt are scored both ways, so you can see exactly where the two advantage estimates come from and why one of them needs an entire second neural network and the other doesn't.
The one-sentence version
Both algorithms update the policy with the same clipped surrogate objective from Schulman et al.'s original PPO paper: for each sampled token, take the probability ratio between the new and old policy, multiply it by an advantage estimate, and clip that ratio to [1-&epsilon, 1+&epsilon] (commonly ε=0.2) so no single update moves the policy too far in one step. The two algorithms disagree on exactly one thing: where the advantage estimate comes from. PPO learns it from a trained value network. GRPO computes it from the other completions in the same sampled batch.
Side by side
| PPO (InstructGPT-style RLHF) | GRPO (DeepSeekMath / DeepSeek-R1) | |
|---|---|---|
| Baseline for the advantage | A trained value/critic network's prediction, V(s) | The mean reward of G completions sampled for the same prompt |
| Extra network required | Yes — a critic, typically initialized from the same base model | No critic; only the policy, a frozen reference copy, and a reward model |
| Rollouts needed per prompt | Usually 1 | G, commonly 8–64 |
| Works best with | Learned, nuanced reward-model scores (human preference) | Verifiable, rule-checkable rewards (math correctness, unit tests passing) |
| KL penalty against the reference policy | Subtracted from the per-token reward before computing the advantage | Added as a separate term directly in the loss function |
Worked example: one prompt, four completions, two advantages
Say a policy is sampling G=4 completions to the same math prompt, and each gets scored by a reward model (or a rule-based checker) with a score between 0 and 1:
| Completion | Reward |
|---|---|
| A | 0.9 |
| B | 0.7 |
| C | 0.5 |
| D | 0.1 |
GRPO: normalize against the group's own mean and spread
GRPO's advantage for completion i is A_i = (r_i - mean(r)) / std(r), using only the four rewards actually sampled this step. Mean = (0.9+0.7+0.5+0.1)/4 = 0.55. Squared deviations from the mean are 0.1225, 0.0225, 0.0025, 0.2025; their average is 0.0875, so the standard deviation is √0.0875 ≈ 0.296. Dividing each deviation by that std gives:
- A: (0.9-0.55)/0.296 ≈ 1.18
- B: (0.7-0.55)/0.296 ≈ 0.51
- C: (0.5-0.55)/0.296 ≈ -0.17
- D: (0.1-0.55)/0.296 ≈ -1.52
Every token generated as part of completion A gets pushed up by advantage 1.18; every token in D gets pushed down by 1.52. No network computed those numbers — they came entirely from comparing the batch to itself.
PPO: subtract a learned baseline
PPO instead trains a separate critic network to predict the expected reward for a given prompt, V(s), based on everything it has seen across many prior batches — not just these four samples. Suppose the critic, at this point in training, predicts V=0.6 for this prompt (notice it doesn't have to match this batch's mean of 0.55; it's a running estimate from a different model). The advantage is simply A_i = r_i - V(s):
- A: 0.9-0.6 = 0.3
- B: 0.7-0.6 = 0.1
- C: 0.5-0.6 = -0.1
- D: 0.1-0.6 = -0.5
(This is a simplified, response-level version of the calculation for clarity; production PPO/RLHF implementations compute a separate advantage per token via Generalized Advantage Estimation, since the KL penalty is applied at every token while the reward-model score usually lands only on the final token.)
Both algorithms agree on the direction: push up on A and B, push down on C and D. They disagree on where the zero point sits and on the source of that zero point — GRPO's is recomputed from scratch every batch from the samples themselves; PPO's is a slowly-learned prediction that can lag or overfit if the critic hasn't converged yet.
Where the memory actually goes
A standard PPO/RLHF training run keeps four models resident at once: the policy being trained, a frozen reference copy of the pre-RL policy (for the KL penalty), a reward model, and the critic. The critic is typically initialized from the same base model as the policy, so it can cost nearly as much GPU memory — parameters, gradients, and Adam's two optimizer-state buffers — as the policy itself. GRPO removes that fourth model entirely: its baseline comes from arithmetic on already-necessary reward-model scores across the G rollouts, not from a trained network. DeepSeekMath's own paper frames this as a direct resource-reduction motivation for the design, and practitioner write-ups commonly cite total RLHF training memory dropping by roughly a third when the critic is removed — the exact figure depends on model size and how the reference/reward models are hosted, but the direction and rough scale are well established.
The KL penalty: same idea, wired in differently
Both algorithms keep the policy from drifting too far from a frozen reference model, but they apply that constraint in different places. PPO/InstructGPT-style RLHF subtracts a per-token KL term directly from the reward before the advantage is computed, so the KL penalty and the task reward get blended into one number. GRPO instead adds an unbiased, always-non-negative KL estimate (John Schulman's "k3" estimator, KL ≈ r - 1 - log(r) where r is the reference-to-policy probability ratio) as a separate term in the loss, scaled by a coefficient β — DeepSeekMath's paper uses β=0.04. Keeping it separate means the KL constraint no longer distorts the advantage estimate itself; it only adds its own gradient.
When each one is the right call
- Reach for GRPO when rewards are cheap to check mechanically and verifiable — a math answer is either right or wrong, code either passes its test suite or doesn't — and you can afford to sample a group of
Gcompletions per prompt. This is exactly the setting DeepSeekMath and DeepSeek-R1 were built for. - Reach for PPO when rewards come from a learned, noisy reward model trained on human preferences, sampling many completions per prompt is too expensive (long agentic trajectories, for example), or you already have a reliable critic pipeline. This remains the standard for classic instruction-following RLHF.
One sharp edge worth knowing either way: if all G completions in a GRPO group happen to get an identical reward, the standard deviation is zero and the advantage is undefined for every sample in that group — that prompt contributes no learning signal at all. DAPO (Yu et al., 2025), a widely-used follow-up, addresses this directly with "dynamic sampling": it discards zero-variance groups and keeps resampling prompts until it fills a batch with groups that actually have a spread of outcomes, alongside a separate fix ("Clip-Higher") for a length/entropy-collapse bias the original GRPO clipping range can introduce.
For more on how these fine-tuning techniques fit alongside LoRA, quantization, and instruction tuning, work through the AI fine-tuning practice questions.
Source: Schulman et al., "Proximal Policy Optimization Algorithms" (arXiv:1707.06347); Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (arXiv:2402.03300); Ouyang et al., "Training language models to follow instructions with human feedback" (arXiv:2203.02155); J. Schulman, "Approximating KL Divergence" (joschu.net/blog/kl-approx.html, 2020); Yu et al., "DAPO: An Open-Source LLM Reinforcement Learning System at Scale" (arXiv:2503.14476). This is a technical explainer for study purposes, not implementation guidance for a specific training framework.
Drill Fine-tuning & Model Customisation practice questions →