DeepSeekMath's training pipeline (Shao et al.) uses a reinforcement-learning algorithm called Group Relative Policy Optimization (GRPO) instead of standard PPO to reduce the memory cost of the RL stage. What specific change does GRPO make to how the advantage used in the policy update is computed, compared to standard PPO?
- GRPO still trains a separate learned value/critic network exactly as PPO does, and its only change from PPO is removing the separate reward model used to score outputs
- GRPO uses a single fixed baseline value, chosen once by hand before training begins and never recomputed, in place of PPO's learned value function for every prompt across the entire training run
- GRPO requires a human-labeled preference judgment for every one of the sampled outputs before it can compute any output's advantage, replacing PPO's learned value function with direct human scoring
- GRPO removes PPO's separate learned value/critic network entirely; for a given prompt it samples a group of outputs from the current policy, uses that group's own average reward as a baseline, and computes each output's advantage as how far its reward deviates from that group's baseline, normalized by the group's reward variability
Why D? And why not the others?
Correct answer: D. GRPO removes PPO's separate learned value/critic network entirely; for a given prompt it samples a group of outputs from the current policy, uses that group's own average reward as a baseline, and computes each output's advantage as how far its reward deviates from that group's baseline, normalized by the group's reward variability
GRPO's defining change from standard PPO is eliminating the separately learned value or critic network that PPO normally trains to estimate a baseline for computing advantages; instead, for each prompt, GRPO samples a whole group of candidate outputs from the current policy, treats that group's own average reward as the baseline, and computes each individual output's advantage as the deviation of its reward from that group average, normalized using the group's own reward standard deviation, which removes the memory and compute cost of maintaining a separate critic model. The option claiming GRPO still trains a critic network and only removes the reward model is backwards; GRPO keeps a reward signal but removes the critic. The option describing a single fixed, hand-chosen baseline is wrong because GRPO's baseline is computed dynamically per prompt from the sampled group's own rewards, not set once in advance. The option requiring a human preference judgment for every sampled output is wrong because GRPO's advantage computation works with whatever reward signal the setup provides (which in DeepSeekMath is often an automatic correctness check), not a per-output human label.
Source: Shao et al., 'DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models' (arXiv:2402.03300, 2024)