Group Relative Policy Optimization (GRPO) is a recent reinforcement learning method introduced in the DeepSeekMath paper. Developed as an upgrade to the Proximal Policy Optimization (PPO) framework, GRPO aims to improve mathematical reasoning skills while lessening memory use. This technique is especially suitable for functions that require sophisticated mathematical reasoning.
The implementation of GRPO involves several key steps. The current policy generates multiple answers for each input query. These answers are subsequently scored using a reward model. The mean of these rewards serves as a benchmark for calculating the advantages. Finally, the policy is updated to maximize the GRPO objective, which includes the advantages and a KL divergence term.
GRPO distinctly departs from traditional PPO in its dismissal of the need for a value function model, consequently reducing memory and computational complexity. Rather, GRPO uses group scores to estimate the baseline, thus simplifying the training process and resource requirements.
Several innovative features and benefits are introduced by GRPO. The training process is made more efficient and scalable by foregoing the value function model. Unlike other methods, which append the KL divergence term to the reward, GRPO incorporates this term directly into the loss function. This measure aids in stabilizing the training process and has led to significant performance improvements in mathematical benchmarks. The capability of GRPO to improve performance without relying on an independent value function underscores its potential for broader applications in reinforcement learning scenarios.
Although GRPO shares similarities with the Rejection Sampling Fine-Tuning (RFT) method, unique factors distinguish them. One such factor is GRPO’s iterative approach to training reward models. This iterative process fine-tunes the model more effectively by consistently updating it based on the most recent policy outputs.
GRPO positions itself as a fantastic tool for enhancing the capabilities of open language models, due to its efficient use of resources and innovative techniques for calculating advantages and integrating KL divergence. Its application in DeepSeekMath has demonstrated significant advancements in reinforcement learning methods tailored for mathematical reasoning, thereby showcasing the potential to push the boundaries of what language models can achieve in complex, structured tasks like mathematics. This integration has resulted in substantial improvements in in- and out-of-domain tasks during the reinforcement learning phase in DeepSeekMath.
