# CMSC 25700/35100: ## Natural Language Processing ## Lecture 14: Post-training (cont.) <p style="text-align: center;"> <strong>Chenhao Tan</strong><br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/1">Submit Music Recommendations</a>, kudos to Mina Lee <br> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/2">Submit jokes</a> </p>

## Logistics * A4 due on Friday * Project - Week 7 blog due on Sunday - Our goal is to give feedback by the end of Tuesday

Lecture Outline

• Direct Preference Optimization (DPO)
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)

Lecture Outline

• Direct Preference Optimization (DPO)
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)

## The Reward Optimization Zoo There are many algorithms for reward optimization: **PPO** (Schulman et al., 2017), **DPO** (Rafailov et al., 2024), **IPO** (Azar et al., 2024), **KTO** (Ethayarajh et al., 2024), **GRPO** (Shao et al., 2024), **REINFORCE Leave-One-Out** (Ahmadian et al., 2024), SimPO, ORPO, and many more. We will focus on three: **DPO**, **PPO** (Proximal Policy Optimization), and **GRPO** (Group Relative Policy Optimization). We start with DPO because it is the simplest: no RL, no sampling, just a supervised loss. Then we build up to PPO and GRPO, which use RL.

## Why Is the RLHF Objective Hard to Optimize? <div style="font-size: 0.85em;"> The reward optimization objective: $$\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}\left[RM\_\phi(x, y) - \beta \cdot \text{KL}\left(p\_\theta(y|x) \| p\_{\text{ref}}(y|x)\right)\right]$$ This is hard because: 1. **Reward model**: must have a good reward model 2. **Expectation over model samples**: need to generate from $p\_\theta$, which changes every update 3. **Non-differentiable sampling**: cannot backprop through discrete token generation 4. **Requires RL**: must use policy gradient methods (REINFORCE, PPO) Can we avoid all of this? </div>

## DPO: Closed-Form Optimal Policy <div style="font-size: 0.9em;"> Start from the KL-regularized objective: $$\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}\left[RM(x, y) - \beta \log \frac{p\_\theta(y|x)}{p\_{\text{ref}}(y|x)}\right]$$ Write the KL term out and rearrange: $$= \mathbb{E}\_{y \sim p\_\theta}\left[RM(x, y) - \beta \log p\_\theta(y|x) + \beta \log p\_{\text{ref}}(y|x)\right]$$ This is maximized when $p\_\theta$ places all mass where $RM(x,y) + \beta \log p\_{\text{ref}}(y|x)$ is highest, subject to $p\_\theta$ being a valid distribution. The solution is: $$p^*(y|x) = \frac{1}{Z(x)} \cdot p\_{\text{ref}}(y|x) \cdot \exp\left(\frac{1}{\beta} RM(x, y)\right)$$ where $Z(x)$ normalizes the distribution. </div>

## DPO: Key Insight We have the optimal policy: $p^*(y|x) = \frac{1}{Z(x)} \cdot p\_{\text{ref}}(y|x) \cdot \exp\left(\frac{1}{\beta} RM(x, y)\right)$ Take the log of both sides and solve for $RM$: $$\log p^*(y|x) = \log p\_{\text{ref}}(y|x) + \frac{1}{\beta} RM(x, y) - \log Z(x)$$ $$RM(x, y) = \beta \log \frac{p^*(y|x)}{p\_{\text{ref}}(y|x)} + \beta \log Z(x)$$ The reward is fully determined by the policy and the reference model. No need for a separate reward model. <p class="citation"><a href="https://arxiv.org/abs/2305.18290">Rafailov et al. (2024)</a></p>

## DPO: From Insight to Loss We know the reward can be written as: $RM(x, y) = \beta \log \frac{p^*(y|x)}{p\_{\text{ref}}(y|x)} + \beta \log Z(x)$ But how do we turn this into a training loss? We need a way to compare responses. Since we have **pairwise preference data** $(y\_w, y\_l)$, we can use the **Bradley-Terry model** to define a loss over pairs. <p class="citation"><a href="https://arxiv.org/abs/2305.18290">Rafailov et al. (2024)</a></p>

## Bradley-Terry Model for Preferences <div style="font-size: 0.9em;"> Given a winning response $y\_w$ and a losing response $y\_l$, the Bradley-Terry model defines the probability that $y\_w$ is preferred: $$P(y\_w \succ y\_l | x) = \sigma(RM(x, y\_w) - RM(x, y\_l))$$ where $\sigma$ is the sigmoid function. The maximum likelihood loss over preference data is: $$\mathcal{L} = -\mathbb{E}\_{(x, y\_w, y\_l) \sim D}[\log \sigma(RM(x, y\_w) - RM(x, y\_l))]$$ DPO will substitute the closed-form reward expression into this loss, replacing $RM$ with the policy itself. <p class="citation"><a href="https://doi.org/10.2307/2334029">Bradley and Terry (1952)</a></p> </div>

## DPO: Deriving the Loss <div style="font-size: 0.85em;"> Start from the Bradley-Terry preference model: $$P(y\_w \succ y\_l | x) = \sigma(RM(x, y\_w) - RM(x, y\_l))$$ Now substitute $RM(x, y) = \beta \log \frac{p^*(y|x)}{p\_{\text{ref}}(y|x)} + \beta \log Z(x)$: $$P(y\_w \succ y\_l | x) = \sigma\left(\beta \log \frac{p^*(y\_w|x)}{p\_{\text{ref}}(y\_w|x)} - \beta \log \frac{p^*(y\_l|x)}{p\_{\text{ref}}(y\_l|x)}\right)$$ The $\beta \log Z(x)$ terms cancel! Replace $p^*$ with our learnable policy $p\_\theta$ and maximize the log-likelihood over a dataset of preference pairs $(x, y\_w, y\_l)$: $\mathcal{L}\_{\text{DPO}}(\theta) = -\mathbb{E}\_{(x, y\_w, y\_l) \sim D}\left[\log \sigma\left(\beta \log \frac{p\_\theta(y\_w|x)}{p\_{\text{ref}}(y\_w|x)} - \beta \log \frac{p\_\theta(y\_l|x)}{p\_{\text{ref}}(y\_l|x)}\right)\right]$ <p class="citation"><a href="https://arxiv.org/abs/2305.18290">Rafailov et al. (2024)</a></p> </div>

## DPO: Reading the Loss <div style="font-size: 0.9em;"> $\mathcal{L}\_{\text{DPO}} = -\log \sigma\Big(\underbrace{\beta \log \frac{p\_\theta(y\_w|x)}{p\_{\text{ref}}(y\_w|x)}}\_{\text{implicit reward of } y\_w} - \underbrace{\beta \log \frac{p\_\theta(y\_l|x)}{p\_{\text{ref}}(y\_l|x)}}\_{\text{implicit reward of } y\_l}\Big)$ Each term $\beta \log \frac{p\_\theta(y|x)}{p\_{\text{ref}}(y|x)}$ measures how much the policy has moved toward $y$ relative to the reference. This is the **implicit reward**. The loss says: the implicit reward of the winning response $y\_w$ should be higher than that of the losing response $y\_l$. The sigmoid + log make this a logistic regression on the reward gap. Gradient pushes $p\_\theta(y\_w|x)$ up and $p\_\theta(y\_l|x)$ down, weighted by how "surprised" the current model is by the preference. </div>

## Why DPO Is Easier The RLHF objective requires sampling from the model and using RL to estimate gradients: $$\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}\left[RM\_\phi(x, y) - \beta \cdot \text{KL}\right] \quad \leftarrow \text{need RL}$$ DPO turns this into a supervised loss on a fixed dataset: $$\min\_\theta \mathcal{L}\_{\text{DPO}}(\theta) \quad \leftarrow \text{standard gradient descent}$$ No sampling from the model. No reward model. No value function. No RL. Just compute log-probabilities on existing preference pairs and take gradient steps.

## DPO: Algorithm **Input**: preference dataset $D = \\{(x, y\_w, y\_l)\\}$, reference model $p\_{\text{ref}}$ 1. Initialize $p\_\theta$ from the SFT model (which also serves as $p\_{\text{ref}}$) 2. For each batch of preference pairs $(x, y\_w, y\_l)$: - Compute log-probs under $p\_\theta$ and $p\_{\text{ref}}$ for both $y\_w$ and $y\_l$ - Compute the DPO loss - Update $\theta$ with gradient descent 3. Return the trained policy $p\_\theta$ That is it. Standard supervised training loop. No sampling from the model during training.

## Quiz: DPO After DPO training, you find that $\log \frac{p\_\theta(y\_w|x)}{p\_{\text{ref}}(y\_w|x)} = 0.1$ and $\log \frac{p\_\theta(y\_l|x)}{p\_{\text{ref}}(y\_l|x)} = -2.0$. What happened? - A) The model failed to learn the preference - B) The KL penalty was too large - C) The model learned a strong preference by increasing the winning response probability - D) The model mostly decreased the losing response probability without increasing the winning one

## Quiz: Answer **D) The model mostly decreased the losing response probability without increasing the winning one** The implicit reward gap is $\beta(0.1 - (-2.0)) = \beta \cdot 2.1$, which is large, so the loss is small. But almost all of the gap comes from pushing $y\_l$ down ($-2.0$), not from pushing $y\_w$ up ($0.1$). This is the model collapse issue: the model learned what not to say rather than what to say.

## Limitations of DPO: Model Collapse <div style="font-size: 0.9em;"> The DPO loss can be minimized in two ways: 1. **Increase** $p\_\theta(y\_w|x)$ (make the winning response more likely) 2. **Decrease** $p\_\theta(y\_l|x)$ (make the losing response less likely) In practice, the model often takes the easier path: it decreases the probability of $y\_l$ without increasing the probability of $y\_w$. The model learns **what not to say** rather than **what to say**. Over training, this can lead to **model collapse**: the model's generations become degenerate because it has only learned to suppress bad outputs, not to produce good ones. Online methods (PPO, GRPO) avoid this because the model must actually generate good responses to receive high rewards. These issues motivated online methods like PPO and GRPO. </div>

## Limitations of DPO (cont.) **Offline / no exploration**: DPO optimizes over a fixed dataset. The model never generates new responses during training. It cannot discover better strategies beyond what the dataset contains. **Distribution mismatch**: As training progresses, $p\_\theta$ drifts from $p\_{\text{ref}}$. The preference pairs in $D$ were generated from $p\_{\text{ref}}$, so the training signal becomes stale.

Lecture Outline

• Direct Preference Optimization (DPO)
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)

## RL Basics: Setup In reinforcement learning, an **agent** takes **actions** in an environment and receives **rewards**. For language models: - **Agent**: the language model $p\_\theta$ - **Action**: generating a response $y$ given prompt $x$ - **Reward**: $R(x, y)$ (from a reward model or verifier) The goal is to maximize expected reward: $\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}[R(x, y)]$ But we cannot backpropagate through sampling (discrete tokens). How do we compute gradients?

## RL Basics: Policy Gradient Intuition <div style="font-size: 0.8em;"> Idea: **trial and error**. Generate responses, see which ones get high reward, make those more likely. 1. Sample a response $y \sim p\_\theta(y|x)$ 2. Compute its reward $R(x, y)$ 3. If $R$ is high: increase $p\_\theta(y|x)$ 4. If $R$ is low: decrease $p\_\theta(y|x)$ How much to increase/decrease? Use $\nabla\_\theta \log p\_\theta(y|x)$, the direction that makes $y$ more likely. Scale it by the reward: $R(x,y) \cdot \nabla\_\theta \log p\_\theta(y|x)$ High reward $\rightarrow$ large step toward $y$. Low reward $\rightarrow$ small step (or step away if negative). </div>

## The Gradient Problem We want to maximize: $J(\theta) = \mathbb{E}_{y \sim p_\theta(y|x)}[R(x,y)] = \sum_y p_\theta(y|x) \cdot R(x,y)$ Taking the gradient: $\nabla_\theta J = \sum\_y \nabla\_\theta p\_\theta(y|x) \cdot R(x,y)$ Two problems: 1. **Intractable sum**: $y$ ranges over all possible token sequences. We need to estimate this sum by sampling from $p\_\theta$. 2. **Biased estimator**: If we sample $y \sim p\_\theta$ and compute $\nabla\_\theta p\_\theta(y|x) \cdot R(x,y)$, the samples are weighted by $p\_\theta$, introducing an extra $p\_\theta$ factor. We need to correct for this.

## Deriving the Policy Gradient We need a way to estimate this gradient using only **samples** from $p_\theta$. Starting from: $\nabla\_\theta J = \sum\_y \nabla\_\theta p\_\theta(y|x) \cdot R(x,y)$ Use the **log-derivative trick**: $\nabla\_\theta p\_\theta = p\_\theta \cdot \nabla\_\theta \log p\_\theta$ $$= \sum\_y p\_\theta(y|x) \cdot R(x,y) \cdot \nabla\_\theta \log p\_\theta(y|x)$$ $$= \mathbb{E}\_{y \sim p\_\theta}[R(x,y) \cdot \nabla\_\theta \log p\_\theta(y|x)]$$

## The Log-Derivative Trick Why does $\nabla\_\theta p\_\theta = p\_\theta \cdot \nabla\_\theta \log p\_\theta$? By the chain rule: $\nabla\_\theta \log p\_\theta = \frac{\nabla\_\theta p\_\theta}{p\_\theta}$ Rearranging: $\nabla\_\theta p\_\theta = p\_\theta \cdot \nabla\_\theta \log p\_\theta$ This is the key step: it converts the gradient of a probability (hard to work with) into an expectation under that probability (easy to estimate by sampling). We never need to differentiate through the sampling process or the reward function.

## RL Basics: REINFORCE This gives us the **policy gradient** (REINFORCE): $$\nabla\_\theta J = \mathbb{E}\_{y \sim p\_\theta}[R(x,y) \cdot \nabla\_\theta \log p\_\theta(y|x)]$$ In practice, we estimate this with Monte Carlo samples: generate $N$ responses, compute rewards, average the gradients. No need to differentiate through the reward or through sampling. We only need $\nabla\_\theta \log p\_\theta(y|x)$, which the LM already computes. <p class="citation"><a href="https://link.springer.com/article/10.1007/BF00992696">Williams (1992)</a></p>

## RL Basics: Variance Reduction <div style="font-size: 0.8em; text-align: left;"> Raw policy gradient has high variance. Two standard fixes: **1. Baseline subtraction**: subtract a baseline $b$ from the reward $\nabla\_\theta J = \mathbb{E}\_{y \sim p\_\theta}[(R(x,y) - b) \nabla\_\theta \log p\_\theta(y|x)]$ This does not change the expected gradient but reduces variance. A common baseline is the **value function** $V(x)$: the expected reward from prompt $x$. **2. Clipping**: limit how much the policy can change in a single update (this is what PPO does). Different RL algorithms for LLMs differ mainly in how they handle the baseline and the update step. </div>

## PPO: Motivation Unlike DPO, PPO is **online**: the model generates new samples and learns from them. This is the classic RLHF approach used in InstructGPT and ChatGPT. **Setup**: We have a trained reward model $RM\_\phi$ and want to optimize the policy $p\_\theta$. <p class="citation"><a href="https://arxiv.org/abs/1707.06347">Schulman et al. (2017)</a>; <a href="https://arxiv.org/abs/2203.02155">Ouyang et al. (2022)</a></p>

## PPO: Overview <img src="images/rlhf.png" style="max-height: 500px;"> <p class="citation">Ouyang et al. (2022)</p>

## PPO: Algorithm **Input**: reward model $RM\_\phi$, reference model $p\_{\text{ref}}$, prompts $D$ Repeat: 1. **Sample**: generate responses $y \sim p\_\theta(y|x)$ for a batch of prompts 2. **Score**: compute rewards $RM\_\phi(x, y) - \beta \cdot \text{KL}(p\_\theta \| p\_{\text{ref}})$ 3. **Estimate advantages**: using the value function $V$ 4. **Update**: take gradient steps on the clipped surrogate objective 5. **Update value function**: fit $V$ to the observed returns

## PPO: What is the Advantage? <div style="font-size: 0.8em; text-align: left;"> The **advantage** $\hat{A}\_t$ measures how much better an action is compared to the average: $$\hat{A}\_t = R\_t - V(s\_t)$$ - $R\_t$: the reward (from $RM\_\phi$) received for the response - $V(s\_t)$: the **value function** (a learned baseline that predicts expected reward) Positive advantage: this action was better than expected. Increase its probability. Negative advantage: this action was worse than expected. Decrease its probability. *Note*: this is a simplification. The subscript $t$ indexes token positions. The reward $R$ is only given for the complete response (at the last token). Per-token advantages are computed by propagating this final reward backward using the value function. </div>

## Reward Model vs. Value Function <div style="font-size: 0.9em;"> They serve different roles in PPO: **Reward Model** $RM\_\phi(x, y)$: scores a **complete** response. Trained on human preferences. Fixed during PPO training. Provides the learning signal. **Value Function** $V(s\_t)$: predicts the **expected reward** the current policy will achieve from state $s\_t$ (prompt + tokens generated so far). Serves as a baseline for variance reduction. Updated during training. In practice, the value function is typically implemented as the **policy model with a scalar head** (one extra linear layer), not a fully separate model. This is also why PPO is expensive: it needs to keep the policy, reference model, reward model, and value head all in memory. GRPO removes the value function entirely. </div>

## PPO: Surrogate Objective <div style="font-size: 0.9em;"> REINFORCE updates the policy using: $\mathcal{L}(\theta) = \mathbb{E}\_t\left[\hat{A}\_t \log p\_\theta(y\_t|x)\right]$ Problem: we sample from $p\_{\theta\_{\text{old}}}$ but update $p\_\theta$. Use importance sampling: $$\mathcal{L}(\theta) = \mathbb{E}\_t\left[\frac{p\_\theta(y\_t|x)}{p\_{\theta\_{\text{old}}}(y\_t|x)} \hat{A}\_t\right] = \mathbb{E}\_t\left[r\_t(\theta) \cdot \hat{A}\_t\right]$$ where $r\_t(\theta) = \frac{p\_\theta(y\_t | x, y\_{\lt t})}{p\_{\theta\_{\text{old}}}(y\_t | x, y\_{\lt t})}$ is the probability ratio. If $\hat{A}\_t > 0$: maximize $r\_t$ (make this token more likely). If $\hat{A}\_t < 0$: minimize $r\_t$ (make this token less likely). But what if $r\_t$ becomes very large? The policy changes too much and training collapses. </div>

## PPO: Clipping Visualized PPO clips the surrogate objective to prevent large updates: $\mathcal{L}\_{\text{PPO}}(\theta) = \mathbb{E}\_t\left[\min\left(r\_t(\theta) \hat{A}\_t, \; \text{clip}(r\_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}\_t\right)\right]$ <img src="images/ppo_clipping.png" style="max-width: 50%;"> <p class="citation"><a href="https://arxiv.org/abs/1707.06347">Schulman et al. (2017)</a></p>

## PPO: Clipped Objective <div style="font-size: 0.8em;"> $\mathcal{L}\_{\text{PPO}}(\theta) = \mathbb{E}\_t\left[\min\left(r\_t(\theta) \hat{A}\_t, \; \text{clip}(r\_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}\_t\right)\right]$ **Example** ($\epsilon = 0.2$, $\hat{A}\_t = +1$, good token): | $r\_t$ | in $[1-\epsilon, 1+\epsilon]$? | unclipped $r\_t \hat{A}\_t$ | clipped | $\min$ | |--------|-------------------------------|---------------------------|---------|--------| | 0.5 | No (too low) | 0.5 | 0.8 | **0.5** (keep: still learning) | | 1.0 | Yes | 1.0 | 1.0 | **1.0** (no clipping needed) | | 1.5 | No (too high) | 1.5 | 1.2 | **1.2** (clip: too greedy) | The $\min$ lets the policy recover when $r\_t$ is too low, but caps the gain when $r\_t$ is too high. This keeps each update close to the old policy. *Note*: In practice, a KL penalty $\beta \cdot \text{KL}(p\_\theta \| p\_{\text{ref}})$ is also added to prevent drift from the reference model over the full training run. Clipping handles per-step stability; KL handles long-term drift. <p class="citation"><a href="https://arxiv.org/abs/1707.06347">Schulman et al. (2017)</a></p> </div>

## PPO: Clipped Objective <div style="font-size: 0.8em;"> $$\mathcal{L}\_{\text{PPO}}(\theta) = \mathbb{E}\_t\left[\min\left(r\_t(\theta) \hat{A}\_t, \; \text{clip}(r\_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}\_t\right)\right]$$ **Example** ($\epsilon = 0.2$, $\hat{A}\_t = -1$, bad token): | $r\_t$ | in $[1-\epsilon, 1+\epsilon]$? | unclipped $r\_t \hat{A}\_t$ | clipped | $\min$ | |--------|-------------------------------|---------------------------|---------|--------| | 0.5 | No (too low) | -0.5 | -0.8 | **-0.8** (zero gradient: already suppressed) | | 1.0 | Yes | -1.0 | -1.0 | **-1.0** (no clipping needed) | | 1.5 | No (too high) | -1.5 | -1.2 | **-1.5** (gradient flows: still too likely) | At $r\_t = 0.5$: the token is already half as likely. The $\min$ picks the clipped term, which has no gradient. The policy stops pushing this token down further. At $r\_t = 1.5$: the token is still too likely. The $\min$ picks the unclipped term, and the gradient keeps suppressing it. </div>

## PPO: Algorithm **Input**: reward model $RM\_\phi$, reference model $p\_{\text{ref}}$, prompts $D$ Repeat: 1. **Sample**: generate responses $y \sim p\_\theta(y|x)$ for a batch of prompts 2. **Score**: compute rewards $RM\_\phi(x, y) - \beta \cdot \text{KL}(p\_\theta \| p\_{\text{ref}})$ 3. **Estimate advantages**: using the value function $V$ 4. **Update**: take gradient steps on the clipped surrogate objective 5. **Update value function**: fit $V$ to the observed returns Requires 4 models in memory: policy, reference, reward model, value function.

## InstructGPT: PPO at Scale InstructGPT applied PPO to a 175B parameter model: - ~30K tasks with human-written demonstrations (SFT) - ~33K comparison pairs for the reward model - PPO optimization with the KL penalty The 1.3B InstructGPT model was preferred over the 175B base GPT-3. This work led directly to ChatGPT. <p class="citation"><a href="https://arxiv.org/abs/2203.02155">Ouyang et al. (2022)</a></p>

## PPO: Challenges - **4 models in memory**: policy, reference, reward model, value function - **On-policy sampling**: must generate new responses each iteration (slow) - **Hyperparameter sensitivity**: clipping range $\epsilon$, KL coefficient $\beta$, learning rate - **Training instability**: reward hacking, value function collapse - **Engineering complexity**: requires careful distributed training setup These challenges motivated simpler and scalable alternatives like GRPO.

## Quiz: PPO In PPO, what happens if we set the clipping range $\epsilon = \infty$ (i.e., no clipping)? - A) PPO becomes equivalent to DPO - B) The value function is no longer needed - C) The policy updates become unconstrained and can change drastically in a single step - D) The reward model becomes unnecessary

## Quiz: Answer **C) The policy updates become unconstrained and can change drastically in a single step** Without clipping, PPO reduces to standard policy gradient (REINFORCE with baseline). The clipping is what makes PPO "proximal": it keeps each update close to the previous policy. Large unconstrained updates often lead to training collapse.

Lecture Outline

• Direct Preference Optimization (DPO)
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)

## GRPO: Motivation PPO requires a learned value function (critic) to estimate advantages. This adds memory and complexity. **Key idea**: Instead of learning a value baseline, estimate it from a **group of samples**. Generate multiple responses to the same prompt, and use the group's mean reward as the baseline. <p class="citation"><a href="https://arxiv.org/abs/2402.03300">Shao et al. (2024)</a></p>

## GRPO: Objective Function <div style="font-size: 0.85em;"> For each prompt $x$, sample a group of $G$ responses $\\{y\_1, \ldots, y\_G\\} \sim p\_{\theta\_{\text{old}}}(y|x)$. $\mathcal{L}\_{\text{GRPO}}(\theta) = \mathbb{E}\_{x \sim D} \left[\frac{1}{G} \sum\_{i=1}^{G} \min\left(r\_i(\theta) \hat{A}\_i, \; \text{clip}(r\_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}\_i\right)\right]$ where $r\_i(\theta) = \frac{p\_\theta(y\_i|x)}{p\_{\theta\_{\text{old}}}(y\_i|x)}$ and the advantage is: $$\hat{A}\_i = \frac{R\_i - \text{mean}(\\{R\_1, \ldots, R\_G\\})}{\text{std}(\\{R\_1, \ldots, R\_G\\})}$$ The group statistics replace the learned value function. <p class="citation"><a href="https://arxiv.org/abs/2402.03300">Shao et al. (2024)</a></p> </div>

## GRPO: Algorithm **Input**: reward function $R$ (learned or verifiable), reference model $p\_{\text{ref}}$, prompts $D$ Repeat: 1. **Sample a group**: for each prompt $x$, generate $G$ responses $\\{y\_1, \ldots, y\_G\\}$ from $p\_{\theta\_{\text{old}}}$ 2. **Score**: compute rewards $R\_i = R(x, y\_i)$ for each response 3. **Normalize**: compute group-normalized advantages $\hat{A}\_i$ 4. **Update**: take gradient steps on the clipped objective with KL penalty No value function needed. Only 2 models in memory: policy and reference.

## GRPO: Why Group Normalization Works <div style="font-size: 0.8em;"> Consider a math problem with $G = 8$ sampled solutions: | Response | Correct? | Reward | Normalized Advantage | |----------|----------|--------|---------------------| | $y\_1$ | Yes | 1 | +1.15 | | $y\_2$ | No | 0 | -0.58 | | $y\_3$ | Yes | 1 | +1.15 | | ... | ... | ... | ... | Correct solutions get positive advantage (reinforced). Wrong solutions get negative advantage (suppressed). The normalization ensures stable gradients regardless of the reward scale. </div>

## DeepSeek-R1: GRPO for Reasoning DeepSeek-R1 used GRPO with verifiable rewards (math correctness) to train reasoning models. The model learned to produce long chains of thought, including self-verification and backtracking, without any human demonstrations of reasoning. GRPO + verifiable rewards enabled "reasoning emergence" purely through RL. <p class="citation"><a href="https://arxiv.org/abs/2501.12948">DeepSeek-AI (2025)</a></p>

## Quiz: PPO vs GRPO What is the main difference between how PPO and GRPO estimate advantages? - A) PPO uses a learned value function; GRPO uses the mean reward of a group of samples - B) PPO uses verifiable rewards; GRPO uses learned rewards - C) PPO generates one sample per prompt; GRPO generates multiple - D) PPO clips the objective; GRPO does not

## Quiz: Answer **A) PPO uses a learned value function; GRPO uses the mean reward of a group of samples** Both generate multiple samples and both use clipping. The key difference is the baseline: PPO learns a separate value network, while GRPO computes the baseline from the group statistics of the current batch. This removes one model from memory and simplifies the training pipeline.

## Summary: Methods <div style="font-size: 0.85em;"> | Method | Data | Online? | Advantage | Models in Memory | |--------|------|---------|-----------|-----------------| | DPO | Preference pairs $(y\_w, y\_l)$ | No | N/A (supervised loss) | 2 (policy, reference) | | PPO | Prompts + RM | Yes | Value function (GAE) | 3-4 (policy, reference, RM, value head) | | GRPO | Prompts + reward | Yes | Group normalization | 2 (policy, reference) | </div>

## Summary: Objective Functions <div style="font-size: 0.8em; text-align: left;"> **DPO**: $\mathcal{L}\_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\left(\beta \log \frac{p\_\theta(y\_w|x)}{p\_{\text{ref}}(y\_w|x)} - \beta \log \frac{p\_\theta(y\_l|x)}{p\_{\text{ref}}(y\_l|x)}\right)\right]$ **PPO**: $\mathcal{L}\_{\text{PPO}} = \mathbb{E}\_t\left[\min\left(r\_t(\theta) \hat{A}\_t, \; \text{clip}(r\_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}\_t\right)\right]$ **GRPO**: Same clipped objective as PPO, but $\hat{A}\_i = \frac{R\_i - \text{mean}(R)}{\text{std}(R)}$ from a group of $G$ samples Key differences: - DPO: offline, supervised, no sampling during training - PPO: online, per-token advantages from learned value function - GRPO: online, per-sequence advantages from group statistics </div>

## Key Takeaways 1. **DPO** derives a closed-form supervised loss from preference data, avoiding RL entirely 2. **PPO** uses online RL with clipping for stable updates and a value function for per-token credit assignment 3. **GRPO** simplifies PPO by replacing the value function with group-level reward normalization 4. All three methods use a **KL penalty** to prevent drift from the reference model 5. The **policy gradient** (REINFORCE) is the foundation: the log-derivative trick converts an intractable sum into a sample-based estimate

# Questions?

## Notation Summary <div style="font-size: 0.75em; text-align: left;"> | Symbol | Meaning | |--------|---------| | $p\_\theta$ | Policy (the LM being trained) | | $p\_{\text{ref}}$ | Reference model (frozen copy, usually the SFT model) | | $p\_{\theta\_{\text{old}}}$ | Policy from the previous iteration (for importance sampling) | | $RM\_\phi(x, y)$ | Reward model (scores a complete response) | | $V(s\_t)$ | Value function (predicts expected reward at token $t$) | | $\hat{A}\_t$ | Advantage estimate at token $t$ | | $r\_t(\theta)$ | Probability ratio $\frac{p\_\theta(y\_t|x,y\_{\lt t})}{p\_{\theta\_{\text{old}}}(y\_t|x,y\_{\lt t})}$ | | $\beta$ | KL penalty coefficient | | $\epsilon$ | Clipping range for PPO/GRPO | </div>