# CMSC 25700/35100: ## Natural Language Processing ## Lecture 13: Post-training <p style="text-align: center;"> <strong>Chenhao Tan</strong><br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/1">Submit Music Recommendations</a>, kudos to Mina Lee <br> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/2">Submit jokes</a> </p>

## Lecture Outline * Lecture maintenance * Perplexity notebook * Post-training recipe * Reward optimization

## Min-k-prob: Intuition If a model has **seen** a text during training, it will assign high probability to most tokens, even the unusual ones. **Key idea**: Look at the $k$% of tokens with the **lowest** probabilities. If even those have high probabilities, the model likely memorized the text. <p class="citation"><a href="https://arxiv.org/abs/2310.16789">Detecting Pretraining Data from Large Language Models. Shi et al. (2024)</a></p>

## Min-k-prob: How to Decide Min-k-prob score alone is meaningless. You need a **comparison**: 1. Compute min-k-prob on **suspected data** (e.g., benchmark test examples) 2. Compute min-k-prob on **reference data** (non-member text of similar style/domain) 3. If the suspected data has significantly higher scores, it suggests contamination Example: compute min-k-prob on MMLU questions vs. newly written questions in the same format. If MMLU scores are much higher, the model likely saw MMLU during training. This is a **membership inference** problem: "is this specific text in the training set?" <p class="citation"><a href="https://arxiv.org/abs/2310.16789">Detecting Pretraining Data from Large Language Models. Shi et al. (2024)</a></p>

## Exchangeability Test: Intuition Benchmark examples are often stored as **ordered lists** (e.g., Example 1, Example 2, ..., Example 100). If a model saw the benchmark in order during training, it will find the **original ordering** more likely than a random shuffle. **Key idea**: Compare the model's log-probability on the original order vs. random permutations of the same examples. A contaminated model will prefer the original order. **Limitation**: Only detects contamination when the benchmark was ingested in its original order (e.g., as a document or webpage). <p class="citation"><a href="https://arxiv.org/abs/2310.17623">Proving Test Set Contamination in Black Box Language Models. Oren et al. (2024)</a></p>

## Quiz You want to check if a model was trained on the MMLU test set. You only have API access (no model weights). Which method can you use? - A) Only Min-k-prob - B) Only the exchangeability test - C) Both Min-k-prob and the exchangeability test - D) Neither, because both require access to model weights

## Quiz: Answer **C) Both Min-k-prob and the exchangeability test** - Min-k-prob needs per-token probabilities of the actual tokens in the text. Most APIs expose this via the `logprobs` parameter. - The exchangeability test needs to compare log-probabilities of different orderings of examples, which can also be done through the API. - Neither method requires access to model weights. Both work with black-box API access.

Lecture Outline

• Lecture maintenance
• Perplexity notebook
• Post-training recipe
• Reward optimization

Lecture Outline

• Lecture maintenance
• Perplexity notebook
• Post-training recipe
• Reward optimization

## Goal of Post-training Pretraining produces a model that predicts the next token. It learns from all of the internet: textbooks, forums, code, nonsense, toxic content, ... The model can complete documents, but it does not know **what you want**. Post-training goals: - **Follow instructions**: do what the user asks - **Be helpful**: give useful, accurate responses - **Be safe**: refuse harmful requests, avoid generating toxic content - **Be honest**: say "I don't know" rather than hallucinate

## Base Model vs. Post-trained Model **Prompt**: "What is the capital of France?" **Base model** (document completion): might continue with "What is the capital of Germany? What is the capital of Italy? ..." **Post-trained model** (assistant): "The capital of France is Paris." The base model has the knowledge. Post-training teaches it to use that knowledge in the way users expect.

## Instruction Finetuning (SFT) Collect a dataset of (instruction, desired response) pairs. Finetune the pretrained model using the standard language modeling objective: $$\mathcal{L}\_{\text{SFT}}(\theta) = -\mathbb{E}\_{(x, y) \sim D}\left[\sum\_{t=1}^{|y|} \log p\_\theta(y\_t | x, y\_{\lt t})\right]$$ Only compute loss on the **response tokens** $y$, not the instruction $x$. This teaches the model the format and style of helpful responses.

## Instruction Finetuning: Data Where do (instruction, response) pairs come from? - **Human-written**: contractors write both instructions and responses (expensive, high quality) - **Distillation**: use a stronger model (e.g., GPT-4) to generate responses (cheap, scalable) - **Self-instruct**: the model generates its own instruction-response pairs, filtered for quality <p class="citation"><a href="https://arxiv.org/abs/2203.02155">Ouyang et al. (2022)</a>; <a href="https://arxiv.org/abs/2304.03277">Taori et al. (2023)</a>; <a href="https://arxiv.org/abs/2212.10560">Wang et al. (2023)</a></p>

## Why Is SFT Not Enough? SFT is **imitation learning**: the model learns to copy human-written responses token by token. **Problem 1: Token-level vs. response-level quality** The SFT loss treats every token equally. But response quality is a property of the whole response, not individual tokens. A single wrong token (e.g., "2 + 2 = 5") can ruin an otherwise good response. **Problem 2: No way to distinguish "good" from "great"** SFT maximizes the probability of the reference response. It cannot learn that one response style is preferred over another.

## Why Is SFT Not Enough? (cont.) **Problem 3: Exposure bias** During training, the model always sees gold tokens as context. During inference, it sees its own generated tokens. Errors compound. **Problem 4: Bounded by the demonstrations** SFT can only be as good as the training data. If the demonstrations are mediocre, the model will be mediocre. It cannot discover better strategies on its own. RL-style approaches address these issues by optimizing a **response-level reward** and letting the model **explore** its own outputs.

## Quiz: SFT Which of the following is a limitation of supervised finetuning? - A) It requires generating samples from the model during training - B) It cannot leverage pretrained model weights - C) The loss function treats every token equally regardless of importance - D) It requires pairwise preference annotations

## Quiz: Answer **C) The loss function treats every token equally regardless of importance** SFT uses a token-level cross-entropy loss that weights all tokens the same. A critical token like "5" in "2+3 = 5" gets the same weight as "the". Options A and D describe RL-based methods (PPO, DPO), not SFT. Option B is wrong because SFT starts from a pretrained model.

## The Post-training Pipeline **Step 1**: Instruction Finetuning (SFT) - Teaches format and instruction following **Step 2**: Reward Optimization - Optimizes a response-level objective - Learns from the model's own outputs (online methods) - Multiple approaches: RLHF (PPO), DPO, GRPO, ... Most production LLMs follow this two-step recipe. <p class="citation"><a href="https://arxiv.org/abs/2203.02155">Ouyang et al. (2022)</a>; <a href="https://arxiv.org/abs/2307.09288">Touvron et al. (2023)</a></p>

## The Reality Is More Complicated <img src="images/deepseek.png" style="max-height: 500px;"> <p class="citation"><a href="https://arxiv.org/abs/2501.12948">DeepSeek-AI (2025)</a></p>

Lecture Outline

• Lecture maintenance
• Perplexity notebook
• Post-training recipe
• Reward optimization

## From Imitation to Reward Optimization **SFT** says: "here is a good response, make it more likely." **RL-style training** says: "generate a response, I will tell you how good it is, improve from there." The key difference: RL-style training requires a **reward signal** $R(x, y) \in \mathbb{R}$. This reward is computed on the **complete response**, not token by token. It captures response-level quality.

## The Reward Optimization Goal For an instruction $x$ and LM sample $y$: $$\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}[R(x, y)]$$ Generate responses from the current policy, score them, and update the policy to produce higher-scoring responses. But where does $R(x, y)$ come from?

## Where Do Rewards Come From? **Option 1: Human feedback** (RLHF) - Ask humans to compare responses - Train a reward model to predict human preferences **Option 2: Verifiable rewards** (RLVR) - Use an automatic checker: math answer correct? code passes tests? - No learned model needed, no human in the loop **Option 3: AI feedback** (RLAIF) - Use a stronger LLM to judge response quality - Cheaper than humans, more scalable Each source has different trade-offs in cost, accuracy, and coverage.

## RLHF: Learned Reward Models Asking humans to rate responses directly is noisy and expensive. **Better approach**: ask for **pairwise comparisons**. "Is response A better than response B?" This is more reliable than "Rate this response from 1-10." Train a **reward model** $RM\_\phi(x, y)$ on these comparisons.

## RLVR: Verifiable Rewards For some tasks, we can check correctness automatically: | Task | Reward signal | |------|--------------| | Math | Is the final numerical answer correct? | | Code | Do the unit tests pass? | | Formal proofs | Does the proof checker accept? | | Constrained generation | Does the output satisfy the constraints? | Verifiable rewards are **exact**: no approximation error, no reward hacking. But they only work for tasks with checkable outputs. Open-ended tasks (writing, conversation) still need learned rewards.

## Why RLVR Matters RLVR removes the reward model entirely. The reward comes from the task itself. This is an important component in DeepSeek-R1. The simplicity of the reward enabled the model to develop its own reasoning strategies (chain of thought, backtracking) through exploration. <p class="citation"><a href="https://arxiv.org/abs/2501.12948">DeepSeek-AI (2025)</a></p>

## Reward Hacking Learned reward models are imperfect. The policy can learn to exploit their weaknesses. Examples: - Generating longer responses (reward models often prefer length) - Repeating confident-sounding phrases - Producing outputs that "look good" but are factually wrong **Solution**: add a KL penalty to stay close to the reference model: $$\max\_\theta \mathbb{E}\_{y \sim p\_\theta(y|x)}\left[RM\_\phi(x, y) - \beta \, \text{KL}\left(p\_\theta(y|x) \| p\_{\text{ref}}(y|x)\right)\right]$$

## Quiz: Reward Hacking A reward model trained on human preferences tends to give higher scores to longer responses. After RL training, the model produces very long, repetitive responses. What is the most likely cause? - A) The KL penalty $\beta$ is too large - B) The SFT training data contained long responses - C) The model has too few parameters - D) The model is reward hacking by exploiting the length bias in the reward model

## Quiz: Answer **D) The model is reward hacking by exploiting the length bias in the reward model** The policy found that producing longer outputs reliably increases the reward model's score, even if the extra content is low quality. This is a classic example of reward hacking. Increasing $\beta$ (the KL penalty) or improving the reward model can help mitigate this.

# Questions?