# CMSC 25700/35100: ## Natural Language Processing ## Lecture 12: Benchmarks and Evaluation <p style="text-align: center;"> <strong>Chenhao Tan</strong><br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/1">Submit Music Recommendations</a>, kudos to Mina Lee <br> <a href="https://github.com/uchicago-nlp-course/winter-2025-students/issues/2">Submit jokes</a> </p>

## Logistics * [Project](https://docs.google.com/spreadsheets/d/1gMsgMXXWlo4jfq7qCSumu-0HmY2TKxN0hWFiBwkZpQE/edit?gid=0#gid=0) - mentors have been assigned - a new run for all projects have been done, check it out, please fork the repo if you plan to use part of the code * [Feedback form](https://forms.gle/uSRgFc21m9xcoYhh9) * Chris Potts is giving linguistics colloquium next week

## Lecture Outline * Evaluation Basics * The Benchmark Crisis * Modern Evaluation

Lecture Outline

• Evaluation Basics
• The Benchmark Crisis
• Modern Evaluation

## Close-ended Evaluation Tasks with a limited number of correct answers (classification, NLI, NER, ...) | Metric | Definition | |--------|------------| | Accuracy | Correct predictions / Total predictions | | Precision | True positives / Predicted positives | | Recall | True positives / Actual positives | | F1 | Harmonic mean of precision and recall | Fast and cheap to compute. Similar to standard ML evaluation.

## Close-ended Evaluation **Confusion matrix** (binary classification): | | Predicted Positive | Predicted Negative | |--|--------------------|--------------------| | **Actual Positive** | TP = 40 | FN = 10 | | **Actual Negative** | FP = 5 | TN = 45 |

## Metrics from the Confusion Matrix | | Pred Positive | Pred Negative | |--|---------------|---------------| | **Actual Positive** | TP = 40 | FN = 10 | | **Actual Negative** | FP = 5 | TN = 45 | **Accuracy** = $\frac{TP + TN}{TP + TN + FP + FN} = \frac{40 + 45}{100} = 0.85$ **Precision** = $\frac{TP}{TP + FP} = \frac{40}{40 + 5} = 0.89$   (of those we predicted positive, how many are correct?) **Recall** = $\frac{TP}{TP + FN} = \frac{40}{40 + 10} = 0.80$   (of the actual positives, how many did we find?) **F1** = $\frac{2 \cdot P \cdot R}{P + R} = \frac{2 \cdot 0.89 \cdot 0.80}{0.89 + 0.80} = 0.84$   (harmonic mean of precision and recall)

## Open-ended Evaluation Tasks with many valid outputs (translation, summarization, dialogue, ...) **ROUGE-N** (recall-based): what fraction of the reference's n-grams appear in the candidate? $$\text{ROUGE-N} = \frac{|\text{n-grams in ref} \cap \text{n-grams in cand}|}{|\text{n-grams in ref}|}$$ - $N$: n-gram size (ROUGE-1 = unigrams, ROUGE-2 = bigrams) - Numerator: number of n-grams that appear in both reference and candidate - Denominator: total number of n-grams in the reference

## ROUGE-1 Worked Example Reference: **"Well heck yes it is"** (5 unigrams: Well, heck, yes, it, is) Candidate: **"Heck yes!"** (2 unigrams: Heck, yes) $$\text{ROUGE-1} = \frac{\text{matched ref unigrams}}{\text{total ref unigrams}} = \frac{|\lbrace\text{heck, yes}\rbrace|}{|\lbrace\text{Well, heck, yes, it, is}\rbrace|} = \frac{2}{5} = 0.40$$ Note: ROUGE is **recall** over the reference. It asks: how much of the reference is covered by the candidate?

## Beyond N-gram Overlap Reference: "Well heck yes it is"    (ROUGE-1 recall) | Response | Matched | ROUGE-1 | Correct? | |----------|---------|---------|----------| | "Heck yes!" | heck, yes | 2/5 = 0.40 | Yes | | "Yes!" | yes | 1/5 = 0.20 | Yes (false negative) | | "You know it!" | it | 1/5 = 0.20 | Yes (false negative) | | "Heck no!" | heck | 1/5 = 0.20 | No (false positive) | ROUGE cannot distinguish "Heck yes!" from "Heck no!" when they share similar surface tokens with the reference. **BERTScore** addresses this by using contextual embeddings and cosine similarity instead of exact matching. <p class="citation"><a href="https://arxiv.org/abs/1904.09675">BERTScore: Evaluating Text Generation with BERT. Zhang et al. (2020)</a></p>

## BLEU: Precision-based N-gram Overlap BLEU flips the direction: what fraction of the **candidate's** n-grams appear in the reference? **Step 1**: Compute modified precision to get BLEU-N: $p_n = \frac{\sum_{\text{ngram} \in \text{cand}} \min(C_{\text{cand}}, C_{\text{ref}})}{\sum_{\text{ngram} \in \text{cand}} C_{\text{cand}}}$ **Step 2**: Combine using a geometric mean (that is where the $\exp$ comes from): $\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{N} \frac{1}{N} \log p_n\right) = BP \cdot \left(\prod_{n=1}^{N} p_n\right)^{1/N}$ **Step 3**: Brevity penalty $BP = \min\left(1,\; e^{1 - r/c}\right)$ penalizes candidates shorter than the reference ($c$ = candidate length, $r$ = reference length). Commonly used for machine translation evaluation.

## Human Evaluation The "gold standard" for text generation, but: - Slow and expensive - Not repeatable: "Only 5% of human evaluations are repeatable" - Inter-annotator and intra-annotator disagreement - Biases and shortcuts when incentives are misaligned <p class="citation"><a href="https://aclanthology.org/2023.findings-acl.226/">Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP. Belz et al. (2023)</a></p>

## Chatbot Arena Side-by-side blind comparison at scale - Users type their own prompts - Two anonymous models respond - Users vote for the better response - Votes aggregated using Elo rating system The most widely trusted LLM ranking today. <p class="citation"><a href="https://arxiv.org/abs/2403.04132">Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Chiang et al. (2024)</a></p>

## LLM-as-a-Judge Use an LLM as a reference-free evaluator - Much faster and cheaper than human evaluation - "GPT-4 can match human preferences, achieving over 80% agreement" **Benchmarks using LLM-as-a-Judge**: - **MT-Bench**: 80 multi-turn questions across 8 categories (writing, math, coding, ...). GPT-4 rates responses on a 1-10 scale. - **AlpacaEval**: ~800 instructions. GPT-4 picks the better response between a candidate model and a reference model. **Limitations**: Length bias (prefers longer outputs), position bias (prefers the first response), self-bias (models prefer their own outputs) <p class="citation"><a href="https://arxiv.org/abs/2306.05685">Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al. (2023)</a></p>

Lecture Outline

• Evaluation Basics
• The Benchmark Crisis
• Modern Evaluation

## The Benchmark Explosion | Model | Year | # Benchmarks Evaluated | |-------|------|------------------------| | Transformer | 2017 | 2 (WMT datasets) | | GPT-1 | 2018 | ~8 tasks | | GPT-2 | 2019 | ~15 tasks | | GPT-3 | 2020 | 40+ tasks | Single numbers (BLEU, perplexity) are no longer sufficient. GPT-3 was essentially an evaluation paper, with much less focus on training or architecture.

## The Streetlight Effect > A man searches for his keys under a streetlight. Someone asks, "Why are you looking here?" He replies, "Because the light is better here." We measure what is *easy* to measure, not what *matters*. - Token-level metrics let us track progress - But they did not capture what we actually cared about - The things that matter most (reasoning, reliability, safety) are the hardest to evaluate

## Three Horsemen of the Benchpocalypse **1. Data Contamination** Models trained on "the entire internet" inevitably see test data **2. Benchmark Saturation** Models exceed human performance; ceiling effects render benchmarks useless **3. Gaming the Metrics** Optimizing for benchmark scores instead of real capabilities

## Data Contamination Models trained on the internet have likely seen test data.

## Discussion: Why Is Contamination Hard to Detect? A model scores 95% on a benchmark. How would you determine whether it has been contaminated?

## Discussion: Why Is Contamination Hard to Detect? A model scores 95% on a benchmark. How would you determine whether it has been contaminated? 1. What makes contamination different from genuine learning? 2. If we paraphrase the test questions, what would we expect to see? 3. Why can't we just check whether the test data appears in the training set?

## Data Contamination Models trained on the internet have likely seen test data. **Detection methods**: - **Min-k-prob**: Check if token probabilities are suspiciously high - **Exchangeability test**: Look for ordering signatures in training data No detection method currently works reliably. Closed models make verification even harder. <p class="citation"><a href="https://arxiv.org/abs/2310.16789">Detecting Pretraining Data from Large Language Models. Shi et al. (2024)</a>; <a href="https://arxiv.org/abs/2310.17623">Proving Test Set Contamination in Black Box Language Models. Oren et al. (2024)</a></p>

## Min-k-prob: Intuition If a model has **seen** a text during training, it will assign high probability to most tokens, even the unusual ones. **Key idea**: Look at the $k$% of tokens with the **lowest** probabilities. If even those have high probabilities, the model likely memorized the text. <p class="citation"><a href="https://arxiv.org/abs/2310.16789">Detecting Pretraining Data from Large Language Models. Shi et al. (2024)</a></p>

## Min-k-prob: How to Decide Min-k-prob score alone is meaningless. You need a **comparison**: 1. Compute min-k-prob on **suspected data** (e.g., benchmark test examples) 2. Compute min-k-prob on **reference data** (non-member text of similar style/domain) 3. If the suspected data has significantly higher scores, it suggests contamination Example: compute min-k-prob on MMLU questions vs. newly written questions in the same format. If MMLU scores are much higher, the model likely saw MMLU during training. This is a **membership inference** problem: "is this specific text in the training set?" <p class="citation"><a href="https://arxiv.org/abs/2310.16789">Detecting Pretraining Data from Large Language Models. Shi et al. (2024)</a></p>

## Exchangeability Test: Intuition Benchmark examples are often stored as **ordered lists** (e.g., Example 1, Example 2, ..., Example 100). If a model saw the benchmark in order during training, it will find the **original ordering** more likely than a random shuffle. **Key idea**: Compare the model's log-probability on the original order vs. random permutations of the same examples. A contaminated model will prefer the original order. **Limitation**: Only detects contamination when the benchmark was ingested in its original order (e.g., as a document or webpage). <p class="citation"><a href="https://arxiv.org/abs/2310.17623">Proving Test Set Contamination in Black Box Language Models. Oren et al. (2024)</a></p>

## Quiz You want to check if a model was trained on the MMLU test set. You only have API access (no model weights). Which method can you use? - A) Only Min-k-prob - B) Only the exchangeability test - C) Both Min-k-prob and the exchangeability test - D) Neither, because both require access to model weights

## Quiz: Answer **C) Both Min-k-prob and the exchangeability test** - Min-k-prob needs per-token probabilities of the actual tokens in the text. Most APIs expose this via the `logprobs` parameter. - The exchangeability test needs to compare log-probabilities of different orderings of examples, which can also be done through the API. - Neither method requires access to model weights. Both work with black-box API access.

## Benchmark Saturation Benchmarks expire faster than we can create them: <img src="images/dynabench-saturation.png" alt="Benchmark saturation over time" style="max-width: 50%;"> <p class="citation"><a href="https://arxiv.org/abs/2104.14337">Dynabench: Rethinking Benchmarking in NLP. Kiela et al. (2021)</a></p>

## Gaming the Metrics When a measure becomes a target, it ceases to be a good measure (Goodhart's Law). - Leaderboard pressure incentivizes benchmark-specific optimization - Models can be tuned to score well on benchmarks without real capability gains - Prompt formatting sensitivity: up to 76 accuracy points difference from subtle prompt changes <p class="citation"><a href="https://arxiv.org/abs/2402.01781">When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. Alzahrani et al. (2024)</a></p>

Lecture Outline

• Evaluation Basics
• The Benchmark Crisis
• Modern Evaluation

## From LLM Benchmarks to Agent Benchmarks Traditional LLM benchmarks test isolated capabilities (Q&A, classification, generation). Modern AI systems are **agents**: LLM + tools + environment + multi-step reasoning. New benchmarks must test **end-to-end task completion** in real environments: - Can the agent solve a real GitHub issue? - Can it set up a server from scratch? - Can it navigate a real website to complete a task?

## SWE-bench: Solving Real GitHub Issues Real GitHub issues from popular Python repositories. The agent must produce a working patch. **SWE-bench Verified** (500 human-validated issues): - Best: ~78.8% - Rapid progress: 4% (2023) to ~79% (2025) <p class="citation"><a href="https://arxiv.org/abs/2310.06770">SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Jimenez et al. (2024)</a></p>

## SWE-bench: Example Task **Repo**: `sqlfluff/sqlfluff`   **Issue**: [#1834](https://github.com/sqlfluff/sqlfluff/issues/1834) **Problem**: Add a `--quiet` flag to the `sqlfluff fix` CLI so it works cleanly in pre-commit hooks. Only options to increase verbosity exist. **Solution**: Add `-q`/`--quiet` (mutually exclusive with `--verbose`), modify the formatter to suppress output when verbosity is below zero, add test cases. Patch touches 3 files, ~47 lines changed. The agent must: read the issue, find the relevant code, produce a working patch, and pass existing tests. [Browse SWE-bench dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench)

## TerminalBench: Real Terminal Tasks 89 tasks in real terminal environments: compiling code, debugging, configuring servers, recovering data. - Tests practical system administration and development skills - Best models: ~65% - Many tasks require multi-step reasoning and tool use - Highlights gap between "knowing" and "doing" <p class="citation"><a href="https://arxiv.org/abs/2601.11868">TerminalBench: Benchmarking LLM Agents in Real Terminal Environments. Zhu et al. (2026)</a></p>

## TerminalBench: Example Task **Task**: `train-fasttext`   Category: Model Training   Difficulty: Hard **Problem**: Given Yelp review data, train a FastText text classification model that achieves at least 0.62 accuracy while keeping the model size under 150MB. **What the agent must do**: 1. Prepare training data in the correct format 2. Write and run a training script 3. Tune hyperparameters to balance accuracy vs. model size 4. Verify the result passes the automated checks All work happens in a terminal. No GUI, no notebook. [Browse TerminalBench tasks](https://www.tbench.ai/)

## OSWorld **OSWorld**: 369 real computer tasks across Ubuntu/Windows/macOS - The best agent now exceeds human performance slightly - Tasks: file management, app usage, system configuration Trend: agents are approaching (and sometimes exceeding) human performance on computer use tasks. <a href="https://arxiv.org/abs/2404.07972">OSWorld. Xie et al. (2025)</a></p>

## OSWorld: Example Task **Task**: "Could you help me extract data in the table from a new invoice uploaded to my Google Drive, then export it to a Libreoffice calc .xlsx file in the desktop?" <iframe src="https://drive.google.com/file/d/1rcw53toCpRIYih_R5-bvgEEoDKSLGqHK/preview?t=19" width="640" height="360" allow="autoplay" allowfullscreen></iframe> [Browse OSWorld tasks](https://os-world.github.io/explorer.html)

## METR Time Horizon **50% time horizon**: the task duration (for a human expert) at which an AI agent succeeds 50% of the time. Methodology: human experts attempt each task, completion times are averaged, then a logistic curve is fit to predict agent success as a function of task duration. <p class="citation"><a href="https://arxiv.org/abs/2503.14499">Measuring AI Ability to Complete Long Tasks. METR (2025)</a></p>

## METR Time Horizon <img src="images/metr.png" alt="METR time horizon trend" style="max-height: 500px;"> <p class="citation"><a href="https://arxiv.org/abs/2503.14499">Measuring AI Ability to Complete Long Tasks. METR (2025)</a></p>

## METR: Key Observations - AI agents complete tasks **several times faster** than humans when successful (they code much faster and take fewer actions) - Performance drops substantially when scoring **holistically** rather than algorithmically - Tasks are primarily in software engineering, ML, and cybersecurity - Human baselines use skilled contractors (~5 years experience, top-100 universities) with **low context** (like a new hire) <p class="citation"><a href="https://metr.org/time-horizons/">metr.org/time-horizons</a></p>

## METR: Limitations **Time horizon does NOT mean autonomous work duration.** It measures task difficulty (how long a human takes), not how long the agent runs. **Domain-specific.** Tasks cover software/ML/cybersecurity only. AI capabilities are "jagged" relative to humans and vary across domains. **Tasks are cleaner than real work.** They are well-specified and algorithmically scored. Real jobs involve interacting with people, ambiguous requirements, and success metrics that cannot be automatically checked. **Human baselines may overestimate difficulty.** Contractors have low context (like a new hire), not the familiarity of someone who works on that codebase daily. An "8-hour time horizon" does not mean AI can do an 8-hour job.

## Discussion: What Does the METR Trend Mean? Discuss with your neighbor: * What types of tasks will remain hard even if the time horizon keeps growing? * Is task duration even the right way to measure AI capability? What does it miss?

## Summary: Old vs. Modern Evaluation | Aspect | Traditional | Modern | |--------|-------------|--------| | What | Isolated NLP tasks | End-to-end task completion | | Where | Static datasets | Real environments (terminals, browsers, OSes) | | How | Accuracy on test set | Success rate on real tasks | | Who | The model alone | Agent (model + tools + planning) | | Metric | BLEU, F1, accuracy | Task completion rate, time horizon | The shift from "can the model answer correctly?" to "can the agent complete the job?"

# Questions?