# 论文《Do LLMs Capture the Diversity in their Training Data？》发现 LLM 输出多样性低于训练数据

- 来源：Rohan Paul (@rohanpaul_ai)
- 发布时间：2026-09-04 14:59
- AIHOT 分数：47
- AIHOT 链接：https://aihot.virxact.com/items/cmtmm6w7702k3romybv8wmmiu
- 原文链接：https://x.com/rohanpaul_ai/status/2095768574245884058

## AI 摘要

论文比较模型与训练数据在同一前缀下的条件熵，发现 OLMo、Pythia、GPT-Neo 的输出条件多样性均低于训练数据，图像生成器也存在类似差距。OLMo 20K 样本上训练数据得分 338.58，贪心解码为 218.88，nucleus 采样 287.31，ancestral 采样 297.31；调高温度到 T=1.9 虽匹配训练熵但精度和外部条件 NLL 变差。

## 正文

LLMs learn a narrower set of answers than their training data, even when you sample instead of using greedy decoding.

The paper compares model continuations with training continuations for the same prefixes, measuring output variety after accounting for the prompt.

Across OLMo, Pythia, and GPT-Neo, training data had higher conditional diversity under every decoding method tested.

For OLMo at 20K samples, the training score was 338.58, versus 218.88 with greedy decoding, 287.31 with nucleus sampling, and 297.31 with ancestral sampling.

Larger models did not remove the gap, and the same pattern appeared in image generators.

Simply raising temperature is not a clean fix: around T=1.9 matched OLMo's training entropy, but produced worse precision and external conditional NLL.

Their alternative generates multiple candidates and reweights plausible but underused outputs.

That increased measured diversity without retraining or generating new samples.

– arxiv. org/abs/2609.02275

Title: "Do LLMs Capture the Diversity in their Training Data?"