Rohan Paul@rohanpaul_ai

2026-04-12 19:45·81天前

AI 摘要

研究通过贪婪剪枝方法（逐个删除对模型似然度影响最小的token）评估LLM推理token的功能重要性。发现符号数学token比语法叙述更能经受剪枝，表明模型内部存在重要性排序。重要性具有动态性，早期可丢弃的token可能在上下文减少后变得关键。注意力模式可预测剪枝分数，说明功能重要性在模型内部可见。该发现有助于使chain-of-thought更可解释，而非仅仅缩短长度。

Reasoning tokens in LLMs are not equal.

Models seem to know which parts of their own reasoning matter most.

What survives pruning is usually the part doing actual computational work， not the fluent narration wrapped around it.

The method is clever in a plain way. Start with a full chain of thought， delete one token at a time， and keep deleting whichever removal hurts the model's likelihood least. The resulting order becomes a functional ranking， not of what sounds important to us， but of what the model itself seems to need.

Here's the interesting part.

If a model's reasoning were just verbose decoration， pruning should look mostly random once you preserve the answer. Instead， the paper finds structure. Symbolic math tokens survive pruning far more than grammar， narration， and referential bookkeeping， which means the model is not treating all tokens as equally useful.

That matters because the test is behavioral， not rhetorical.

Students trained on these greedily pruned chains do better than students trained on several other pruning baselines， including a method supervised by a frontier model， at the same reasoning length. So the pruning signal is not merely interpretable. It is useful.

The deeper point is that importance is dynamic.

A token that looks expendable early can become important later as surrounding context disappears， which argues against the comforting idea that reasoning has a fixed salience map you can read off once and reuse forever.

And yet the signal is not inaccessible. The paper shows attention patterns alone can predict pruning scores surprisingly well， suggesting that functional importance is partly visible in the model's internals before you do the expensive deletion game.

So this is less about making chain-of-thought shorter than about making it legible.

The claim is not that pruned tokens are causally irrelevant in any philosophical sense. The cleaner claim is better： LLMs appear to encode a workable internal ranking of which reasoning tokens are carrying the load.