人们很容易认为,任何能用简短程序解决的任务,都可以通过思维链教给模型:写出步骤、进行微调,模型就能照做。本文表明,对于某一类可识别的程序,这一假设并不成立。实验平台包含九个推理任务,每个任务都来自一个确定性生成器;公开集和隐藏集共享相同的生成器,因此保留数据可作为测试准确率的代理。我将这些生成器逆向工程为 Python 求解器,将其呈现为思维链,并通过秩 ≤ 32 的 LoRA 将其蒸馏到一个 30B(3.5B 激活)的 Nemotron 模型上。前向可计算的任务很容易安装:查找/算术任务和一个 8 位布尔任务成功迁移(准确率分别 ≥ 0.99 和 0.68)。但密码算术任务则不然:即便一个搜索求解器能回答 71% 的实例,对其回溯搜索进行蒸馏,在十一种思维链设计、基于可验证奖励的强化学习以及自训练下,准确率仍停留在 0.01-0.07。这并非能力差距。模型在 97-100% 的行上完成了算术运算,并在 71% 的情况下将正确密码排在前八位;但它无法将搜索作为从左到右的推导过程持续推进。微调学会了可验证消除步骤的形态,但其判断却变成了无条件的模板,正确率仅为 16-57%(“判断即 token”)。这一上限在 3B 到 671B 的不同基座模型、微调和提示方法中均成立;一项受控干预隔离了原因:揭示密码密钥(这使推导变为前向)将相同实例的准确率从 0.03 提升至 0.57。当一个程序的唯一解法是在无信息结构上进行搜索时,就不存在可供模仿的忠实前向思维链。只有移除搜索、将其组合核心预计算成目录、并将追踪过程简化为回忆加验证,该任务才变得可学习;第一名解决方案正是通过这种方式达到了私有排行榜 0.92 的分数。蒸馏所得到的是记忆和验证,而非搜索。
It is tempting to assume any task solvable by a short program can be taught to a model as its chain-of-thought: write the steps out, fine-tune, and the model follows. This paper shows the assumption fails for an identifiable class of procedures. The testbed is nine reasoning tasks, each from a deterministic generator; public and hidden splits share generators, so held-out data proxies test accuracy. I reverse-engineer the generators into Python solvers, render them as chain-of-thought, and distill into a rank-<= 32 LoRA over a 30B (3.5B-active) Nemotron model. Forward-computable tasks install readily: lookup/arithmetic and an 8-bit boolean task transfer (>= 0.99 and 0.68). Cryptarithm does not: distilling its backtracking search holds at 0.01-0.07 across eleven chain-of-thought designs, RL from verifiable rewards, and self-training, even though a search solver answers 71% of instances. This is not a capability gap. The model does the arithmetic on 97-100% of lines and ranks the correct cipher in its top eight on 71%; it cannot carry the search forward as a left-to-right derivation. Fine-tuning learns the shape of a verifiable elimination step while its verdicts become unconditional templates, correct only 16-57% of the time ("verdict-as-token"). The ceiling holds across backbones from 3B to 671B and across fine-tuning and prompting; a controlled intervention isolates the cause: revealing the cipher key, which turns the derivation forward, lifts the same instances from 0.03 to 0.57. When a procedure's only solution is search over information-free structure, no faithful forward chain-of-thought exists to imitate. The task becomes learnable only by removing the search, precomputing its combinatorial core into a catalog and reducing the trace to recall plus verification; the 1st-place solution reaches Private LB 0.92 this way. What distills is memorization and verification, not search.