What if most RL gains come from 1 transformer layer?

They test a strange idea and find it often works: train 1 layer, freeze the rest.

This paper finds that reward training is less like whole-model learning and more like layer selection.

The problem is that RL post-training usually updates the whole model, so it hides which parts actually learn the useful behavior.

The paper freezes almost every layer, trains only 1 transformer layer at a time, then checks how much of full RL improvement that single layer recovers.

This score, called layer contribution, reveals that the useful RL changes are not spread evenly through the network.

Across 7 models, 3 RL methods, and math, code, and agent tasks, the best layers usually sit near the middle, while early and late layers help much less.

The practical result is that training only the best middle layers can beat full RL, such as 69.1 math accuracy versus 66.4 on Qwen3-8B.

----

Link – arxiv. org/abs/2607.01232v1

Title: "Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training"