原文 · 未翻译
Researchers pinpoint why larger language models pick up skills that small ones miss
A new study suggests that instead of endlessly inflating models, it may be more efficient to increase the frequency of specific tasks in training data to anchor rare skills in smaller models.
A new study from researchers at Anthropic, Stanford, and other institutions explains why larger language models learn certain tasks that smaller ones fail at. The finding goes beyond the conventional wisdom that big models simply learn faster.
In some cases, small models can't reliably learn rare tasks even with extremely long training runs. Even well-known scaling laws show that a small model never reaches the loss of a large one, no matter how much data you throw at it.
Common tasks crowd out rare ones
To isolate the mechanism, the researchers tested a mix of tasks with varying frequency and complexity. A model with N neurons gets assigned the N "most useful" features, where usefulness is based on how often a task appears and how important it is. Frequent, simple tasks get priority. Rare, complex ones get dropped. In the experiments, only models that were large enough learned tasks that made up just 0.25 percent of the training data.
The core of the paper is its explanation of why size helps. As long as frequent tasks aren't well-learned yet, they pull the model strongly in their direction at every training step, overwriting much of what the model picked up about rare tasks. Once a large model has mostly mastered the frequent tasks, that pull fades. The freed-up capacity goes to rare tasks, and learned signals are more likely to stick.
Small models rarely reach that point, according to the study. They fall into an "update-and-forget" loop. A rare example gets briefly learned, then largely erased by the next training steps on frequent tasks. When the next rare example shows up, the model starts over from scratch.
One experiment was designed to cleanly separate this effect. The total frequency of a rare task stays constant, but the gap between individual observations varies. The larger the gap, the more the signal decays in narrow models. Wide models hold onto it better between observations and build on it.
Real language models show the same pattern
To test the theory during pre-training, the team trained OLMo models ranging from 4 million to 4 billion parameters on up to 210 billion tokens from the Dolma corpus. They mixed two artificial tasks into the data, a number comparison and a modular addition, with frequencies ranging from about 1,000 instances per batch down to one instance every ten batches.
Only the larger OLMo models picked up the rare tasks by learning the rule behind them and applying it to new cases, rather than just memorizing individual examples.