# Parcae：稳定循环语言模型的缩放定律

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-14 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo0iu82g01g1sli23sdqv42i
- 原文链接：https://arxiv.org/abs/2604.12946

## AI 摘要

研究团队提出稳定循环架构Parcae，通过将循环建模为非线性时变动力系统并约束注入参数的谱范数，解决了传统循环模型的残差爆炸与损失尖峰问题。该架构验证困惑度较先前模型降低6.3%，并建立了训练FLOPs与循环次数、数据量的可预测幂律关系。在13亿参数规模下，Parcae在固定预算下较Transformer基线在CORE和Core-Extended基准上分别提升2.99和1.18分，达到两倍规模Transformer模型87.5%的性能。

## 正文

Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop. While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes. We address these challenges by recasting looping as a nonlinear time-variant dynamical system over the residual stream. Via a linear approximation to this system, we find that instability occurs in existing looped architectures as a result of large spectral norms in their injection parameters. To address these instability issues, we propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization. As a result, Parcae achieves up to 6.3% lower validation perplexity over prior large-scale looped models. Using our stable looped architecture, we investigate the scaling properties of looping as a medium to improve quality by increasing FLOPs in training and test-time. For training, we derive predictable power laws to scale FLOPs while keeping parameter count fixed. Our initial scaling laws suggest that looping and data should be increased in tandem, given a fixed FLOP budget. At test-time, we find that Parcae can use looping to scale compute, following a predictable, saturating exponential decay. When scaled up to 1.3B parameters, we find that Parcae improves CORE and Core-Extended quality by 2.99 and 1.18 points when compared to strong Transformer baselines under a fixed parameter and data budget, achieving a relative quality of up to 87.5% a Transformer twice the size.