# Fast Weight Attention 用于持续学习：在线学习规则下的循环状态压缩

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-08-27 08:00
- AIHOT 分数：33
- AIHOT 链接：https://aihot.virxact.com/items/cmtgx3e700cimrochffigt6ku
- 原文链接：https://arxiv.org/abs/2608.27763

## AI 摘要

论文将循环快速权重记忆与选择性状态空间模型的状态转换视为在线学习规则，在自回归语义下研究该规则。针对前缀预测目标，推导了平方误差回归与负内积目标的归一化一阶更新，提出 Falcon-1/2/3 及其内积变体 Falcon-1A/2A/3A，并提供循环、掩码并行与分块并行形式。代表性变体在语言建模上保持竞争力，并改善了变长数字加法的长度外推。

## 正文

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_{t-1}),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.