# HOLA：为线性注意力引入海马体式补充记忆

- 来源：elvis (@omarsar0)
- 发布时间：2026-07-03 23:38
- AIHOT 分数：44
- AIHOT 链接：https://aihot.virxact.com/items/cmr53qqs707k9sll5lmpkqemr
- 原文链接：https://x.com/omarsar0/status/2073068756293869685

## AI 摘要

新论文提出 HOLA，通过将压缩递归状态与小精确记忆配对，提升线性注意力的长程召回能力。它保留 delta 规则状态作为压缩记忆，并添加有界精确 KV 缓存，形成半参数测试时记忆。缓存无需学习式逐出模块，仅保留预测残差实际写入状态的 token。在 340M 参数、15B SlimPajama 上，HOLA 将 Wikitext 困惑度从 27.32 降至 22.92，低于全注意力 Transformer++ 的 26.88；RULER 长程 needle 召回在 32k tokens（16 倍训练长度）仍保持稳定。论文：arxiv.org/abs/2607.02303。

## 正文

NEW paper worth reading.

(bookmark it)

The basic idea is to pair a compressive recurrent state with a small exact memory, which helps to recover long-range recall without giving up the efficiency of linear attention.

More on it below:

Linear-attention and state-space models compress the whole prefix into a fixed-size state. That buys O(1) memory, but when many key-value associations compete, earlier facts get overwritten and needle recall degrades.

HOLA gives linear attention a hippocampal complement. It keeps the usual delta-rule state as compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory.

The state models linearly compressible structure while the cache stores associations that should not be forced through it. The cache writes without a learned eviction module, keeping only tokens whose prediction residual was actually committed to the state.

At 340M parameters on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92, below a full-attention Transformer++ at 26.88, and stays robust on RULER needle recall out to 32k tokens, 16x its training length.

Paper: https://arxiv.org/abs/2607.02303

Learn to build effective AI agents in our academy: https://academy.dair.ai/