FlashMemory-DeepSeek-V4: 通过前瞻稀疏注意力实现闪电索引超长上下文
阅读原文· arxiv.orgFlashMemory-DeepSeek-V4(FM-DS-V4)提出Lookahead Sparse Attention(LSA)推理范式,基于DeepSeek-V4架构构建神经记忆索引器,主动预测未来上下文需求,仅保留查询关键KV块。采用解耦训练策略,索引器作为独立双编码器训练,无需加载主干模型。在LongBench-v2、LongMemEval、RULER等长上下文基准上,平均物理KV缓存压缩至全上下文基线的13.5%,下游精度平均提升0.6%;在500K极端长度下,物理KV开销减少超过90%,且不损害主干模型的核心推理能力。
Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose Lookahead Sparse Attention (LSA), a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a backbone-free decoupled training strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this "less is more" paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6% absolute margin on average). Crucially, at extreme 500K scales, FlashMemory suppresses the physical KV cache overhead by over 90% without destabilizing the backbone's core reasoning capacities.