ReFreeKV:迈向无阈值KV缓存压缩
阅读原文· arxiv.orgReFreeKV 提出一种无阈值的 KV 缓存压缩方法,通过自适应调整预算分配来消除对输入特定阈值的依赖,解决了传统方法因阈值选择不当导致的性能退化问题。在涵盖不同上下文长度、任务类型和模型规模的 13 个数据集上的实验表明,该方法在保持全缓存性能的同时实现了高效压缩。相关代码已公开。
To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection. As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for "threshold-free" methods that adaptively adjust budget allocation while preserving full-cache performance. We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective. Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency. Our code is publicly released at https://github.com/Patrick-Ni/ReFreeKV.