Tangram:解锁非均匀KV缓存压缩以高效服务多轮对话大语言模型
阅读原文· arxiv.org多轮LLM服务中,对话历史KV缓存随轮次增长,内存成为吞吐瓶颈。非均匀KV压缩可在注意力头间分配不同预算以保留精度,但现有服务栈因头间异构性导致页面碎片和预填充延迟增加。Tangram发现头级保留遵循输入无关的两层结构规律,仅需50样本离线校准,并通过Budget Reservation、Ragged Paging和Ahead-of-Time Load Balancing三个静态方案替代动态处理。基于vLLM实现,Tangram匹配现有非均匀压缩方法的精度,端到端吞吐量相比全KV基线最高提升2.6倍。代码已开源。
Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput. Non-uniform KV compression, which allocates heterogeneous budgets across attention heads, preserves accuracy far better than uniform schemes, yet remains impractical: modern serving stacks assume identical KV lengths across heads, so heterogeneity traps freed memory as page fragmentation, spends up to 25% of prefill time reclaiming scattered pages, and skews GPU workloads that inflate decode latency by up to 1.7times or burn 15--20% of each decode step on re-planning. We observe that this heterogeneity need not be discovered at runtime: head-wise retention follows a two-level structural regularity -- an input-invariant head ranking with narrowly bounded per-head ratios -- that can be calibrated offline from as few as 50 samples. Building on this insight, we present Tangram, a serving framework that statically resolves what prior systems handle dynamically: Budget Reservation fixes each head's post-compression footprint at scheduling time, eliminating page reclamation; Ragged Paging clusters similar-budget heads into independent page tables, turning fragmentation into reclaimable memory; and Ahead-of-Time Load Balancing precomputes balanced GPU partitions with zero runtime planning. Implemented on vLLM, Tangram serves as a drop-in substrate for existing non-uniform compression methods, matching their accuracy while improving end-to-end throughput by up to 2.6times over the full-KV baseline. Our implementation is publicly available at https://github.com/aiha-lab/TANGRAM.