SigmaScale:基于SVD低秩分解与学习缩放矩阵的LLM压缩方法
阅读原文· arxiv.orgSigmaScale通过学习辅助缩放矩阵S改进基于截断SVD的大语言模型压缩。该方法在激活感知压缩损失下优化两组向量,定义对角行和列缩放变换。学习缩放降低了权重矩阵的有效内在秩(有效秩熵减少),且降低幅度与压缩损失强相关。在Llama 3.1 8B Instruct和Qwen3-8B上的实验表明,SigmaScale在困惑度和零样本基准上与当前最先进SVD压缩方法竞争力相当,在特定任务上表现出优势,成为降低LLM推理计算成本的有效选项。
We present SigmaScale, a method for learning auxiliary scaling matrices S to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.