CAVEWOMAN:语言输入与输出压缩对大语言模型的影响
阅读原文· arxiv.org研究提出两通道评估协议Cavewoman,同时测量任务准确率、实际成本及与无约束基准的文本一致性。在5个数据集上对8个模型进行5级压缩测试,发现输出压缩降低API模型实际成本1.4–2.4倍(最佳达3倍),开源权重模型同样受益;输入压缩则严格双输:平均成本升高约1.15倍(最差1.8倍,强压缩2.7倍),准确率暴跌,半数正确生成不再蕴含模型自身无约束基准文本。该分歧在长度控制重评分、多重比较校正及补充语义度量下仍然稳健。
"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at https://github.com/danielle34/cavewoman.