# LoRA 如何记忆？大语言模型微调中的参数记忆定律

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-28 08:00
- AIHOT 分数：65
- AIHOT 链接：https://aihot.virxact.com/items/cmpqb0qos03dqslno8pq37m3b
- 原文链接：https://arxiv.org/abs/2605.30260

## AI 摘要

本研究使用 LoRA 作为控制探针，系统量化了大语言模型的精确参数记忆能力。提出了参数记忆定律，建立了损失减少量与有效参数及序列长度之间的稳健幂律关系。在 token 层面的分析揭示确定性相变，表明预测概率 p > 0.5 是贪心解码下实现逐字记忆的充分条件。基于此，设计了阈值引导的优化策略 MemFT，能动态重分配训练预算以提升记忆保真度与效率。代码将发布于 https://github.com/zjunlp/ParametricMemoryLaw。

## 正文

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.