# ShadowPEFT：用于参数高效微调的影子网络

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-21 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo9rx7qe0495sls26nk9opvo
- 原文链接：https://arxiv.org/abs/2604.19254

## AI 摘要

研究团队提出ShadowPEFT，一种集中式参数高效微调框架，通过深度共享的影子模块实现层级别细化。该方法在每个Transformer层维护并行影子状态并反复演化，将适应机制从分布式权重扰动（如LoRA）转向集中式层空间优化。影子模块与主干解耦，支持跨深度复用、独立预训练及分离部署，适用于边缘计算。实验表明，在同等可训练参数预算下，ShadowPEFT在生成与理解任务上达到或超越LoRA与DoRA性能。

## 正文

Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.