# MNAFT：面向图像翻译的多模态大语言模型模态神经元感知微调

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-18 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo82l6vo03lbslmljcnhecs0
- 原文链接：https://arxiv.org/abs/2604.16943

## AI 摘要

针对多模态大语言模型在图像翻译中难以捕捉细粒度文本信息的问题，本文提出模态神经元感知微调方法MNAFT。该方法通过指令驱动的激活分析，识别视觉与语言模块中的语言无关及语言特定神经元，选择性更新与目标任务相关的神经元参数，同时保留其他预训练知识。实验表明，MNAFT在多个基准测试中显著优于级联模型、标准全量微调及现有参数高效微调方法，有效缩小了视觉文本与文本间的模态差距。

## 正文

Multimodal large language models (MLLMs) have shown impressive capabilities, yet they often struggle to effectively capture the fine-grained textual information within images crucial for accurate image translation. This often leads to a modality gap between visual text inputs and textual inputs/outputs for image translation. Existing methods, primarily relying on instruction fine-tuning, risk parameter redundancy of pre-trained knowledge, hindering generalization performance. To address this, we introduce modality neuron-aware fine-tuning (MNAFT), a novel approach that takes advantage of the specialized roles of individual neurons within MLLMs for enhanced image translation. MNAFT identifies language-agnostic and language-specific neurons in both vision and language modules through an instruction-driven activation analysis, evaluating their importance in various translation tasks. We then perform selective fine-tuning, updating only the parameters of language-specific and language-agnostic neurons within the selected layers relevant to the target task, while preserving the knowledge encoded in other neurons and layers. Our extensive experiments on multiple benchmarks demonstrate that MNAFT significantly outperforms state-of-the-art image translation methods, including cascaded models, standard full fine-tuning, and parameter-efficient tuning techniques. Furthermore, we provide comprehensive analysis, including visualizations of neuron activations and clustering patterns, to offer insights into the roles of different neuron groups in mediating cross-modal understanding and facilitating accurate language-specific translation.