返回
原文
Apple Machine Learning Research（RSS）
51AI 编辑部评分，满分 100

多模态大语言模型对齐的全面研究：Apple 团队独立拆解偏好对齐各环节

2026-08-03 08:00· 18小时前
跳到正文
AI 摘要

Apple 研究团队系统梳理了多模态大语言模型（MLLM）中的偏好对齐方法，将算法分为离线（如 DPO）与在线（如 online-DPO）两类，并发现两者结合可在特定场景下提升模型性能。团队还提出无需额外标注或外部模型的新型多模态偏好数据构建方法 Bias-Driven Hallucination Sampling（BDHS），在多项基准上取得与既有对齐工作相当的竞争力。

原文 · 未翻译

AuthorsElmira Amirloo*, Jean-Philippe Fauconnier*, Christoph Roesmann*, Christian Kerl†, Rinu Boney†, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently, multiple works have introduced preference datasets for MLLMs and examined different alignment methods, including Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). However, due to variations in datasets, base model types, and alignment methods, it remains unclear which specific elements contribute most significantly to the reported improvements in these works. In this paper, we independently analyze each aspect of preference alignment in MLLMs. We start by categorizing the alignment algorithms into two groups, offline (such as DPO), and online (such as online-DPO), and show that combining offline and online methods can improve the performance of the model in certain scenarios. We review a variety of published multimodal preference datasets and discuss how the details of their construction impact model performance. Based on these insights, we introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) that needs neither additional annotation nor external models, and show that it can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks.

  • * Authors contributed equally as first authors.
  • † Authors contributed equally.

Related readings and updates.

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

September 22, 2025research area Computer Visionconference NeurIPS

Despite Contrastive Language-Image Pretraining (CLIP)‘s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality…

多模态安全/对齐论文/研究
阅读原文导出 Markdown
返回精选列表
51导出 Markdown

多模态大语言模型对齐的全面研究：Apple 团队独立拆解偏好对齐各环节

Apple Machine Learning Research（RSS）·2026-08-03 08:00·18小时前
阅读原文· machinelearning.apple.com
AI 摘要

Apple 研究团队系统梳理了多模态大语言模型（MLLM）中的偏好对齐方法，将算法分为离线（如 DPO）与在线（如 online-DPO）两类，并发现两者结合可在特定场景下提升模型性能。团队还提出无需额外标注或外部模型的新型多模态偏好数据构建方法 Bias-Driven Hallucination Sampling（BDHS），在多项基准上取得与既有对齐工作相当的竞争力。

原文 · 保持原样，未翻译

AuthorsElmira Amirloo*, Jean-Philippe Fauconnier*, Christoph Roesmann*, Christian Kerl†, Rinu Boney†, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently, multiple works have introduced preference datasets for MLLMs and examined different alignment methods, including Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). However, due to variations in datasets, base model types, and alignment methods, it remains unclear which specific elements contribute most significantly to the reported improvements in these works. In this paper, we independently analyze each aspect of preference alignment in MLLMs. We start by categorizing the alignment algorithms into two groups, offline (such as DPO), and online (such as online-DPO), and show that combining offline and online methods can improve the performance of the model in certain scenarios. We review a variety of published multimodal preference datasets and discuss how the details of their construction impact model performance. Based on these insights, we introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) that needs neither additional annotation nor external models, and show that it can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks.

  • * Authors contributed equally as first authors.
  • † Authors contributed equally.

Related readings and updates.

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

September 22, 2025research area Computer Visionconference NeurIPS

Despite Contrastive Language-Image Pretraining (CLIP)‘s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality…

多模态安全/对齐论文/研究
阅读原文machinelearning.apple.com