# FlowBP：一种面向Flow Matching的奖励反向传播设计空间探索框架

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-10 00:36
- AIHOT 分数：47
- AIHOT 链接：https://aihot.virxact.com/items/cmqq2kjqj060pslp59w2rkbmx
- 原文链接：https://arxiv.org/abs/2606.11075

## AI 摘要

针对文本到图像Flow Matching模型与人类偏好对齐时，完整采样轨迹无法存储及跨步雅可比积导致梯度膨胀的问题，FlowBP提出统一代理轨迹框架，将反向轨迹本身作为设计对象。该框架分离奖励模型输入、活跃集、积分权重和桥耦合四个选择，并实例化三个变体：FlowBP-Sparse（稀疏Euler重建）、FlowBP-Bridge（受控桥耦合）和FlowBP-Lagrange（高阶跳跃求积）。三者通过活跃集大小限制内存，梯度链至多含一个雅可比因子。在SD3.5-M、FLUX.1-dev和FLUX.2-Klein-base上，三个变体在偏好、质量和组合指标上均优于直接梯度基线。

## 正文

Aligning text-to-image flow matching models with human preferences via direct reward backpropagation is sample-efficient but hampered by two well-known pathologies: activations cannot be stored across the full sampling trajectory at modern model scale, and chained Jacobian products across steps inflate the reward gradient as it travels back to early indices. Connector-based methods, such as LeapAlign, address these issues by replacing the full backward trajectory with a short pinned path, highlighting a useful decoupling between sampling and optimization. However, the quality of the resulting gradient depends on how accurately this short path approximates the full rollout, especially over long intervals. We propose FlowBP, a unified surrogate-trajectory framework that treats the backward trajectory itself as the design object. FlowBP keeps a no-gradient cached rollout for sampling, then builds a lightweight backward surrogate from cached and selectively re-forwarded velocities. This view separates four choices: the reward-model input, active set, integration weights, and bridge coupling, and recovers prior direct-gradient methods as particular settings. Within this framework, we instantiate three variants: FlowBP-Sparse uses sparse Euler reconstruction, FlowBP-Bridge adds controlled bridge coupling, and FlowBP-Lagrange raises the order of leap quadrature. All three bound memory by the active-set size and limit gradient chaining to at most one Jacobian factor. Across SD3.5-M, FLUX.1-dev, and FLUX.2-Klein-base on preference, quality, and compositional metrics, the three variants improve over direct-gradient baselines on most metrics.