Large vision-language models have shown strong progress in UI-to-code generation, yet their test-time self-evolution remains unstable. We first identify a fundamental obstacle, termed visual repair coupling: a local code edit may propagate through layout, style, and component dependencies, correcting one visual mismatch while degrading regions that were previously faithful. To address this issue, we present RubSE, a Rubric-guided Self-Evolution framework that uses rubrics to represent visual feedback as a structured visual-repair context. At each refinement round, RubSE generates typed candidate rubrics, selects one prioritized repair target, and stores previously selected rubrics as history, thereby steering each revision toward a well-scoped visual repair while discouraging repeated or over-broad changes. Evaluations across six VLMs and three UI-to-code benchmarks demonstrate that RubSE substantially outperforms naïve self-evolution in final-round and best-round settings, achieving more stable refinement trajectories and a higher trajectory-level performance ceiling. Further analysis shows that RubSE mitigates trajectory collapse by improving recovery from severe visual regressions, and that stronger rubric generators can transfer effective visual-repair guidance to weaker code improvers.
RubSE:用评分规则作为视觉修复上下文实现UI到代码生成的自进化
AI 导读
RubSE框架通过将视觉反馈表示为结构化评分规则上下文,解决了大视觉语言模型在UI到代码生成中测试时自进化不稳定的问题。该方法每轮生成候选规则、选择优先修复目标并保留历史,引导修订聚焦于明确范围。在六个VLM和三个基准上,RubSE显著优于朴素自进化,轨迹更稳定且能缓解崩溃、提升对严重视觉回退的恢复能力。
HuggingFace Daily Papers(社区热门论文)
43
AI 编辑部评分,满分 100RubSE:用评分规则作为视觉修复上下文实现UI到代码生成的自进化
RubSE框架通过将视觉反馈表示为结构化评分规则上下文,解决了大视觉语言模型在UI到代码生成中测试时自进化不稳定的问题。该方法每轮生成候选规则、选择优先修复目标并保留历史,引导修订聚焦于明确范围。在六个VLM和三个基准上,RubSE显著优于朴素自进化,轨迹更稳定且能缓解崩溃、提升对严重视觉回退的恢复能力。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org