8 月 23 日，Surya Narreddi 发布了一段精美的视频，展示了一个语言模型绘制的水彩画。该模型通过 p5.brush（一个“为 p5.js 添加自然绘图工具”的库）编写 JavaScript。这段视频迅速走红，截至撰写本文时，观看量已超过 150 万次。
这段视频附带了一篇博客文章，解释了该项目早期、范围更窄阶段的训练过程——当时是特写花卉，而非视频中的完整构图，遗憾的是尚未开放任何工件。他的网站显示，完整的技术报告即将发布，所以请务必关注他。最初的创意来自他本人，源于艺术与设计领域，他在那方面的技能远超于我。我的尝试则侧重于工程方面，以开放的方式复现这一方法，并公开每一个环节。
注意：关于该项目的背景，由 Surya 本人讲述，请观看他的论文答辩视频。
在本文中，我尝试使用 TRL 和 OpenEnv 复现他的想法。参考池数据集、RL 环境、训练脚本以及训练好的模型，全部开放。
整个流水线端到端运行在 Hugging Face 上：
- 在 Jobs 上进行训练
- RL 环境和评分模型作为 Spaces 部署
- 通过 Inference Providers 进行成对评判
- 所有工件都托管在 Hub 上，并汇集在一个集合中
两个 Space 就绪后，配方就只剩一条命令。复制环境和评分模型，为奖励混合设置两个环境变量，然后启动：
hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
--env-url https://<you>-watercolour-env.hf.space \
--model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
--subject 'a peach hibiscus' --references 4 \
--top-p 0.95 --top-k 20 \
--lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
--scale-rewards none \
--steps 110 --n-episodes 240 --num-generations 8 \
--per-device-batch-size 1 --gradient-accumulation-steps 8 \
--max-completion-length 8192 \
--run-tag my-run --out <you>/watercolour-grpo --push-to-hub
本文其余部分讲述的是达成这一结果的过程，所有内容都已放入代码仓库。
我一步步跟随原博客的步骤，只在确有必要时才做改动。我自己的每个想法都记入清单而非直接用于实验，这份清单最终成为文末的“我接下来想尝试的方向”，旁边附有全部已发布产物的完整列表。如果你已经读过他的文章，那么框架和奖励设计会让你感到熟悉。新增内容在于开源的实现、人工评分的样本池，以及三组经过训练并相互对比的奖励混合方案，这一切从“你需要构建的 RL 环境”一节开始。
三次运行，对应三种奖励混合方案，并行演进。每一帧展示的是某一步的中位数画作。目前还无需区分它们，文章会说明哪次运行对应哪种方案。
人们为何喜爱它
这些画作看起来松散、不完美、带着手工感，而当下图像模型产出的却是完美（统计上平均）的画面。我猜测，这种反差正是视频走红的重要原因。这让我想起生成式 AI 艺术的早期，那时的意义在于探索媒介本身。DeepDream（2015）本是调试工具，却被人们变成了艺术；像 Edmond de Belamy（2018）这样的作品出自艺术家对 GAN 能力的探索；而 Mario Klingemann 等艺术家在那几年用神经网络创作出梦幻般的肖像。
这个项目让人感觉更接近那些早期岁月。在他的论文中，Surya 描述了通往这里的路径。他最初是给文生图模型写提示词，在那种模式下，提示词是你唯一能操控的杠杆，而增加细节能带来的控制力提升终究是有限的。直接训练模型本身则更进一步。这个想法的另一半在于媒介。模型会编写一段约 150 行的 JavaScript 程序来绘制图像。模型的输出就是代码。你可以阅读它、编辑它并重新运行它，每一笔笔触背后的决策都是可见的。而这种风格源自一个限制：模型只被允许使用该库中的十个方法。更多细节见下文。
在同一时期，Anna Ridler 拍摄了数千朵郁金香，为每一朵都手工标注，将数据集本身作为艺术品展出，后来又用这个数据集训练了一个模型。我是在构建这个项目时，通过 AI 智能体带回的参考文献发现她的作品的，我非常喜欢，因为这个项目做了非常相似的事情——手工策展一组图像，然后针对它们进行训练。
基于品味的强化学习
近期大多数关于语言模型的强化学习工作，使用的都是可以被验证的奖励。例如，有已知答案的数学题、能通过测试的代码，或者对错分明且运行成本低廉的评分器。而这个项目更接近那个较早的例外——RLHF，即模型从人类偏好中学习出一个奖励模型。
在这里，奖励是审美偏好。没有正确答案。这个项目真正要探讨的问题是：你是否能对品味进行强化学习。
这个奖励，正如他的博客所定义的那样，也正如我所构建的强化学习环境所实现的那样：
| 术语 | 权重 | 衡量内容 |
|---|---|---|
| 门控 | 0.05 | 草图可编译、能绘制出内容、不偷工减料 |
| 长度 | 0.05 | 温和引导生成更长的代码片段 |
| 成对评判器 | 0.60 | 风格，与从样本池中抽取的参考作品进行对比 |
| HPSv3 | 0.30 | 对渲染结果的审美偏好 |
HPSv3 是一个开源的 7B 偏好模型。给它一张图片和一段文字描述，它就会返回一个分数，表示一个人对这张图片的偏好程度。它是在大量人类对图片对进行选择的标注数据上训练的，因此它的分数是许多人品味的平均值。成对评判器是 Qwen3-VL-30B-A3B-Instruct，这是一个通过 HF Inference Providers 调用的通用视觉模型。成对评判器会将候选画作与从图池中随机选取的四张参考图放在一起进行对比，并依据一段书面描述来指导评判时应侧重哪些方面（晕染、透明水洗、柔和边缘），每次对比都会以两种呈现顺序进行，最终得分是候选画作在对比中获胜的比例。它唯一的标准就是图池本身，因此它的分数就是我的品味——正如那些评分所编码的那样。
奖励函数中的四个项里有两个是模型。它们都是对某人品味的代理指标。
这些就是 Narreddi 收敛得到的权重。图池在这里定义了品味。这就把工作从调整超参数转移到了构建那个决定何为美的集合上。
我用这个奖励函数训练了三次运行。它们的区别仅在于两个模型评判器之间的权重分配方式不同：
| 运行 | 成对评判器 | HPSv3 | 角色 |
|---|---|---|---|
| 评判器主导 | 0.60 | 0.30 | 原始混合，在第 110 步停止 |
| hps 主导 | 0.30 | 0.60 | 中间点，在第 110 步停止 |
| 仅 hps | 0.00 | 0.90 | 验证运行，在第 60 步停止 |
我先从仅 hps 开始，以验证该流程确实能够学习。一旦奖励开始上升且各项指标正常，就没有必要继续运行更久，因此我转而启动了两个更长的运行。这两个更长运行要回答的问题是：HPSv3 的能力有多少可以交给成对评判器？评判器权重越大，奖励就越代表我的品味而非大众品味，爬升的难度也就越大。顺带一提，如果你把权重推得足够高，或者你的风格与平均水平相差太远，模型可能会完全停止。
幸运的是，它并没有停止，两轮开启成对评判器的运行也都成功学到了东西。人工评分池至少从指标和最终画作来看，能够引导策略。具体数字如下。
免责声明。如果我们使用前沿模型，它已经能够根据提示词生成绘制水彩画的 JavaScript 代码。这是起点。这里的工作重点在于教会一个较小的模型完成这件事，并结合个人自身的艺术偏好。
你需要构建的 RL 环境
该环境封装了模型与奖励之间的所有环节，包括模型用于绘画的 JavaScript 库、限制其行为的系统提示词、渲染每幅草稿的无头 Chromium，以及拒绝作弊行为的门控机制。
这个库所做的工作比表面上看起来更多。p5.brush 由 @acamposuribe 开发，它模拟的是一种绘画媒介而非绘制形状：颜料会渗出填充区域的边缘，纸张具有纹理，笔触具有质量，流场会带动笔触移动。当模型调用 brush.fillBleed(0.25) 时，它是在决定墨水洇开的程度。
备注。p5.brush 的作者早在这一切发生之前就一直在尝试教机器绘画。2022 年，他创作了一个生成艺术系列，其中隐藏着一篇关于教 p5.js 像孩子一样绘画的日记：“它几乎不会用蜡笔……它无法遵循简单的指令。今天就到这里吧，太让人恼火了。”该系列原本计划包含三件作品，而他完成了两件。当 Surya 的视频走红后，他引用了该视频，分享了那篇日记，并表示这件作品是第三件，它自行到来了。
p5.brush 暴露了 47 个方法。提示词只允许使用其中 10 个：scaleBrushes、noStroke、fill、noFill、fillBleed、fillTexture、beginShape、vertex、endShape 和 circle。其余三十七个方法（线条、阴影线、自定义笔刷等）会破坏水彩效果。有了这十个方法，模型只能绘制填充形状，而库会为每一个形状添加颜料晕染效果。
某次 rollout 中 draw() 的一部分及其渲染结果。注释是模型自己写的。奖励值 0.864，129 行，第 22 步。每幅画的完整源代码都包含在 rollouts 数据集中。
他的博客文章为我节省了大量本可能浪费在反复迭代提示词上的时间。冗长的 API 参考会让模型编造出不存在的方法，而他经过 200 次 GEPA 迭代收敛到了一个没有文档的严格白名单。我遇到了同样的失败，于是手动编写了白名单。我对那个方案唯一的补充是一句话：每片花瓣画两到三次，先画一大笔，再在里面画一笔更小、更不透明的一笔。这个小小的改动让我的输出色彩丰富了许多。
备注。如果你是第一次听说 GEPA，它是一个自动提示词优化器。语言模型用通俗的语言反思当前提示词在哪些地方失败了，并提出一个更好的提示词，如此循环往复。
这道关卡是最后一块拼图。草图必须能够编译，必须使用该库而不是直接调用 p5，必须将真正的颜料画到画布上，并且不能试图欺骗评分器，例如通过在画布上写文字。
池子就是奖励函数
这个画库包含 178 幅画作，根据我个人偏好分为两档：喜爱和一般。所有画作实际上都是由模型生成的。四个开放权重模型通过 Inference Providers 调用，各自基于 iNaturalist 上一张真实、开放许可的芙蓉花照片，编写了 p5.brush 草图。一个视觉模型对每张草图给出了书面反馈，并经过三轮迭代优化。随后，每一幅最终渲染图都由我逐一评分，最终有 178 幅入选。
| 生成器 | 画作数量 |
|---|---|
| GLM-5.2 | 64 |
| Kimi-K3 | 57 |
| Qwen3-Coder-Next | 35 |
| Qwen3.5-122B-A10B | 22 |
在这里，我选择了四个不同的模型家族来测试它们的不同风格。这四款是在快速可靠性检查中每次都能生成有效草图的开放模型，另外两个候选模型因未通过该检查而被淘汰。如果你想生成自己的画库，也可以选择其他模型。
这两个层级，实际看起来就是这样。对评分持不同意见是合理的——现在，某个人的判断就成了奖励函数。
这些层级在奖励机制中发挥着实际作用。当两两比较的评判器抽取四个参考样本时，一半来自“喜爱”层级，一半来自“尚可”层级，因此策略总会面对一些它有时能击败的对手，而且无论对手来自哪个层级，获胜获得的奖励都是一样的。这是我自己为数不多的刻意改动之一：原始版本只与最高层级进行比较，而我保留了较容易的层级参与抽取，这样早期较弱的策略仍然能获得有效信号。
其中没有任何人类创作的画作，这是一个实实在在的局限。p5.brush 是一个小众库，其中带有可获取代码的人类作品也只有寥寥几件，远达不到训练语料库所需的数量，他的博客也提到了这一点。
这里有趣的想法，正如我之前已经讨论过的，是模型将学会模仿语料池中所包含的内容。如果我们把环境指向一个不同的数据集，奖励就会自动改变，而无需改动一行代码。对于我生成并公开分享的数据集，我也附带了源草图。
如果我们更仔细地审视这些评判器，会发现它们回答的是不同的问题。HPSv3 判断它是不是一朵花，而两两比较的评判器判断它是否以我选择的风格被画得很好。
再来一次 yolo 运行
在一切开始奏效之前，有一段漫长的奖励曲线平直期。如果你曾尝试在没有开放工件的情况下复现一篇研究论文或博客，你大概能感同身受。每一次运行都在检验我当时认为合理的、关于问题出在哪里的理论。一次运行要花很长时间，所以我一边分析上一次运行的结果，一边把下一次运行排进队列。一如既往，答案是从一个更简单、能跑通的东西开始，然后再在其基础上叠加。第一个真正学会的尝试是一个简单的控制任务，没有浏览器、没有裁判，而原因在于我的学习率实在太低了。
针对奖励做了三个实验，得到三条平直线。我更换了样本池、去掉了渲染器噪声、关闭了成对裁判，但曲线纹丝不动。真正让曲线动起来的那次运行改的是训练器配置，所以差异出在训练器上，而不是奖励组合。
另一个让我花了不少时间才发现的问题是正确调整 LoRA 参数。通常的 target_modules 列表假设的是稠密模型，而 Qwen/Qwen3.5-35B-A3B 是一个混合专家模型，它的大多数投影层命名方式不同，所以适配器只训练了四十层中的十层。我通过把它改成 all-linear 解决了这个问题，这样能覆盖每一个线性层。该架构中的路由专家是融合张量，即使 all-linear 也会让它们保持冻结，但其余每一层都挂上了适配器，而这已经足够让模型学会任务了。
修复方案是对 TRL 的 GRPOTrainer 做四处改动：
| 把 | 从 | 改为 | 为什么 |
|---|---|---|---|
| 学习率 | 2e-5 | 5e-5 | LoRA Without Regret 为 GRPO 设置的学习率上限 |
| 调度器 | 线性 | 带预热的常数 | 线性衰减在训练中途就已消耗掉大部分学习率，因此奖励始终没有起色 |
| scale_rewards | 组 | 无 | 组内一个门控被拒绝，导致其他所有优势都被压缩 |
| target_modules | 手工列表 | 全线性层 | 覆盖每一个线性层 |
这四项改动带来了第一次成功运行（仅超参数调整），奖励明显提升。
在这种配置下，三次运行都能正常学习。两次 judge 运行均设定为 200 步，在 110 步时停止，此时奖励仍在缓慢上升。每一步耗时十五到十八分钟，且不同混合方式之间的对比已经趋于稳定，因此我停止了这两次运行以节省算力。每次运行前三分之一和后三分之一的组平均奖励如下：
| 运行 | 步数 | 前三分之一 | 后三分之一 | Δ |
|---|---|---|---|---|
| 仅 hps | 60 | 0.58 | 0.71 | +0.13 |
| judge 主导 | 110 | 0.45 | 0.72 | +0.27 |
| hps 主导 | 110 | 0.57 | 0.82 | +0.24 |
三条曲线与我的品味所占权重完全对应。评判权重越高，起点就越低，攀升过程中的波动也越大。judge-led 在最初三十步几乎持平，之后才开始移动。这与解决调试问题用的是同一种方法：先把问题缩小到能学到东西的程度，再把困难的部分一次一个地加回去。
在使用了成对评判的两组运行中，成对评判项本身都在攀升。随着训练推进，模型在与样本池的对比中赢得更多比较，这是仅用 hps 无法做到的。任何一组运行中都没有出现各组奖励收敛到完全相同的情况——那是会扼杀梯度的 GRPO 失败模式。好奇的读者可以在仓库中找到按指标划分的曲线（HPSv3、画面覆盖率、熵），格式为 CSV。
完整的启动命令、硬件配置，以及将本次运行转变为另外两次运行所需的两个环境变量，都记录在配方中。
它实际学到了什么
在每一次运行中，模型首先学到的是停止生成糟糕的画作——那些近乎空白的画布和不成形的色块，其总奖励低于 0.3。在仅用 hps 的运行中，组均值上升的四分之三来自劣质画作的减少。在加入评判的运行中，下降更为陡峭：在 judge-led 中，低于 0.3 的 rollout 从 99 降至 16（按三段划分）；在 hps-led 中则从 37 降至 4。
这就是为什么最直观的视觉呈现——每一步的最佳画作——在仅用 HPS 的跑动中几乎看不出差异。它在整个跑动过程中移动了 +0.034，而中位数移动了 +0.155。学习效果在分布的中段清晰可见。
成对评判器改变的是顶端。在仅用 HPS 的情况下，画作变得更可靠，却没有变得更好。优质画作的质量仅为组均值增加了 +0.03，而一旦 HPSv3 看到了中心周围的花瓣和花茎，它就不再要求更多颜料。有了评判器参与，故事的另一半才显现出来。这里的“更好”意味着更接近样本池，也就是更接近我评为“好”或我更喜欢的作品。这在评判器主导的跑动中增加了 +0.12，在 HPS 主导的跑动中增加了 +0.16，每一步的最佳画作也有所提升，并且两次跑动中的颜料覆盖率都翻倍了（0.11 到 0.23，以及 0.13 到 0.30），而仅用 HPS 时几乎没怎么变动。有了可供超越的参照物，一幅好画还能变得更好，模型也开始因使用更多颜料而获得奖励。
还有一个发现。模型忽略了一条明确的指令，而它这样做是对的。系统提示词要求十五到三十个填充形状。如果我们看实际均值，它在 7 到 9 之间，而且 n_shapes 在任何跑动中与奖励的相关性都微乎其微（+0.000、−0.14、+0.07）。策略并不会因为遵守那句话而获得奖励，所以它不遵守。
仅用 HPS 的路线也存在一个上限。如果每次 rollout 都能达到其最佳水平，那次跑动的均值会停在 0.771。更多步骤是否能突破这个上限，仍是一个悬而未决的问题。
这些画作还展现了表格所遗漏的东西。在每一轮运行内部，所有画作看起来都很相似。随着训练推进，每组内部的奖励值越来越接近，开场视频中处于中位水平的画作看起来就像同一朵花的不同拍摄版本。这正是 GRPO 在基于单一主题构建的样本池上运行时，所发挥的它应有的作用。样本池决定了什么算作多样性，就像它决定了什么算作质量一样。如果奖励只奖励画出一朵特定的花，模型就会学会画那一朵花。更多样的输出需要更多样的样本池，而构建这样的样本池需要更多的策展工作。Surya 较新的构图就是这方面的一个例子。Alex Yango 的动物画作采用了同样的配方，只是在样本池中做了不同的选择。这是审美奖励与数学评分器之间最大的区别。在数字背后，有一项非常人性化的工作，即决定什么该被纳入奖励集。Jason Liu 关于品味（taste）的文章用一句话概括了普遍版本：AI 将瓶颈从“创作”转移到了“发现”。
Surya 在博客结尾分享了他最喜欢的一些画作。与其由我来挑选，不如在下面展示一面墙，上面有奖励机制在两轮评审运行中评分最高的 178 幅画作，数量与参考样本池中的数量相同，排列不分先后。打开它，挑选你自己的最爱吧。
奖励机制在两轮评审运行中选出的 178 幅最爱画作，已随机打乱顺序。现在，来挑选你的最爱吧。
为了做出选择，你浏览了许多画作并保留了几幅，而这正是构建本项目奖励机制所需的工作。每一轮运行的每一幅画作，连同其草图与奖励值，都包含在 rollout 数据集中，并可在本画廊中浏览。
每一轮最后一次运行的中位绘画结果，排列顺序与开场视频一致。相同的基础模型、相同的候选池、三种奖励混合方式、三种风格。
由于奖励部分基于我的个人审美，最后以我作为观看者的评价来收尾是公平的。在我看来，judge-led 是最终结果最多样化、艺术趣味最丰富的一轮。hps-led 画出了令人信服的水彩画，但它最好的作品共享一种柔和、湿画法的质感，几乎自成一种风格。hps-only 收敛得最厉害，它的大多数画作都落在相同的配色上。你可以在画廊中自行判断，那里有每一轮的每一幅画，可按步骤和奖励排序。
基础设施很难
这个项目主要是基础设施工作。一轮运行需要一个训练器、两个 Space、一个推理路由和一个 WebSocket 来连续数小时保持健康运行，而每一个悄悄失败的组件都会在别处变成一个错误的数字。一半的工作是检查你读到的数字是否与实际发生的情况相符。
基础设施的故障会以零值的形式进入奖励。一次渲染超时或评分器无响应，与一幅糟糕的画作得分相同，组内都是 0.0。在我所有的运行中，这大约占 rollouts 的 1.5%，在最差的一轮中达到了 5.2%。这会让模型在噪声上训练，因此这些路径现在返回 None，该 rollout 会被排除在组外。
我还在 OpenEnv 中发现了一个 bug，并把修复提交到了上游。客户端会保持一条持久的 websocket 连接，而远端关闭的 socket 仍会留在缓存中，导致即使环境运行正常，后续每次调用也都会失败。我为此损失了两次跑到一半的实验才找到问题。修复已提交到上游，使用该修复启动的实验运行至今一直保持正常。
这是由裁判主导的那次运行中连续两步的对比，每一步取四张中最好的结果。第 12 步的画作是否看起来差了半分，由你来判断。
每一步的奖励取决于它抽到了哪些参照图。成对裁判每一步会采样四张参照图，因此每一步面对的对手组合都不同，有些抽签本身就更难。GRPO 本身基本是安全的，因为优势值是在组内计算的，抽到难签会让整组一起移动。而我所观察的曲线并不安全，一些看起来像是表现不佳的步骤，其实只是抽到了难签。上图就是一个例子。第 12 步比第 11 步低了半分，主要是因为它在整个运行中抽到了最难的参照图，而画作本身看起来相差无几。
成本是多少
取整后的数字，且仅针对已完成的运行。
| 张 | 需要什么资源 |
|---|---|
| 训练器 | 1 块 H200。60 步约需 18 小时，110 步约需 34 小时 |
| HPSv3 | 一个 a100-large 规模的 Space，在整个运行期间保持在线 |
| 运行环境 | 一个 cpu-upgrade 规模的 Space，渲染时间充裕、流畅完成 |
| 成对比较评判器 | Qwen/Qwen3-VL-30B-A3B-Instruct 的 Inference Providers 配额 |
| 共享资源池，一次性使用 | 来自 iNaturalist 的开放许可照片、四个生成模型的 Inference Providers 配额，以及你自己投入的评分时间 |
一步包含八次 rollout，耗时十五到十八分钟，其中 70% 到 80% 的时间用于渲染。单次渲染需要 69 到 96 秒，而截止时间是 90 秒。部分原因在意料之中：该 Space 没有 GPU，因此 Chromium 以软件方式渲染 WEBGL 画布，而 p5.brush 的晕染和纹理是计算量很大的像素操作。即便如此，我原本预期会更快，而且我还没有找到全部原因。
一个评分模型的成本可能超过使用它的训练本身：HPSv3 必须在整个运行期间保持在线，所以运行结束后要暂停该 Space，或设置其休眠定时器。
四个付费服务必须同时保持健康运行。只有 Hub 和 trackio 能比这次运行活得更久。
一切都在 HF Jobs 上运行，环境采用 Docker Space 形式，指标记录在 trackio 中。
我接下来会尝试什么
这个项目的规则是用全部开放资源复现配方，而不是改进它，所以一路走来积累了一堆未尝试的想法。以下是我真正会去尝试的，按证据充分程度排序。
多步生成，并让模型看到自己画了什么。这是我会首先尝试的。原博客训练的是单轮，所以我也训练了单轮，在这种设置下，模型是闭着眼睛作画的。没有任何图像输入，它得到的唯一反馈就是一个数字。反馈回路有效的证据就是画作池本身。参考画作出自一个在视觉评判器下迭代三轮的模型，而后面的轮次质量更好。定义奖励的材料是用策略模型永远接触不到的回路制作出来的。
更小的模型。有证据表明 35B 超出了实际需要。在我的旁路实验中，一个 4B 模型已经能写出通过门控的有效草图。如果 4B 就能学会这个，实验成本会下降一个数量级。
清单上的其他想法包括：在开始强化学习之前先在画作池来源上做 SFT；显式奖励颜料用量；随着训练推进，把评判器的参考混合从简单逐渐过渡到困难，直到只剩下“爱”；扩大十种方法的允许清单以获得更多视觉范围（我在这方面的尝试导致更多草图崩溃，也破坏了水彩质感）；以及通过给同一张图打两次分来检验成对评判器到底有多一致。
而且这种方法并不局限于花朵。Alex Yango 用同样的机制绘制了动物，Brendan Hogan 则用一批人工评分的视频片段来训练画布动画。我之前也用过 Simon Willison 的 pelican 基准做过类似的尝试——把代码渲染成图像再打分。
而在这一切背后，悬着一个这个项目无法回答的问题：由模型生成的 178 幅画作，定义了这款训练模型认为什么是美。这个评分池是瓶颈所在，也是整条流程中最没有原理性答案的一环。
我相对原作做了哪些改动
对于任何想复现这项工作的人，需要说明的是，我在两个地方有意偏离了 Narreddi 的做法，这两点在前文都已讨论过：
- 成对评判器的参考样本一半取自“喜爱”档、一半取自“还行”档，而不是只与最高档对比，这样早期较弱的策略仍然能获得有效信号。
- 提示词里有一句很小的工艺性描述：每片花瓣画两到三遍——先大笔铺一遍，再在内部叠一层更小、更不透明的笔触。
其余部分——LoRA 全程使用全线性调度、基础设施故障时返回 None 而不是 0.0、评判器运行在第 110 步停止——都是我在过程中不得不做的决定，因为他的博客没有写明这些细节。他的实现没有公开，所以我无法判断这些选择是与他的做法一致还是有所偏离。
所有内容均已公开
| 产物 | 位置 |
|---|---|
| 配方及复现方法 | 02-watercolour/ |
| 参考池，含全部源草图 | watercolour-reference-pool |
| 环境，可直接复制 | watercolour-env |
| HPSv3 评分器，可直接复制 | watercolour-hpsv3 |
| 仅 HPS 的适配器与 rollout | watercolour-grpo-hps-only · watercolour-rollouts-hps-only |
| 裁判主导的适配器与 rollout | watercolour-grpo-judge-led · watercolour-rollouts-judge-led |
| HPS 主导的适配器与 rollout | watercolour-grpo-hps-led · watercolour-rollouts-hps-led |
| 画廊，全部画作可浏览 | 水彩画廊 |
| 训练曲线 | 实时：裁判主导 · HPS 主导 · 仅 HPS，以及 results/ 目录下的 CSV 文件 |
| 全部内容 | 用代码作画 |
本文中每次 rollout 的数据都可以根据已发布的数据集重新计算。本文内容不依赖任何 Space 保持在线运行。
该方法和原始创意出自 Surya Narreddi。该库由 Alejandro Campos Uribe 编写。
On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush, a library that "adds natural drawing tools to p5.js". The video went viral fast, over 1.5M views at the time of writing.
The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine. My attempt is on the engineering side, reproducing the recipe in the open with every piece published.
Note: for the context behind the project, told by Surya himself, watch this video of his thesis.
In this article I try and reproduce his idea with TRL and OpenEnv. The reference pool dataset, the RL environment, the training scripts and the trained models, all open.
The whole pipeline runs on Hugging Face, end to end:
- training on Jobs
- the RL environment and the scorer model as Spaces
- the pairwise judge through Inference Providers
- and every artifact on the Hub, gathered in one collection
Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model, set two environment variables for the reward mix, and launch:
hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
--env-url https://<you>-watercolour-env.hf.space \
--model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
--subject 'a peach hibiscus' --references 4 \
--top-p 0.95 --top-k 20 \
--lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
--scale-rewards none \
--steps 110 --n-episodes 240 --num-generations 8 \
--per-device-batch-size 1 --gradient-accumulation-steps 8 \
--max-completion-length 8192 \
--run-tag my-run --out <you>/watercolour-grpo --push-to-hub
The rest of this article is the story of getting there, and every piece is in the repo.
I have followed the original blog step by step, and only changed something when strictly needed. Every idea of my own went into a list instead of into the experiment, and that list became "What I would try next" at the end, next to the full list of published artifacts. If you have already read his post, the framing and the reward design will be familiar. The new material is the open implementation, the hand-rated pool, and three reward mixes trained and compared, and it starts at The RL environment you need to build.
Three runs, one per reward mix, evolving in parallel. Each frame shows the median painting of a step. No need to tell them apart yet, the article explains which run is which.
Why people loved it
The paintings look loose, imperfect, handmade, at a moment when image models produce perfect (statistically average) pictures. My guess is that this contrast is a big part of why the video went viral. It reminded me of the early days of generative AI art, when the point was to explore the medium. DeepDream (2015) was a debugging tool that people turned into art, works like Edmond de Belamy (2018) came from artists probing what a GAN could do, and artists like Mario Klingemann spent those years making dreamy portraits with neural networks.
This project feels closer to those early days. In his thesis, Surya describes the path that led here. He started by prompting text-to-image models, where the prompt is the only lever you can pull, and more detail buys more control only up to a point. Training the model itself goes further. The other half of the idea is the medium. The model writes a program of about 150 lines of JavaScript that paints the image. That model output is code. You can read it, edit it and run it again, and the decision behind each brushstroke is visible. And the style comes from a restriction where the model is only allowed ten of the library's methods. More on that below.
In that same period, Anna Ridler photographed thousands of tulips, hand-labelled every one, exhibited the dataset itself as the artwork, and later trained a model on it. I found her work through the references AI agents brought back while building this project and loved it because this project does something very similar by curating a set of images by hand, and then training against them.
RL over taste
Most of the recent RL work on language models uses rewards you can verify. For example, math problems with a known answer, code that passes tests, or graders that are right or wrong and cheap to run. This project is closer to the older exception, RLHF, where the model learns a reward model from human preferences.
Here the reward is aesthetic preference. There is no correct answer. The real question of the project is whether you can do RL over taste.
The reward, as his blog defines it and as the RL environment I built implements it:
| term | weight | what it measures |
|---|---|---|
gate | 0.05 | the sketch compiles, paints something, does not cheat |
length | 0.05 | a soft push towards longer code snippets |
| pairwise judge | 0.60 | style, compared against references drawn from a pool |
| HPSv3 | 0.30 | aesthetic preference on the render |
HPSv3 is an open 7B preference model. Give it an image and a text description, and it returns a score for how much a person would prefer that image. It was trained on a large set of human choices between pairs of images, so its score is an average of many people's taste. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. The pairwise judge sees the candidate painting next to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges), each comparison in both presentation orders, and its score is the share of comparisons the candidate wins. Its only standard is the pool, so its score is my taste, as encoded in those ratings.
Two of the four terms in the reward function are models. Both are proxies for someone's taste.
Those are the weights Narreddi converged on. The pool defines taste here. That moves the work from tuning hyperparameters to building the set that decides what is beautiful.
I trained three runs with this reward. They differ only in how the weight splits between the two model judges:
| run | pairwise judge | HPSv3 | role |
|---|---|---|---|
judge-led | 0.60 | 0.30 | the original mix, stopped at step 110 |
hps-led | 0.30 | 0.60 | the middle point, stopped at step 110 |
hps-only | 0.00 | 0.90 | the validation run, stopped at step 60 |
I started with hps-only to validate that the pipeline could learn at all. Once the reward was going up and the metrics were healthy, there was no reason to run it longer, so I launched the two longer runs instead. The question that the longer runs ask is how much of HPSv3's power can you hand to the pairwise judge? The more weight the judge carries, the more the reward means my taste instead of everyone's, and the harder it should be to climb. Incidentally, if you push it far enough or your style is too far from the average, the model could stop entirely.
Fortunately, it did not stop and both runs with the pairwise judge on learned too. The hand-rated pool can steer the policy, at least as far as the metrics and the final paintings show. The numbers are below.
Disclaimer. If we use a frontier model, it can already generate the JavaScript code that paints watercolour from a prompt. That's the starting point. The work here is about teaching a smaller model to do it combined with a person's own artistic preferences.
The RL environment you need to build
The environment wraps everything that sits between the model and the reward, including the JavaScript library that the model uses for painting, the system prompt that restricts it, the headless Chromium that renders each sketch, and the gate that rejects cheats.
The library does more of the work than it seems. p5.brush, by @acamposuribe, simulates a medium rather than drawing shapes: pigment bleeds past the edges of a fill, paper has texture, strokes have mass, flow fields drag brushwork around. When the model calls brush.fillBleed(0.25) it is deciding how far the ink runs.
Note. The author of p5.brush had been trying to teach a machine to paint long before any of this. In 2022 he made a generative art series that hides a diary about teaching p5.js to draw like a child: "It is barely able to use the crayons [...] It cannot follow simple commands. I'm done for today, very infuriating." The series was meant to have three pieces, and he made two. When Surya's video went viral, he quoted it, sharing that diary and saying this work is the third piece arriving on its own.
p5.brush exposes 47 methods. The prompt allows 10: scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape and circle. What the other thirty-seven add, lines, hatching, custom brushes, would break the watercolour look. With these ten the model can only paint filled shapes, and the library adds the bleed to every one of them.
Part of one rollout's draw() and what it renders to. The comments are the model's own. Reward 0.864, 129 lines, step 22. The full source of every painting is in the rollouts dataset.
His blog post saved me a lot of time I could have wasted iterating on the prompt. A long API reference makes the model invent methods that do not exist, and his 200 GEPA iterations converged on a strict allowlist with no documentation. I saw the same failures and wrote the allowlist by hand. My only addition to that recipe is one sentence: paint each petal two or three times, a big pass first and a smaller, more opaque one inside it. This small change made my outputs a lot more colorful.
Note. If it's the first time you hear about GEPA, it is an automatic prompt optimizer. A language model reflects in plain words on where the current prompt failed and proposes a better one, and the loop repeats.
The gate is the final piece. The sketch has to compile, use the library instead of direct p5 calls, put real pigment on the canvas, and not try to trick the scorer, for example by writing text on the canvas.
The pool is the reward function
The pool consists of 178 paintings divided into two tiers based on my personal preferences, love and okay. All of them are actually generated by a model. Four open-weight models, called through Inference Providers, wrote p5.brush sketches, each one working from a real, openly licensed photo of a hibiscus from iNaturalist. A vision model gave written feedback on every sketch, over three refinement iterations. Every final render was then rated one at a time, by me, and 178 made the cut.
| generator | number of paintings |
|---|---|
| GLM-5.2 | 64 |
| Kimi-K3 | 57 |
| Qwen3-Coder-Next | 35 |
| Qwen3.5-122B-A10B | 22 |
Here I chose four different families of models to test their different styles. These four were the open models that produced a valid sketch every time in a quick reliability check, and two other candidates were dropped for failing it. If you want to produce your own pool, you might choose others.
The two tiers, as they actually look. Disagreeing with the rating is reasonable, somebody's judgement is now the reward function.
The tiers do real work in the reward. When the pairwise judge draws four references, half come from love and half from okay, so the policy always faces some rivals it can sometimes beat, and a win pays the same against either tier. This is one of my few deliberate changes: the original compares against its top tier only, and I kept the easier tier in the draw so a weak early policy still gets signal.
No human-made painting is in there, which is a real limitation. p5.brush is a niche library, and the human work that exists in it with accessible code is a handful of pieces, nowhere near what a training corpus would need, as his blog also notes.
The interesting idea here, as I've already discussed previously, is that the model will learn to imitate what the pool contains. If we point the environment at a different dataset, the reward would automatically change without touching a single line of code. For the dataset I generated and openly share, I also include the source sketch.
If we look closer at the judges, the two of them answer different questions. HPSv3 decides whether it is a flower, and the pairwise judge decides whether it is well painted in the style I chose.
Just one more yolo run
Before anything worked, there was a long stretch of flat reward curves. If you've tried to reproduce a research paper/blog without open artifacts, you probably can relate. Every run tested what I thought was a reasonable theory about what was wrong. A run takes a lot of time, so I queued the next one while still going through the last one's results. As always, starting from something easier that works and then building on top of that was the answer. A simple control task, with no browser and no judges, was the first attempt that learned, and the reason was that my learning rate was just too low.
Three experiments on the reward, three flat lines. I swapped the pool, removed renderer noise and turned the pairwise judge off, and none of it moved the curve. The run that moves changed the trainer configuration, so the difference is the trainer, not the reward mix.
Another change that cost me time to find was adjusting the LoRA parameters correctly. The usual target_modules list assumes a dense model, and Qwen/Qwen3.5-35B-A3B is a mixture of experts that names most of its projections differently, so the adapter was training ten layers out of forty. I solved this by changing it to all-linear, which reaches every linear layer. The routed experts in this architecture are fused tensors that even all-linear leaves frozen, but everything else gets an adapter, and that was enough to learn.
The fix was four changes in TRL's GRPOTrainer:
| setting | from | to | why |
|---|---|---|---|
| learning rate | 2e-5 | 5e-5 | the ceiling LoRA Without Regret uses for GRPO |
| scheduler | linear | constant_with_warmup | linear decay had spent most of the learning rate by mid-run, so the reward never took off |
scale_rewards | group | none | one gate rejection was shrinking every other advantage in the group |
target_modules | hand list | all-linear | reach every linear layer |
These four changes unlocked the first successful run (hps-only), with the reward clearly improving.
With that configuration, all three runs learn. Both judge runs were launched for 200 steps and stopped at 110, with the reward still climbing slowly. A step takes fifteen to eighteen minutes, and the comparison between mixes was already stable, so I stopped both to save compute. Mean group reward over the first and final third of each run:
| run | steps | first third | final third | Δ |
|---|---|---|---|---|
hps-only | 60 | 0.58 | 0.71 | +0.13 |
judge-led | 110 | 0.45 | 0.72 | +0.27 |
hps-led | 110 | 0.57 | 0.82 | +0.24 |
The three curves line up with how much weight my taste carries. The more the judge weighs, the lower the start and the noisier the climb. judge-led spent its first thirty steps nearly flat before it moved. It is the same move that solved the debugging. Shrink the problem until something learns, then add the hard parts back one at a time.
The pairwise judge term itself climbed in the two runs that used it. The model wins more comparisons against the pool as training advances, which is the claim hps-only could not make. No group in any run collapsed to identical rewards, the GRPO failure mode that kills the gradient. For the curious, the per-metric curves (HPSv3, paint coverage, entropy) are in the repository as CSV.
The full launch command, the hardware and the two environment variables that turn this into the other two runs are in the recipe.
What it actually learned
In every run, the first thing the model learned was to stop producing bad paintings, the near-blank canvases and shapeless washes that score under 0.3 in total reward. In hps-only, three quarters of the rise in the group mean comes from bad paintings becoming rare. In the judge runs the collapse is even steeper: rollouts under 0.3 fall from 99 to 16 across judge-led's thirds, and from 37 to 4 in hps-led.
This is why the obvious visual, the best painting of each step, shows almost no difference in hps-only. It moves +0.034 across the run while the median moves +0.155. The learning is visible in the middle of the distribution.
What the pairwise judge changes is the top. In hps-only, paintings got more reliable without getting better. The quality of the good ones added just +0.03 to the group mean, and once HPSv3 saw petals around a centre and a stem, it stopped asking for more pigment. With the judge on, the other half of the story appears. Better here means closer to the pool, so closer to what I rated as "good" or something I liked more. That added +0.12 in judge-led and +0.16 in hps-led, the best of each step rose too, and paint coverage doubled in both runs (0.11 to 0.23, and 0.13 to 0.30) where hps-only barely moved it. With a reference left to beat, a good painting can still get better, and the model starts being rewarded for using more pigment.
One more finding. The model ignores an explicit instruction, and it is right to. The system prompt asks for fifteen to thirty filled shapes. If we look at the real mean, it is between 7 and 9, and n_shapes barely correlates with reward in any run (+0.000, −0.14, +0.07). The policy is not rewarded for obeying that sentence, so it does not obey it.
There is also a ceiling on the hps-only route. If every rollout matched its good ones, that run's mean would sit at 0.771. Whether more steps would break it is an open question.
The paintings also show something that the tables miss. Within each run, they all look similar. As training advances, the rewards inside each group get closer together, and the median paintings in the opening video look like takes of the same flower. That is GRPO doing what it is designed to do with a pool built from one subject. The pool decides what counts as variety, the same way it decides what counts as quality. If the reward only pays for matching one flower, the model learns to paint that one flower. More diverse output would need a more diverse pool, and building one is more curation work. Surya's newer compositions are an example of this. Alex Yango's animal paintings are the same recipe with different choices in the pool. This is the biggest difference between an aesthetic reward and a maths grader. Behind the number there is a very human job, deciding what belongs in the reward set. Jason Liu's essay on taste says the general version in one line. AI shifted the bottleneck from making to noticing.
Surya closes his blog with some of his favourites. Instead of picking mine, below is a wall with the 178 paintings the reward scored highest across the two judge runs, the same number the reference pool holds, in no particular order. Open it and pick your own.
The reward's 178 favourites from the two judge runs, shuffled. Now pick yours.
To choose, you looked at many and kept a few, and that is exactly the job that built the reward of this project. Every painting of every run, with its sketch and its reward, is in the rollouts datasets, and browsable in this gallery.
The last step's median painting of each run, in the same order as the opening video. Same base model, same pool, three reward mixes, three styles.
Since the reward was partly based on my taste, it is fair to close with my verdict as a viewer. To my eye, judge-led is the run that ends up the most diverse and the most artistically interesting. hps-led paints convincing watercolours, but its best ones share a soft, wet-on-wet look that is almost a style of its own. hps-only converges the hardest, and most of its paintings settle on the same colours. You can judge for yourself in the gallery, which has every painting of every run, sortable by step and by reward.
Infra is hard
This project is mostly infra. A run needs a trainer, two Spaces, an inference router and a websocket to stay healthy for hours straight, and every piece that fails quietly turns into a wrong number somewhere else. Half the work is checking that the number you read matches what actually happened.
Failures of the infrastructure were entering the reward as zeros. A render that timed out or a scorer that did not answer scored the same as a bad painting, 0.0 inside the group. Across all my runs that was about 1.5% of rollouts, and in the worst run it reached 5.2%. That trains the model on noise, so those paths now return None and the rollout is excluded from the group.
I also found a bug in OpenEnv, and sent the fix upstream. The client keeps one persistent websocket, and a socket closed by the far end stayed cached, so every later call failed even though the environment was healthy. It cost me two half-finished runs to find it. The fix is submitted upstream, and the runs launched with it have been running clean since.
Two consecutive steps of the judge-led run, best four of each. Whether step 12's paintings look half a point worse is for you to decide.
The reward of a step depends on which references it drew. The pairwise judge samples four references per step, so every step faces a different set of rivals, and some draws are simply harder. GRPO itself is mostly safe, because advantages are computed inside the group and a hard draw moves the whole group together. The curve I was reading was not safe, and some of what looked like a bad step was just a hard draw. The image above is one example. Step 12 scored half a point below step 11 mostly because it drew the hardest references of the run, while the paintings themselves look close.
What it costs
Rounded numbers, and only for the runs that finished.
| piece | what it needs |
|---|---|
| trainer | 1 H200. 18 hours for 60 steps, about 34 for 110 |
| HPSv3 | an a100-large Space, up for the whole run |
| the environment | a cpu-upgrade Space, which renders comfortably in time |
| the pairwise judge | Inference Providers quota for Qwen/Qwen3-VL-30B-A3B-Instruct |
| the pool, one-off | openly licensed photos from iNaturalist, Inference Providers quota for the four generators, and rating is your own hours |
A step is eight rollouts and takes fifteen to eighteen minutes, of which 70 to 80% is rendering. A single render takes 69 to 96 seconds against a 90 second deadline. Part of that is expected, the Space has no GPU, so Chromium renders the WEBGL canvas in software and p5.brush's bleeds and textures are heavy pixel work. Even so, I expected it to be faster, and I have not found the full cause.
A scorer can cost more than the training that uses it: HPSv3 has to be up for the whole run, so pause the Space, or set its sleep timer, when the run ends.
Four paid services have to stay healthy at once. Only the Hub and trackio outlive the run.
Everything runs on HF Jobs, with the environment as a Docker Space and metrics in trackio.
What I would try next
The rule of this project was to reproduce the recipe with every resource open, not to improve it, so a list of untried ideas piled up along the way. These are the ones I would actually try, in order of how much evidence there is.
Multi-step, and letting the model see what it paints. This is the first thing I would try. The original blog trains single turn, so I trained single turn, and in this setup the model paints with its eyes closed. No image ever goes in, and the only feedback it gets is one number. The evidence that a feedback loop works is the pool itself. The reference paintings came from models iterating three rounds under a vision critic, and the later rounds are better. The material that defines the reward was made with a loop the policy never gets.
Smaller models. There is evidence that 35B is more than needed. In my side experiments a 4B already wrote valid sketches that passed the gate. If a 4B can learn this, the cost of the experiment drops by an order of magnitude.
Other ideas on the list are SFT on the pool sources before starting RL, rewarding pigment explicitly, moving the judge's reference mix from easy to hard as the run advances until only love remains, widening the ten-method allowlist for more visual range (my attempts on this crashed more sketches and broke the watercolour look), and checking how consistent the pairwise judge really is by scoring the same image twice.
And the method is not specific to flowers. Alex Yango painted animals with the same mechanism, and Brendan Hogan trained canvas animations against a pool of hand-rated clips. I had also played with something similar before using Simon Willison's pelican benchmark, where code is rendered to an image and scored.
And underneath all of it sits the question this project cannot close. 178 paintings made by models define what this trained model considers beautiful. The pool is the bottleneck, and it is the part of the pipeline with no principled answer.
What I changed from the original
For anyone reproducing this, there are two deliberate divergences from Narreddi's recipe, both argued earlier in the article:
- The pairwise judge draws its references half from
loveand half fromokay, instead of comparing against the top tier only, so a weak early policy still gets signal. - One small sentence of craft in the prompt, paint each petal two or three times, a big pass first and a smaller, more opaque one inside it.
The rest, all-linear for the LoRA, infrastructure failures returning None instead of 0.0, stopping the judge runs at step 110, are decisions I had to make along the way because his blog does not specify them. His implementation is not published, so I cannot tell whether they match his choices or diverge from them.
Everything is published
| artifact | where |
|---|---|
| the recipe, and how to reproduce it | 02-watercolour/ |
| the reference pool, with every source sketch | watercolour-reference-pool |
| the environment, ready to duplicate | watercolour-env |
| the HPSv3 scorer, ready to duplicate | watercolour-hpsv3 |
| the hps-only adapter and rollouts | watercolour-grpo-hps-only · watercolour-rollouts-hps-only |
| the judge-led adapter and rollouts | watercolour-grpo-judge-led · watercolour-rollouts-judge-led |
| the hps-led adapter and rollouts | watercolour-grpo-hps-led · watercolour-rollouts-hps-led |
| the gallery, every painting browsable | watercolour-gallery |
| the training curves | live: judge-led · hps-led · hps-only, and the CSV files in results/ |
| all of it | Paint with Code |
The per-rollout numbers in this article can be recomputed from the published datasets. Nothing here depends on any Space staying switched on.
The method and original idea are Surya Narreddi's. The library is Alejandro Campos Uribe's.