长话短说，因为您还有模型要训练，我们理解这一点：
- 异步强化学习有一个不为人知的秘密：每一步，训练器都必须将整个模型传输到推理引擎。对于一个 7B 参数、bf16 精度的模型，这相当于 14 GB。对于一个前沿的 1T 参数模型检查点，每次传输的数据量大约在 TB 级别。每一步都是如此。
- 事实证明，你并不需要这样做。在两个连续的强化学习优化器步骤之间，大约 99% 的 bf16 权重是比特完全相同的（在最坏情况下也从未低于 98%）。实际的差异非常微小。
- 我们提交了一个 TRL 拉取请求，它仅将发生变化的元素编码为一个稀疏的 safetensors 文件，上传到 Hugging Face 存储桶，并通知 vLLM 去获取它。在 Qwen3-0.6B 模型上，每一步的传输负载从 1.2 GB 降到了 20 到 35 MB。
- 锦上添花的是：我们运行了一个完全分离的训练流程，其中训练器在一台机器上，vLLM 运行在一个 Hugging Face Space 中，Wordle 环境运行在另一个 Space 中，而权重则通过一个单一的 Hub 存储桶流动。无需共享集群，无需 RDMA，无需 VPN。
异步强化学习变得便宜多了。请继续阅读。
传输相同权重的两种方式。红色部分表示没有生成任何 token 的挂钟时间。
1. 一 TB 的问题
如果你读过我们之前关于异步强化学习训练格局的文章，你已经知道关键点了。每一个异步强化学习库，无论它如何拼写“actor 模型”，或者它的 NCCL 后端是什么颜色，最终都会遇到同一个根源问题：权重同步。
推理引擎使用的是第 N 步的策略。训练器刚刚完成了第 N+1 步。新的权重必须在推理引擎开始危险地偏离策略之前，从一端传输到另一端。无论你运行的是同步还是异步模式，这都处于关键路径上：一次阻塞式传输意味着 GPU 在浪费空闲算力，没有生成任何 token。通过稀疏增量路径，你可以将这段空闲时间压缩到几秒钟，而且训练器甚至不需要等待推理引擎准备好：它只需在优化器步骤完成的那一刻发布“权重已就绪”并上传权重到共享存储桶，而推理引擎则可以在自己的时间线内获取。
Fireworks 在其文章《前沿强化学习比你想象的更便宜》中给出了一个令人印象深刻的数字：对于一个 fp8 格式的前沿 1T 参数检查点，完整快照大小为 1024 GiB，而传统观念认为，每次更新你的推理集群时都必须传输这个完整快照。正是这样的数字，让人们开始绘制包含超级集群、RDMA 网络和专用跨区域链路的架构图。他们测量到的相邻检查点之间的平均差值为 20.3 GiB，仅占完整模型的 1.98%，并且“超过 98% 的 bf16 格式权重在连续检查点之间保持比特级一致”。
Cursor 的 Composer 2 报告讲述了一个类似的故事。他们在不同区域运行训练和推理，并通过一个共享的 S3 存储桶（他们的原话）将它们连接起来，训练器会在每个训练步骤将压缩后的权重差异上传到该存储桶。每个集群独立地从共享的差异链中下载并重建模型，“无需与训练集群建立直接连接”。双方从不直接通信参数信息。这个存储桶就是连接它们的“线”。
两篇论文在三点上达成共识，我们想慢慢重复这几点，因为本文的其余部分本质上是对这些共识的忠实开源翻译：
- 在相邻的两个强化学习步骤之间，大部分权重实际上并未发生变化。
- 如果你只发送发生变化的部分，你的带宽成本将大约降低两个数量级。
- 如果你通过一个共享对象存储来路由这些微小的差异数据，那么训练器和推理集群就不再需要位于同一个数据中心。
唯一缺少的是一个可以通过 `pip install` 安装的版本。于是我们编写了一个。
2. 为什么 bf16 强化学习权重几乎总是稀疏的
在我们连接任何东西之前，有必要理解为什么整个策略是可行的。“98% 的权重不变”这个说法听起来可疑，像是那种在演示中有效、但在实际应用中就会失效的数字。但事实并非如此。这是由 bf16 算术在强化学习所使用的学习率下的工作方式所决定的。
一个 bf16 数有 7 位尾数。在两个相邻的 2 的幂之间，恰好有 $2^{7} = 128$ 个可表示的值，因此在 $\mid w \mid$ 附近相邻 bf16 数之间的间隔大约为 $\mid w \mid \cdot 2^{- 7}$。当一次更新量低于该间隔的一半时，即当 $\mid \Delta w \mid < \mid w \mid / 256$ 时，该更新就会被 bf16 的数值转换所吸收。这就是 PULSE 在其图 3 中绘制的“bf16 可见性阈值”。
现在来看看 Adam 的做法。在强化学习的学习率设为，比如 $3 \times 10^{- 6}$ 时，对单个权重的更新为：$\Delta w = - \eta \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}$
归一化后的步长 $\hat{m} / \left(\right. \sqrt{\hat{v}} + \epsilon \left.\right)$ 大致为 1 的量级，因此 $\mid \Delta w \mid \approx \eta \approx 3 \times 10^{- 6}$。对于大多数权重来说，$\mid w \mid$ 大约在 $10^{- 2}$ 到 $10^{- 1}$ 之间（PULSE 报告称，代表性大语言模型权重的中间值为 0.019）。在该量级下，阈值 $\mid w \mid / 256$ 大约在 $4 \times 10^{- 5}$ 到 $4 \times 10^{- 4}$ 之间，这比更新量要大。
换句话说：优化器在低声细语，而 bf16 听不到。更新被舍入操作所吸收，$w$ 的字节表示没有变化，从推理引擎的角度来看，这个权重根本没有移动。将这种情况乘以数亿个参数，你就得到了大于 99% 的稀疏度数值，无需任何近似，完全免费。
这正是 PULSE 论文（Mihai & Belilovsky, 2026）中正式提出的论点。他们定义了两个阈值。吸收边界 $10 \eta$ 是 Adam 更新的保守最坏情况，而有效边界 $\eta$ 则是实际所处的区间。bf16 可见性阈值是 $\mid w \mid / 256$。每当更新低于可见性阈值时，它就会被吸收，bf16 字节不会发生变化。他们的图 3 将这两个边界与代表性的大语言模型权重云图进行了对比，结论是明确的：在 $\eta = 3 \times 10^{-6}$ 时，吸收边界本身已经低于模型中几乎所有权重的可见性阈值。他们通过 Qwen2.5（0.5B/1.5B/7B）、Llama-3.2-3B 和 Gemma-3-4B 进行了实证测量，一致发现平均每步稀疏度约为 99%，在 400 个训练步上的标准差为 0.2% 到 0.4%。最坏情况下的步数也保持在 98% 以上。因此，<1% 的变化并非幸运的测量结果，而是算术运算所保证的。
我们不需要通过分析来预测这一点（实际上，我们尝试过从 Adam 的 $m$ 和 $v$ 统计量预测变化掩码，但召回率低至可怜的 30%，稍后会详细说明）。我们只需要观察哪些字节发生了翻转。对于每个参数来说，这是一个微小的布尔张量，在优化器步骤附近计算得出。
将学习率拖到强化学习领域，观察回退到 bf16 的标记如何跳回原始刻度。左下角的 256 元素网格显示了在一个小型模型上的总体效果。
3. HF Buckets 与架构
这就是故事的第二部分登场的地方，也是本文从 Fireworks/Cursor 的翻译转变为 Hugging Face 相关内容的地方。
3.1 什么是 Bucket？
Bucket 是 Hub 上的一种仓库类型，专为高频对象存储而设计。没有提交仪式，没有 PR 工作流，也没有 LFS 的古怪问题。你可以添加文件、列出文件、下载文件。其 Python 接口包含两个函数：
from huggingface_hub import batch_bucket_files, download_bucket_files
# Trainer side
batch_bucket_files("my-org/wordle-deltas", add=[(buffer, "deltas/step_000042.safetensors")])
# Inference side
download_bucket_files("my-org/wordle-deltas", files=[("deltas/step_000042.safetensors", local_path)])
仅此而已。两个函数调用，你的权重就开始传输了。
在底层，存储桶由 Xet 支撑，这是 Hub 基于内容分块的存储层。Xet 会检查你上传的每个文件，根据其实际内容（而非固定偏移量）将其切分成多个块，并与存储桶中已有的所有内容进行去重。实际效果——在此场景下非常令人欣喜——是即使我们懒得编写稀疏编码，每一步都直接上传完整的锚点，Xet 也只会传输发生变化的块。稀疏编码 + Xet 技术栈：我们只为发生变化的内容付费，而且只付一次。
这相当于 Fireworks 和 Cursor 所使用的“共享 S3 存储桶”的开源版本，区别在于该存储层已经了解内容哈希，你现有的 HF token 已具备权限，并且它能与栈中的其他部分（Spaces、数据集、模型）原生组合。
3.2 三个盒子
完整架构恰好包含三个盒子和一个共享基础层：
- 训练器。放在你想要的任何地方。一块 GPU、八块 GPU、一台通过 USB 连接 H100 的笔记本电脑，我们都不会评判。它拥有模型权重，运行优化器，并输出稀疏增量。
- HF 存储桶。一个单一的仓库，包含两个前缀：`anchors/` 用于偶尔的完整快照，`deltas/` 用于期间的稀疏补丁。这是双方唯一达成一致的部分。
- vLLM 部署服务器。放在你想要的任何地方，关键是未必与训练器在同一位置。它从存储桶拉取数据，应用增量，并提供部署服务。
- 环境。以常规方式（HTTP、函数调用，或你的环境支持的任何方式）挂载在部署服务器上。
需要内化的特性——也是 Cursor 那篇论文极力推崇、在此处完全适用的——是训练器和部署服务器之间从不直接通信权重。它们交换一个微小的 POST 请求，包含 `{"repo_id": ..., "filename": ...}`，这就是整个控制平面。实际的字节传输发生在每一方与存储桶之间，并行进行，无需共享网络结构。
为什么这在实践中很重要：
- 部署服务器可以位于另一个区域、另一个云，或者 Hugging Face Space 内部的 NAT 之后。它完全不在意。
- N 个推理副本可以从同一个存储桶拉取同一个增量，而 Xet 会在所有副本之间对字节进行去重。
- 训练器永远无需知道存在多少个推理副本，它们位于何处，或者其中某个副本是否刚刚崩溃。
训练器负责写入。副本负责读取。Hub 负责管道连接。
4. 协议
现在我们可以打开引擎盖了。该协议包含四个部分：一种线格式、一种存储桶布局、一个 30 行的 vLLM 扩展，以及一个训练器端的变更检测器。老实说，它的代码量比听起来要少。
4.1 Safetensors 作为线格式
我们选择了 safetensors 作为磁盘和网络传输格式。它已经是 Hub 上标准的检查点格式，所有主流的框架都能读取它，并且其头部可以携带任意的字符串元数据。我们正是利用这个元数据字段来隐藏协议。
存储桶中有两种类型的文件。
锚点文件看起来像一个正常的检查点：每个参数对应一个张量，完整的 bf16 权重，每 $N$N 次同步写入一次（我们默认 $N = 10$N=10）。
anchors/step_000010.safetensors
├── model.layers.0.self_attn.q_proj.weight (bf16, full)
├── model.layers.0.self_attn.k_proj.weight (bf16, full)
└── ...
metadata:
sparse=False, model_version=10, sparsity=0.0
增量文件是其中的关键。对于每个实际发生变化的参数，我们存储两个条目：一个平坦的 int32 张量，包含元素索引；以及一个 bf16 张量，包含这些索引位置的值。
deltas/step_000011.safetensors
├── model.layers.0.self_attn.q_proj.weight.indices (int32, [num_changed])
├── model.layers.0.self_attn.q_proj.weight.values (bf16, [num_changed])
├── model.layers.0.mlp.gate_proj.weight.indices
├── model.layers.0.mlp.gate_proj.weight.values
└── ...
metadata:
sparse=True, model_version=11, sparsity=0.9938, changed_params=[...]
这种选择带来的一些好处：
- 增量文件就是一个文件。你可以在 Python 中用 `safe_open(...)` 打开它，并检查其中的每一个张量。没有专有的帧格式，没有长度前缀，没有版本握手。
- 元数据是自描述的。接收方读取 `sparse=True/False` 并据此分支处理。不需要单独的清单文件。
- 在推理端，它通过 mmap 实现零拷贝，当你每隔几秒就要执行一次此操作时，这一点至关重要。
节奏很直接：每第 N 步写入一个锚点文件，中间步骤写入增量文件。两者都存放在同一个存储桶中，分别位于 `anchors/` 和 `deltas/` 前缀下。每个新的推理副本只需获取最新的锚点文件，然后重放此后的增量文件即可。
十个训练步骤。在第 1 步和第 6 步写入锚点文件（完整快照），在其他每一步写入稀疏增量文件。你可以实时看到文件落入存储桶中。
4.2 训练器端：来自优化器钩子的布尔掩码
训练器需要知道哪些 bf16 元素实际发生了翻转。我们通过一个微型的 `BF16ChangeDetector` 来实现这一点，它在优化器上注册了一个步骤前和步骤后的钩子：
class BF16ChangeDetector:
def __init__(self, model, optimizer):
self._pre_step_bf16: dict[str, torch.Tensor] = {}
self._validated_masks: dict[str, torch.Tensor] = {}
optimizer.register_step_pre_hook(self._pre_step_hook)
optimizer.register_step_post_hook(self._post_step_hook)
def _pre_step_hook(self, opt, args, kwargs):
for p in self._params:
self._pre_step_bf16[name_of(p)] = p.detach().to(torch.bfloat16).cpu().clone()
def _post_step_hook(self, opt, args, kwargs):
for p in self._params:
self._validated_masks[name_of(p)] = (
p.detach().to(torch.bfloat16).cpu() != self._pre_step_bf16[name_of(p)]
)
PR 中的实际代码有更多底层细节（通过 `data_ptr()` 将优化器参数对象与模型参数进行匹配，因为 Accelerate 将它们包装成了不同的 Python 对象），但核心思路简单明了：快照、更新、求差。
这是最可靠的方法。我们尝试过更优雅的路径——利用 Adam 的 $m$ 和 $v$ 统计量来预测掩码，并直接使用 bf16 ULP 阈值。这种方法在原理上可行，但在实际中召回率仅为 30% 左右，这意味着我们最终交付的增量更新会遗漏三分之二的实际更新。Adam 的归一化过程相当复杂，导致分析阈值并不精确。因此我们直接比较字节。代价是在训练端对模型进行一次 bf16 CPU 快照，这个成本我们愿意承担。
新的 `_sync_weight` 流程包含四个阶段：
- 推理持续运行的同时进行上传。训练器将掩码后的元素编码到 safetensors 缓冲区中，并推送到存储桶。在整个步骤中，vLLM 仍在愉快地使用旧策略提供服务。
- 暂停 vLLM。一次短暂的 HTTP 调用，耗时数百毫秒。
- 发送 `/update_weights` 信号。传递存储桶坐标。vLLM 下载、应用更新并返回响应。
- 恢复运行。vLLM 重新上线。
日志行清晰地说明了情况：
Delta: 1234567/200000000 elements changed (sparsity=99.38%)
[delta_engine] uploaded user/wordle-deltas/deltas/step_000042.safetensors (27.4 MB, ...)
Weight sync: done. Total 9.4s (inference paused 1.1s)
关键信息在括号中。推理暂停了 1.1 秒。剩余的 9.4 秒用于上传，而这段时间内 rollout 服务器仍在生成 token。使用 NCCL 时，整个同步时间都算作暂停时间。在这里，我们将其转化为后台时间。
一次端到端的同步。在 delta-over-bucket 和 NCCL 广播之间切换，并尝试切换副本数量以观察扇出效果。
4.3 vLLM 侧：一个 30 行的扩展
vLLM 为此提供了一个清晰的抽象，名为 WeightTransferEngine。我们实现了一个 DeltaWeightTransferEngine，其 `receive_weights` 方法的核心逻辑如下：
def receive_weights(self, update_info, load_weights):
download_bucket_files(update_info.repo_id, files=[(update_info.filename, local_path)])
with safe_open(local_path, framework="pt", device="cpu") as f:
meta = PatchMetadata.from_metadata_dict(f.metadata())
if not meta.sparse:
# Anchor: feed every tensor and snapshot for future deltas
for name in f.keys():
tensor = f.get_tensor(name)
self._bf16_snapshot[name] = tensor.clone()
load_weights([(name, tensor)])
else:
# Delta: apply (indices, values) to snapshot, hand full tensor to vLLM
for name in json.loads(meta.changed_params):
indices = f.get_tensor(f"{name}.indices").long()
values = f.get_tensor(f"{name}.values")
snap = self._bf16_snapshot[name].flatten()
snap[indices] = values
self._bf16_snapshot[name] = snap.reshape(self._bf16_snapshot[name].shape)
load_weights([(name, self._bf16_snapshot[name])])
我们通过 vLLM 的 `--worker-extension-cls` 标志注册它，这意味着无需 fork vLLM。你只需将 TRL 安装到与 vLLM 相同的镜像中，将 CLI 指向我们的类即可完成。
值得一提的是：vLLM 自身正在推进一项将稀疏权重传输原生落地的开发工作，即 vllm-project/vllm#40096。该方案在 WeightTransferEngine 基类上直接添加了 receive_sparse_weights() 和 trainer_send_sparse_weights() 方法，补丁以（索引，数值）形式编码，并通过 index_copy_() 原地应用，完全消除了 GPU/CPU 之间的验证往返。该 PR 报告显示，在 Qwen3-1.7B 上传输一个稀疏补丁仅需 0.40 毫秒传输 0.16 MB，而完整稠密传输则需要 192 毫秒传输 942 MB。
我们在推理侧实现中有一个诚实的注意事项：我们保留了一份模型的 CPU bf16 快照，以便能够从稀疏（索引，数值）补丁中重建完整张量，因为当前 vLLM 中的 load_weights 期望接收完整张量。一旦 #40096（或其后续版本）落地并暴露出一个原地稀疏 load_weights 路径，我们就可以直接在 GPU 上应用索引，并丢弃这份快照！
5. 在 Spaces 上真正部署运行
这是让我们颇为得意的一部分。到目前为止我们描述的所有内容都可以在笔记本电脑上运行，但通过 Hub 存储桶路由权重的意义在于，训练器和部署服务器不必位于彼此附近。因此，我们使用三台机器运行了一次完全分离的训练，这三台机器之间不共享网络：
- 一台配备单 GPU 的机器运行训练器。
- 一个 Hugging Face Space（Docker SDK，L4 GPU）运行带有我们扩展类的 vLLM。
- 第二个 Hugging Face Space（CPU）运行 Wordle 环境服务器，支持 256 个并发会话容量。
- 中间的一个 Hub 存储桶。
设置这一切实际上只需要几个 hf CLI 命令。vLLM Space 的 Dockerfile 本质上就是上游 vLLM 镜像加上 pip install trl@... 再加上入口点：
FROM vllm/vllm-openai:latest
RUN pip install "trl @ git+https://github.com/huggingface/trl.git@delta-weight-sync"
ENV VLLM_SERVER_DEV_MODE=1
EXPOSE 7860
ENTRYPOINT ["vllm", "serve", "Qwen/Qwen3-1.7B", \
"--host", "0.0.0.0", "--port", "7860", \
"--worker-extension-cls", "trl.experimental.async_grpo.delta_engine.DeltaWorkerExtension", \
"--weight-transfer-config", "{\"backend\":\"nccl\"}", \
"--max-model-len", "32768", \
"--gpu-memory-utilization", "0.8"]
将其部署为一个 Space：
hf repos create $USER/vllm-wordle-inference \
--type space --space-sdk docker --flavor l4x1 \
--secrets HF_TOKEN=$HF_TOKEN
hf upload $USER/vllm-wordle-inference examples/scripts/openenv/vllm_space/ --type space
然后从地球上任何能够进行 HTTPS 通信的地方启动训练：
python examples/scripts/openenv/async_wordle.py \
--vllm-server-url https://$USER-vllm-wordle-inference.hf.space \
--env-url https://openenv-wordle.hf.space \
--delta-sync-repo-id $USER/wordle-deltas \
--model Qwen/Qwen3-1.7B
训练器从不开放端口。Space 从未看到训练器的 IP 地址。Wordle 环境不知道它们两者中的任何一个存在。它们都通过 Hub 进行通信。训练在即时 EOS 合理性检查上收敛，然后在真实的 Wordle 部署上收敛：奖励上升，增量负载保持在 20 到 35 MB 区间，每次同步的推理暂停窗口大约为一秒。完整的运行日志链接在随附的 PR 中。
6. 那么这究竟解锁了哪些能力？
有几项，我们认为意义重大。
无需集群即可进行异步强化学习训练。如果你拥有一块 GPU 和一个 Hugging Face 账号，现在就可以实现真正的解耦训练。你的训练器运行在 GPU 上；你的 rollout 集群部署在 Spaces 中；你的环境运行在另一个 Space 里；模型权重通过一个存储桶传输。过去这需要要么是共置部署（会带来所有吞吐量上的妥协），要么是配备共享网络的真实集群。现在不再需要了。
免费的多副本推理。启动两个 vLLM Space，或者十个。它们都从同一个存储桶拉取数据。Xet 采用内容寻址存储，因此连续的锚点在存储时会共享数据块（这能防止你的存储桶无限膨胀），而 Hub 的边缘缓存使得重复下载同一文件变得成本低廉。想要一个全球分布的 rollout 集群？现在这只是一个小型的 DevOps 任务，而不是一个研究项目。
一种可以用现有工具调试的传输格式。一个 delta 文件就是 safetensors 格式。你可以从 notebook 中通过 safe_open 打开它，列出其键名，检查索引，自行计算稀疏度。我们已经在晦涩的 NCCL 流上用 tcpdump 耗费了足够多的时间，因此深知这一点的价值。
一条通往前沿规模的路径。20 到 35 MB 这个数字是针对 Qwen3-0.6B 的。有趣的问题是，当你把参数调大时，曲线会是什么样子。让我们来做一下粗略估算。
以 Llama-3.1-405B 为例。在 bf16 精度下，它在磁盘上占用 810 GB。PULSE 在强化学习学习率下测得平均每步稀疏度为 99%，因此实际的 delta 大约只占参数的 1%。他们在部署中测得的编码在 7B 模型上达到了 108 MB，这正是 PULSE 报告的 **130 倍** 压缩率。按比例线性放大到 405B，每步的 delta 大约为 6 GB。
这在实际时间上能带来什么好处？NCCL 在集群内部确实很快。假设一个慷慨的 100 GB/s 聚合广播带宽（多节点、RDMA，一应俱全）。一次完整同步需要 810 GB / 100 GB/s ≈ 8 秒的推理暂停，每一步都如此。采用增量路径后，训练器在后台将 6 GB 数据流式传输到存储桶，同时生成过程持续运行，而部署服务器的实际暂停窗口仅在于应用步骤，在此规模下只需几秒钟。因此，即使不离开集群，增量方法也能将可见暂停减少 4 倍，并将网络传输字节减少约 130 倍。
现在离开集群。NCCL 完全无法跨云工作。一旦你希望在美东部署一个推理集群，在欧西部署另一个，甚至可能在一个 Hugging Face Space 中再部署一个，基于存储桶的路径就成了唯一路径。在 1 GB/s 的可用互联网带宽下，一次完整的广播需要 13 分钟；而增量方法只需 6 秒。
对于 Fireworks 框架中 1 TB 级别的模型，他们自己的测量数据显示，增量数据为 20.3 GiB，而完整快照为 1024 GiB，减少了约 50 倍。PULSE 更紧凑的稀疏编码将进一步推动这一优势（推算每个增量约 15 GB，接近 65 倍）。无论哪种方式，你都进入了一个通过通用对象存储传输权重不再是权宜之计，而是唯一合理架构的阶段。
7. 我们仍在处理的问题
我们并不假装这已经完成。以下是诚实的清单。
- 两份 CPU bf16 快照，多了一份。训练器保留一份（用于变化检测器），部署服务器保留一份（用于为 vLLM 的 load_weights 重建完整张量）。第一份我们暂时无法摆脱，直到有人找到一种紧凑的分析掩码，这比看起来要难。第二份将在 vLLM 获得稀疏 load_weights API 后消失。PR 即将提交。
- 固定的锚点频率。我们目前每 $N$N 步转储一次完整锚点。自适应策略（“当累积漂移超过 X 时设置锚点”）将在长时间运行中降低锚点成本。
- 多节点 FSDP2 训练器。BF16ChangeDetector 是围绕每个进程的优化器钩子构建的。它应该能干净地推广到 FSDP2，但我们尚未在多节点规模上进行测量。PR 中有一个 TODO，上面写着我们的名字。
- 接入优化器。我们仅从 (m, v) 预测掩码的尝试召回率很低，这意味着分析性 bf16 阈值比教科书公式所暗示的要更精妙。我们非常希望听到任何已破解此问题的人的意见。
- 叠加传输压缩。稀疏 safetensors 和按块 gzip 是正交的。我们尚未尝试将它们结合使用，尽管我们并不期望有巨大的压缩增益。
8. 试试看
- 该 PR：huggingface/trl#5417。分支名为 delta-weight-sync。
- 完整的 Wordle 示例：examples/scripts/openenv/async_wordle.py。
- Spaces 的 Dockerfile：examples/scripts/openenv/vllm_space/ 和 examples/scripts/openenv/wordle_space/。
- 背景阅读：我们的异步强化学习全景文章、Fireworks 1 TB 文章、Cursor Composer 2 报告。
TL;DR, because you have models to train and we respect that:
- Async RL has a dirty secret: every step, the trainer has to ship the whole model to the inference engine. For a 7B in bf16 that is 14 GB. For a frontier 1T model checkpoint that is on the order of a terabyte. Per step.
- It turns out you do not have to. Between two consecutive RL optimizer steps, roughly 99% of bf16 weights are bit-identical (and never less than 98% in the worst case). The actual delta is tiny.
- We landed a TRL PR that encodes just the changed elements as a sparse safetensors file, uploads it to a Hugging Face Bucket, and tells vLLM to fetch it. On Qwen3-0.6B, the per-step payload drops from 1.2 GB to 20 to 35 MB.
- The cherry on top: we ran a full disaggregated training where the trainer was on one box, vLLM lived in a Hugging Face Space, the Wordle environment lived in another Space, and weights flowed through a single Hub bucket. No shared cluster, no RDMA, no VPN.
Async RL just got a lot cheaper. Read on.
Two ways to ship the same weights. Red is wall-clock time during which no tokens are being generated.
1. The One Terabyte Problem
If you read our previous post on the landscape of async RL training, you already know the punchline. Every async RL library, regardless of how it spells "actor model" or which color its NCCL backend is painted, eventually trips over the same root: weight synchronization.
The inference engine speaks the policy of step N. The trainer just finished step N+1. The fresh weights have to get from one side to the other before the inference engine starts drifting hopelessly off-policy. This sits on the critical path whether you are running sync or async: a blocking transfer is wasted idle compute of GPUs not generating tokens. With a sparse delta path you collapse that idle time into seconds, and the trainer does not even have to wait for the inference engine to be ready: it just publishes "weights ready" and uploads the weights to the shared bucket the moment its optimizer step finishes, while the inference engine fetches on its own time.
Fireworks put a very memorable number on this in their post Frontier RL Is Cheaper Than You Think: for a frontier 1T-parameter checkpoint at fp8 (their setting), a full snapshot is 1024 GiB, and that is what conventional wisdom says you have to ship every time you update your rollout fleet. That is the kind of number that gets people to start drawing diagrams with mega-clusters, RDMA fabrics, and dedicated cross-region links. Their measured average delta between adjacent checkpoints lands at 20.3 GiB, or 1.98% of the full model, and "more than 98% of weights in bf16 format remain bit-equivalent between consecutive checkpoints".
Cursor's Composer 2 report tells a parallel story. They run training and inference in different regions and stitch them together with a shared S3 bucket (their exact words), into which the trainer uploads compressed weight diffs every training step. Each cluster independently downloads and reconstructs from the shared delta chain, "requiring no direct connectivity to the training cluster". The two sides never speak to each other about parameters directly. The bucket is the wire.
Both papers agree on three things, and we want to repeat them slowly, because the rest of this post is essentially a faithful open source translation:
- Most of the weights have not actually changed between two adjacent RL steps.
- If you send only the parts that changed, your bandwidth bill collapses by roughly two orders of magnitude.
- If you route those tiny diffs through a shared object store, you no longer need the trainer and the inference cluster to live in the same data center.
The only thing missing was a version of this story that you can pip install. So we wrote one.
2. Why bf16 RL Weights Are Almost Always Sparse
Before we wire anything up, it is worth understanding why this whole game is even winnable. The "98% of weights do not change" claim sounds suspiciously like one of those numbers that works in the demo and falls apart in the wild. It is not. It falls out of how bf16 arithmetic works at the learning rates RL uses.
A bf16 number has 7 mantissa bits. Between two consecutive powers of two, there are exactly $2^{7} = 128$2 7=128 representable values, so the spacing between adjacent bf16 numbers around $\mid w \mid$∣w∣ is roughly $\mid w \mid \cdot 2^{- 7}$∣w∣⋅2−7. An update gets absorbed by the bf16 cast whenever it sits below half of that spacing, i.e., when $\mid \Delta w \mid < \mid w \mid / 256$∣Δ w∣<∣w∣/256. This is the "bf16 visibility threshold" PULSE plots in their Figure 3.
Now look at what Adam does. At an RL learning rate of, say, $3 \times 10^{- 6}$3×1 0−6, the update to a single weight is: $\Delta w = - \eta \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}$Δ w=−η⋅v^+ϵ m^
The normalized step $\hat{m} / \left(\right. \sqrt{\hat{v}} + \epsilon \left.\right)$m^/(v^+ϵ) is roughly order one, so $\mid \Delta w \mid \approx \eta \approx 3 \times 10^{- 6}$∣Δ w∣≈η≈3×1 0−6. For most weights, $\mid w \mid$∣w∣ sits somewhere around $10^{- 2}$1 0−2 to $10^{- 1}$1 0−1 (PULSE reports a median of 0.019 for representative LLM weights). The threshold $\mid w \mid / 256$∣w∣/256 at that magnitude is around $4 \times 10^{- 5}$4×1 0−5 to $4 \times 10^{- 4}$4×1 0−4, which is bigger than the update.
In other words: the optimizer is whispering, and bf16 cannot hear it. The update gets absorbed by rounding, the byte representation of $w$w does not change, and from the inference engine's perspective, this weight did not move. Multiply that by a few hundred million parameters, and you get the >99% sparsity number, for free, with zero approximation.
This is exactly the argument made formal in the PULSE paper (Mihai & Belilovsky, 2026). They define two thresholds. The absorption bound$10 \eta$10 η is the conservative worst case for an Adam update, and the effective bound$\eta$η is the regime you actually live in. The bf16 visibility threshold is $\mid w \mid / 256$∣w∣/256. Whenever the update sits below the visibility threshold, it gets absorbed, and the bf16 byte does not change. Their Figure 3 plots both bounds against a cloud of representative LLM weights, and the conclusion is unambiguous: at $\eta = 3 \times 10^{- 6}$η=3×1 0−6, the absorption bound itself already sits below the visibility threshold for almost every weight in the model. They measure this empirically across Qwen2.5 (0.5B/1.5B/7B), Llama-3.2-3B, and Gemma-3-4B, and consistently find a mean per-step sparsity of ~99%, with a standard deviation of 0.2 to 0.4% over 400 training steps. The worst-case step stays above 98%. So <1% changed is not a lucky measurement; it is what the arithmetic guarantees.
We do not have to predict this analytically (and indeed, we tried predicting the change mask from Adam's $m$m and $v$v statistics, but recall was a sad 30%, more on that later). We just need to observe which bytes flipped. That is a tiny boolean tensor per parameter, computed right around the optimizer step.
Drag the learning rate down to RL territory and watch the cast-back-to-bf16 marker snap to the original tick. The 256-element grid on the bottom left is the aggregate effect across a tiny model.
3. HF Buckets and the Architecture
Here is where the second piece of the story comes in, and where this post stops being a translation of Fireworks/Cursor and starts being a Hugging Face thing.
3.1 What is a Bucket?
A Bucket is a repo type on the Hub designed for high-frequency object storage. No commit ceremony, no PR workflow, no LFS quirks. You add files, you list files, you download files. The Python interface is two functions:
from huggingface_hub import batch_bucket_files, download_bucket_files
# Trainer side
batch_bucket_files("my-org/wordle-deltas", add=[(buffer, "deltas/step_000042.safetensors")])
# Inference side
download_bucket_files("my-org/wordle-deltas", files=[("deltas/step_000042.safetensors", local_path)])
That is it. Two function calls and your weights are in flight.
Under the hood, buckets are backed by Xet, the Hub's content-defined chunking storage layer. Xet looks at every file you upload, slices it into chunks based on its actual content (not fixed offsets), and deduplicates against everything already in the bucket. The practical upshot, which is delightful in this context, is that even if we were too lazy to write the sparse encoding and just uploaded full anchors every step, Xet would still only transfer the changed chunks. Sparse encoding + Xet stack: we pay for what moved, and we pay for it once.
This is the open source equivalent of the "shared S3 bucket" both Fireworks and Cursor reach for, except the storage layer already knows about content hashing, your existing HF token already has permission, and it composes natively with the rest of the stack (Spaces, datasets, models).
3.2 The Three Boxes
The full architecture has exactly three boxes and one shared substrate:
- Trainer. Wherever you want. One GPU, eight GPUs, a laptop with a USB-attached H100, we will not judge. Owns the model weights, runs the optimizer, emits sparse deltas.
- HF Bucket. A single repo, two prefixes:
anchors/for occasional full snapshots anddeltas/for the sparse patches in between. This is the only thing both sides agree on. - vLLM rollout server. Wherever you want, and crucially not necessarily where the trainer is. Pulls from the bucket, applies the delta, and serves rollouts.
- Environment. Hangs off the rollout server in the usual way (HTTP, function calls, whatever your env speaks).
The property to internalize, the one Cursor's paper sells hard and that holds verbatim here: the trainer and the rollout server never talk to each other about weights. They exchange a tiny POST containing {"repo_id": ..., "filename": ...}, and that is the entire control plane. The actual byte transfer happens between each side and the bucket, in parallel, with no shared network fabric.
Why that matters in practice:
- The rollout server can be in another region, another cloud, or behind NAT inside a Hugging Face Space. It does not care.
- N inference replicas can pull the same delta from the same bucket, and Xet deduplicates the bytes across all of them.
- The trainer never has to know how many inference replicas exist, or where, or whether one of them just crashed.
The trainer writes. Replicas read. The Hub does the plumbing.
4. The Protocol
Now we can open the hood. The protocol has four parts: a wire format, a bucket layout, a 30 line vLLM extension, and a trainer side change detector. It is honestly less code than it sounds.
4.1 Safetensors as the Wire Format
We picked safetensors for the on-disk and on-wire format. It is already the canonical checkpoint format on the Hub, every reasonable framework can read it, and the header carries arbitrary string metadata. That metadata field is where we hide the protocol.
There are two kinds of files in the bucket.
Anchors look like a normal checkpoint: one tensor per parameter, full bf16 weights, written every $N$N syncs (we default to $N = 10$N=10).
anchors/step_000010.safetensors
├── model.layers.0.self_attn.q_proj.weight (bf16, full)
├── model.layers.0.self_attn.k_proj.weight (bf16, full)
└── ...
metadata:
sparse=False, model_version=10, sparsity=0.0
Deltas are the interesting bit. For each parameter that actually changed, we store two entries: a flat int32 tensor of element indices, and a bf16 tensor of values at those indices.
deltas/step_000011.safetensors
├── model.layers.0.self_attn.q_proj.weight.indices (int32, [num_changed])
├── model.layers.0.self_attn.q_proj.weight.values (bf16, [num_changed])
├── model.layers.0.mlp.gate_proj.weight.indices
├── model.layers.0.mlp.gate_proj.weight.values
└── ...
metadata:
sparse=True, model_version=11, sparsity=0.9938, changed_params=[...]
A few nice consequences of this choice:
- A delta is a file. You can open it with
safe_open(...)in Python and inspect every tensor in it. No proprietary framing, no length prefixes, no version handshake. - The metadata is self-describing. The receiver reads
sparse=True/Falseand branches. There is no separate manifest. - It is zero-copy via mmap on the inference side, which matters when you are doing this every few seconds.
The cadence is straightforward: anchor every Nth step, delta in between. Both end up in the same bucket under anchors/ and deltas/ prefixes. Each new inference replica only needs to grab the most recent anchor and then replay the deltas since.
Ten training steps. Anchor (full snapshot) on step 1 and step 6, sparse delta on every other step. Files land in the bucket as you watch.
4.2 The Trainer Side: a Boolean Mask From an Optimizer Hook
The trainer needs to know which bf16 elements actually flipped. We do this with a tiny BF16ChangeDetector that registers a pre-step and post-step hook on the optimizer:
class BF16ChangeDetector:
def __init__(self, model, optimizer):
self._pre_step_bf16: dict[str, torch.Tensor] = {}
self._validated_masks: dict[str, torch.Tensor] = {}
optimizer.register_step_pre_hook(self._pre_step_hook)
optimizer.register_step_post_hook(self._post_step_hook)
def _pre_step_hook(self, opt, args, kwargs):
for p in self._params:
self._pre_step_bf16[name_of(p)] = p.detach().to(torch.bfloat16).cpu().clone()
def _post_step_hook(self, opt, args, kwargs):
for p in self._params:
self._validated_masks[name_of(p)] = (
p.detach().to(torch.bfloat16).cpu() != self._pre_step_bf16[name_of(p)]
)
The actual code in the PR has a bit more plumbing (matching optimizer param objects to model params via data_ptr(), because Accelerate wraps them as different Python objects), but the idea fits on a napkin: snapshot, step, diff.
This is ground truth. We tried the more elegant path of predicting the mask from Adam's $m$m and $v$v statistics, using the bf16 ULP threshold directly. It works in principle. In practice, recall was around 30%, which means we would have shipped a delta missing two thirds of the actual updates. Adam's normalization is messy enough that the analytical threshold is not tight. So we just compare bytes. It costs one bf16 CPU snapshot of the model on the trainer side, which we are willing to pay.
The four phases of the new _sync_weight flow are:
- Upload while inference keeps running. The trainer encodes the masked elements into a safetensors buffer and pushes it to the bucket. vLLM is still happily serving the old policy during this whole step.
- Pause vLLM. A short HTTP call, hundreds of milliseconds.
- Signal
/update_weights. Send the bucket coordinates. vLLM downloads, applies, returns. - Resume. vLLM is back on the air.
The log lines tell the story:
Delta: 1234567/200000000 elements changed (sparsity=99.38%)
[delta_engine] uploaded user/wordle-deltas/deltas/step_000042.safetensors (27.4 MB, ...)
Weight sync: done. Total 9.4s (inference paused 1.1s)
The line that matters is the parenthesis. Inference was paused for 1.1 seconds. The remaining 9.4 seconds were spent uploading, which occurred while the rollout server was still generating tokens. With NCCL, we were paying the full sync time as pause time. Here we are paying for it as background time.
A single sync, end to end. Switch between delta-over-bucket and NCCL broadcast, and try the replica count toggle to see the fan-out story.
4.3 The vLLM Side: a 30 Line Extension
vLLM has a clean abstraction for this called WeightTransferEngine. We implement a DeltaWeightTransferEngine whose receive_weights method is, in spirit:
def receive_weights(self, update_info, load_weights):
download_bucket_files(update_info.repo_id, files=[(update_info.filename, local_path)])
with safe_open(local_path, framework="pt", device="cpu") as f:
meta = PatchMetadata.from_metadata_dict(f.metadata())
if not meta.sparse:
# Anchor: feed every tensor and snapshot for future deltas
for name in f.keys():
tensor = f.get_tensor(name)
self._bf16_snapshot[name] = tensor.clone()
load_weights([(name, tensor)])
else:
# Delta: apply (indices, values) to snapshot, hand full tensor to vLLM
for name in json.loads(meta.changed_params):
indices = f.get_tensor(f"{name}.indices").long()
values = f.get_tensor(f"{name}.values")
snap = self._bf16_snapshot[name].flatten()
snap[indices] = values
self._bf16_snapshot[name] = snap.reshape(self._bf16_snapshot[name].shape)
load_weights([(name, self._bf16_snapshot[name])])
We register it via vLLM's --worker-extension-cls flag, which means no fork of vLLM is required. You install TRL into the same image as vLLM, point the CLI at our class, and you are done.
Worth mentioning: vLLM itself has an in-flight effort to land sparse weight transfer natively, vllm-project/vllm#40096. It adds receive_sparse_weights() and trainer_send_sparse_weights() directly on the WeightTransferEngine base class, with patches encoded as (indices, values) and applied in place via index_copy_(), removing the GPU/CPU validation roundtrip entirely. The PR reports a transfer of 0.16 MB in 0.40 ms for a sparse patch on Qwen3-1.7B versus 942 MB in 192 ms for a full dense send.
One honest caveat in our implementation on the inference side: we keep a CPU bf16 snapshot of the model so we can reconstruct full tensors from sparse (indices, values) patches, because load_weights in vLLM today expects full tensors. Once #40096 (or its successor) lands and exposes an in-place sparse load_weights path, we can apply the indices directly on the GPU and drop the snapshot!
5. Standing It Up on Spaces, For Real
This is the part we are smug about. Everything we have described so far works on your laptop, but the point of routing weights through a Hub bucket is that the trainer and the rollout server do not have to live anywhere near each other. So we ran a fully disaggregated training with three machines, none of which share a network:
- A box with one GPU running the trainer.
- A Hugging Face Space (Docker SDK, L4 GPU) running vLLM with our extension class.
- A second Hugging Face Space (CPU) running the Wordle environment server with 256 concurrent session capacity.
- A Hub bucket in the middle.
Setting this up is genuinely a few hf CLI calls. The vLLM Space's Dockerfile is essentially the upstream vLLM image plus pip install trl@... plus the entrypoint:
FROM vllm/vllm-openai:latest
RUN pip install "trl @ git+https://github.com/huggingface/trl.git@delta-weight-sync"
ENV VLLM_SERVER_DEV_MODE=1
EXPOSE 7860
ENTRYPOINT ["vllm", "serve", "Qwen/Qwen3-1.7B", \
"--host", "0.0.0.0", "--port", "7860", \
"--worker-extension-cls", "trl.experimental.async_grpo.delta_engine.DeltaWorkerExtension", \
"--weight-transfer-config", "{\"backend\":\"nccl\"}", \
"--max-model-len", "32768", \
"--gpu-memory-utilization", "0.8"]
Deploy it as a Space:
hf repos create $USER/vllm-wordle-inference \
--type space --space-sdk docker --flavor l4x1 \
--secrets HF_TOKEN=$HF_TOKEN
hf upload $USER/vllm-wordle-inference examples/scripts/openenv/vllm_space/ --type space
And kick off training from anywhere on the planet that can talk HTTPS:
python examples/scripts/openenv/async_wordle.py \
--vllm-server-url https://$USER-vllm-wordle-inference.hf.space \
--env-url https://openenv-wordle.hf.space \
--delta-sync-repo-id $USER/wordle-deltas \
--model Qwen/Qwen3-1.7B
The trainer never opens a port. The Space never sees the trainer's IP. The Wordle environment does not know either of them exists. They all talk to the Hub. Training converged on the immediate-EOS sanity check, then on real Wordle rollouts: reward went up, delta payloads stayed in the 20 to 35 MB band, and the inference-paused window per sync stayed around a second. The full run logs are linked in the companion PR.
6. So What Does This Actually Unlock?
A few things, and we think they are big.
Async RL training without a cluster. If you have one GPU and a Hugging Face account, you can now do real disaggregated training. Your trainer is on the GPU; your rollout fleet lives in Spaces; your environment lives in another Space; weights move through a bucket. This used to require either a colocated setup (with all the throughput compromises that brings) or a real cluster with shared networking. It does not anymore.
Multi-replica inference, for free. Stand up two vLLM Spaces, or ten. They all pull from the same bucket. Xet content-addresses storage so consecutive anchors share chunks at rest (which keeps your bucket from blowing up), and the Hub's edge cache makes repeated downloads of the same file cheap to serve. Want a globally distributed rollout fleet? That is now a small DevOps exercise, not a research project.
A wire format you can debug with your existing tools. A delta is a safetensors file. You can safe_open it from a notebook, list its keys, inspect the indices, compute the sparsity yourself. We have spent enough hours in tcpdump on opaque NCCL streams to appreciate this.
A path to frontier scale. The 20 to 35 MB number is for Qwen3-0.6B. The interesting question is what the curve looks like once you turn the dial up. Let us do the napkin math.
Take Llama-3.1-405B. In bf16 that is 810 GB on disk. PULSE measures 99% mean per-step sparsity at RL learning rates, so the actual delta sits around 1% of the parameters. Their deployment-measured encoding hits 108 MB on a 7B model, which is the **130×** reduction PULSE reports. Scaled linearly to 405B, the delta lands at roughly 6 GB per step.
What does that buy you in wall-clock? NCCL is fast inside a cluster, sure. Assume a generous 100 GB/s aggregate broadcast bandwidth (multi-node, RDMA, the works). A full sync is 810 GB / 100 GB/s ≈ 8 seconds of inference pause, every step. With the delta path, the trainer streams 6 GB to a bucket in the background while generation keeps running, and the rollout server's actual paused window is just the apply step, which on this scale lands at a couple of seconds. So even before we leave the cluster, delta cuts the visible pause by 4× and the bytes on the wire by ~130×.
Now leave the cluster. NCCL straight up does not work across clouds. Once you want a rollout fleet in us-east, another in eu-west, maybe one in a Hugging Face Space, the bucket-based path is the only path. At 1 GB/s of usable internet bandwidth, a single full broadcast would take 13 minutes; the delta does it in 6 seconds.
For a 1 TB-class model in the Fireworks framing, their own measured numbers show 20.3 GiB deltas vs the 1024 GiB full snapshot, a ~50× reduction. PULSE's tighter, sparse encoding would push that further (extrapolating ~15 GB per delta, closer to ~65×). Either way, you are in a regime where shipping weights through commodity object storage stops being a hack and starts being the only sensible architecture.
7. What's Still on Our Plate
We are not pretending this is finished. Here is the honest list.
- Two CPU bf16 snapshots, one too many. The trainer keeps one (for the change detector) and the rollout server keeps one (to reconstruct full tensors for vLLM's
load_weights). The first one we are stuck with until someone finds a tight analytical mask, which is harder than it looks. The second one goes away when vLLM gains a sparseload_weightsAPI. PR forthcoming. - Fixed anchor cadence. We currently dump a full anchor every $N$N steps. An adaptive policy ("anchor when cumulative drift exceeds X") would cut anchor cost on long runs.
- Multi-node FSDP2 trainers. The
BF16ChangeDetectoris built around per-process optimizer hooks. It should generalize cleanly to FSDP2, but we have not measured it at multi-node scale yet. There is aTODOin the PR with our name on it. - Hooking into the optimizer. Our attempt at predicting the mask from $\left(\right. m , v \left.\right)$(m,v) alone gave low recall, which means the analytical bf16 threshold is doing something more subtle than the textbook formula suggests. We would love to hear from anyone who has cracked this.
- Stacking with on-the-wire compression. Sparse safetensors and per-chunk gzip are orthogonal. We have not tried combining them yet. Although we don't expect huge compression gains.
8. Try It
- The PR: huggingface/trl#5417. Branch is
delta-weight-sync. - The full Wordle example:
examples/scripts/openenv/async_wordle.py. - The Spaces Dockerfiles:
examples/scripts/openenv/vllm_space/andexamples/scripts/openenv/wordle_space/. - Background reading: our async RL landscape post, the Fireworks 1 TB post, the Cursor Composer 2 report.