Batch-1 解码的重要性与日俱增。例如，小米 MiMo 在六月发布了 MiMo-V2.5-Pro UltraSpeed，声称在一万亿参数的 MoE 模型上实现了 1000 token/秒的解码速度。
Batch 1 让推理栈毫无隐藏开销的余地。没有批次来分摊启动成本，没有并发来填补流水线气泡，也没有足够的算术强度让巧妙的平铺策略奏效。关键路径上的每一微秒，都是用户等待的一微秒。
本文旨在为 Ling-3.0-flash（一款混合线性注意力 MoE 模型）在 4 块 NVIDIA Blackwell GPU 上降低这一性能下限。文章涵盖两条投机解码路径。在 NEXTN/MTP 路径上，我们将单请求解码速度从 288 token/秒提升至 606 token/秒，平均 TPOT 从 3.33 毫秒降至 1.53 毫秒。第二条路径是 DSpark，一个基于相同技术栈构建的、采用置信度调度的投机解码器：在 1000 请求的测试中，达到了 1120 token/秒的吞吐量，平均 TPOT 为 0.78 毫秒，接受长度为 9.95。最后这一项对比是受控实验：NEXTN 和 DSpark 在同一台机器上使用相同命令进行测量，平均 TPOT 降低了 1.9 倍（从 1.53 毫秒降至 0.78 毫秒）。本文其余部分将探讨这些时间消耗在哪里，以及如何将其挽回。
亮点
- 最终结果：平均 TPOT 降低 54%（从 3.33 毫秒降至 1.53 毫秒），单请求吞吐量提升 2.1 倍（从 288 提升至 606 token/秒）。在受控的 1000 请求对比中，DSpark 达到了 0.78 毫秒的平均 TPOT 和 1120 token/秒的吞吐量。
- 优化路线依次是：主机端预取（host run-ahead）、PDL 链式化、内核优化，最后是 DSpark。移除每步的主机端固定操作让准备工作得以隐藏在 GPU 工作之后；随后 PDL 将 MoE、路由器、KDA 和全归约路径链接起来；两次融合、一次 KDA 重新调优以及 bf16 路由器/lm_head GEMM 运算缩短了剩余的 GPU 关键路径。
- 将数值精度作为带宽调节旋钮：将路由器门控和 lm_head 从 fp32 迁移到 bf16 是结构变更后最大的一项改动，大约带来了 +10% 的性能提升。
- 全程贯彻测量纪律：在得出任何主机端结论之前先进行带剖析与不带剖析的校准，进行冷权重微基准测试，并基于平均 TPOT 而非单窗口峰值来做 A/B 决策。
- DSpark 将每次验证步骤承诺的 token 数提升至：并发数为 1 时，接受长度为 9.95，吞吐量达 1120 tok/s，平均 TPOT 为 0.78 毫秒。与同一 1000 请求基准测试下的 NEXTN 相比，平均 TPOT 降低了 1.9 倍。
图 1. 四种配置下的主要结果。
| 指标（8192 输入 / 1024 输出，单并发，贪心解码，TP4 bf16） | 基线 | 草稿扩展图修复后 | NEXTN，调优后 | 采用 DSpark |
|---|---|---|---|---|
| 平均输出吞吐量 | 288 tok/s | 526 tok/s | 606 tok/s | 1120 tok/s |
| 平均 TPOT | 3.33 毫秒 | 1.76 毫秒 | 1.53 毫秒 | 0.78 毫秒 |
| 中位 TPOT | — | — | 1.56 毫秒 | 0.51 毫秒 |
| 峰值输出吞吐量 | — | — | 1099 tok/s | 1945 tok/s |
| 接受长度 | 3.14 | 3.13 | 3.25 | 9.95 |
在 GSM8K 上，同一技术栈的得分为：准确率 0.889，无效 0.000，延迟 341.5 秒，输出吞吐量 511.1 tok/s。
所有运行均使用 Ling-3.0-flash，在 4 块 Blackwell GPU 上，TP4，bf16，并发数 1，贪心解码，以及相同的固定 8192 输入 / 1024 输出随机工作负载。从左到右，各列依次显示初始 NEXTN 基线、草稿扩展图修复后的 NEXTN、最终调优后的 NEXTN 以及 DSpark。前两项是较短的战役检查点；后两项是受控对比，均在相同机器上对相同的 1000 个请求进行测量。峰值吞吐量仅在后两次运行之间进行比较，因为它是固定一秒窗口内的最大值。
这里有两个定义很重要，因为它们共同解释了为什么即使在并发数为 1 时，输出吞吐量也不仅仅是平均 TPOT 的倒数：SGLang 的 TPOT 不包含 TTFT，而输出吞吐量是将总输出 token 数除以总基准墙钟时间（参见 bench_serving 指南）。本文所有主要基准运行均使用合成随机工作负载；特别是接受长度取决于提示词和输出分布，因此 9.95 是该工作负载的接受长度，而非模型的接受长度。
模型
图 2. Ling-3.0-Flash 架构：42 层，交错排列 35 个 KDA 线性注意力层与 7 个 MLA 全注意力层，覆盖 512 专家的 MoE。
Ling-3.0-flash 是一个混合注意力 MoE 模型（BailingMoeV3），下文的大部分内容都源于“混合”这个词。
| 层数 | 共 42 层：35 个 KDA 线性注意力 + 7 个 MLA 全注意力 |
|---|---|
| MoE | 512 个路由专家 + 1 个共享专家，top-8（+1），moe_intermediate_size 768 |
| 隐藏层大小 | 2560 |
| 词表 | 约 157k，通过词表并行 lm_head 提供服务 |
| 权重 | 每 rank 约 63 GB（bf16 精度） |
| 部署 | 4 块 NVIDIA Blackwell GPU，TP4，bf16，NEXTN 投机解码 |
每六个注意力层中有五个是 KDA。这正是 MLA 注意力在最终配置下、8k 上下文时每步仅需 244 微秒的原因，也是该模型从一开始就适合作为 batch-1 目标的原因：注意力开销低、batch 极小，关键路径上剩下的就是权重带宽和启动延迟——而这正是本文要讨论的场景。
batch-1 单步的形态
我们使用 NEXTN 投机解码进行解码，参数为 steps=5、topk=1、draft_tokens=6。一次解码步骤由三个 CUDA graph 接力完成。
图 3. 每步三个 graph。草稿模型提出一条 6-token 链，目标模型在一次前向中给全部六个 token 打分，extend graph 用目标模型真实的隐藏状态重放被接受的 token 前缀，以生成下一轮的种子。判定本身（eagle_sample）发生在 verify graph 内部；主机在晚一步之后才知道接受了多少个 token。
草稿是一个单层 NEXTN 模型，以自回归方式运行：五个步骤但只有四次前向，因为第一个候选来自上一轮的种子，第五个则从第四次前向的 top-k 中读出。Verify 是对完整 42 层目标模型在全部六个链位置上的单次前向。Extend 修正草稿的 KV cache——该 cache 只见过草稿自身的猜测——并把种子交回给下一轮。
三个 graph 之间在 CPU 上没有任何数据交换。固定形状加填充使得每个依赖接受数量的计数都成为 GPU 索引而非主机值；持久缓冲区让生产者 graph 直接写入消费者缓冲区；而真正需要 CPU 拿到数值的决策（EOS、停止字符串、反 token 化）则通过旁路 D2H 流和晚一步消费的 copy_done 事件完成。下面所有内容都建立在这一性质之上。
两种空闲时间
刚开始时，GPU 在每一步中大约有三分之二的时间在忙。batch 1 下的空闲时间有两种，它们需要分别诊断，因为对应的修复手段毫无共同之处：
- 主机模式空闲。每一步重放三次图，图内执行数百个内核节点，图与图之间的接缝处由 Python 胶水代码衔接。（是三次重放，而非三次前向：草稿图捕获的主体已包含全部四次前向，因此自回归草稿循环只需一次重放，而非四次。）如果主机每步循环耗时超过 GPU 单步耗时，GPU 就会挨饿。解决办法是隐藏并压缩主机端工作。
- GPU 模式空闲与 GPU 模式开销。主机端被隐藏后，剩下的就是权重带宽（每个 MoE 层每步冷读约 94 MB 已激活专家权重）加上数百个小内核节点固有的延迟下限。批大小为 1 时两者都无法摊薄。解决办法是 dtype 处理、算子融合和启动依赖调度。
图 4。两种空闲形态。上图：主机循环比 GPU 工作更长，因此空洞少而宽，落在图与图之间的接缝处。下图：主机端被隐藏后，剩下的是数百个 1.5–6 µs 的内核节点，其启动下限与自身算术量相当，再加上权重读取本身。
这两种空闲描述了 TPOT 的步时侧。另一个杠杆是每步提交多少个 token：平均 TPOT ≈ 步时 / 平均接受长度。本文其余部分沿着这些杠杆展开。主机预跑和接缝工作消除主机模式空闲；PDL、dtype 变更、算子融合和重新调参缩短 GPU 关键路径；投机采样调优和 DSpark 增加每个目标步提交的 token 数。DSpark 稍后会回到第一类问题——当阻塞式 D2H 读取重新引入主机瓶颈时。
先修尺子，再修机器
测量设置的三项属性影响着下面的每一个数字。
分析器会放大主机端事件的开销。CUPTI 会为其记录的每个主机事件增加额外开销。在相同配置下，被分析步骤测得耗时为 5.2 毫秒，而从未被分析运行中通过 TPOT × 接受长度反推计算出的真实步骤耗时为 4.9 毫秒。这 0.3 毫秒的差距与我们想要分析的主机端效应处于同一量级，因此被分析的跟踪记录可能显示出在关闭分析器时并不存在的跨秩等待。GPU 内核耗时来自硬件时间戳，比主机端计时更可信，但也并非完全免疫：跟踪仍会扰动启动时序、并发性、缓存状态以及 CUDA 图执行，而且 Nsight Systems 文档也指出 CUDA 和图节点跟踪可能带来显著开销（用户指南）。因此，这里得出的每一个主机端结论都先经过了“被分析 vs 未被分析”的校准。
微基准测试对冷权重内核的运行结果偏乐观。一个循环反复调用同一内核，会使其 2.6 MB 的门控权重常驻于 L2 缓存中，而真实模型在两次调用同一层之间会用约 94 MB 的专家流量冲刷 L2 缓存。热态 7 微秒，冷态 11 微秒：这一差距足以扭转与库 GEMV 相比的排名。
峰值吞吐量是单窗口统计量。基准测试的峰值数字是固定 1 秒网格上的最大值，因此它带有大约 ±5% 的相位带：TTFT/TPOT 的偏移会重新切分网格，而一项使平均吞吐量提升 2.3% 的改动，打印出来却可能显示为从 909 下降到 858。在固定随机种子下，两种读数都能精确复现，因此可复现性并不能区分真实信号与相位噪声。这里的 A/B 决策基于平均 TPOT × 平均接受长度。该乘积是对步骤时间的推导估计值，而非实测值（两个聚合值的乘积并不等于乘积的聚合值），但它在多次运行中保持稳定，并且在这些运行中对接受长度漂移不敏感——这正是 A/B 判据所需要的特性。我们报告峰值，但从不针对它进行优化。
正确性有自己的一道关卡，在每一项改动被保留之前都会经过它：对 256-token 贪婪生成结果进行逐字节精确比较，接受长度在 0.05 范围内保持不变，并且在交错插入温度采样请求之后重新运行一次贪婪生成，以捕获状态污染。那些合理地改变舍入行为的改动（bf16 门控、单次舍入合并）会在提交信息中说明这一点，并改用接受率和任务指标来验证，而不是逐位一致性。
让主机端提前运行
这是整个行动所依赖的结构性改动，本质上是一个主机端空闲修复。
图 5. 从锁步到深度流水线。之前：每一步主机端都会在 resolve_seq_lens_cpu 中阻塞，等待上一个验证图在 GPU 上完成，因此提前运行深度被重置为零，每个主机端预处理段都变成 GPU 气泡。之后：队列深度达到整整一步，验证 k+1 的启动比其自身执行提前了整整一步，唯一剩下的同步是一个延迟一步消费的 copy_done 事件。
cudaGraphLaunch 一直是异步的，GPU 上的 draft → verify → extend 顺序是免费的：同一条流，FIFO 顺序。所以问题从来不是 verify 是否等待 draft，而是主机端是否每一步都被钉在 GPU 进度上。
确实如此。在 spec-v2 下，调度器不知道接受长度，因此 FutureMap.resolve_seq_lens_cpu() 在构建下一批数据时从 GPU 拉回 new_seq_lens：以 publish 事件为门控，在私有流上拷贝，然后调用 synchronize()。主机端等待的不是微秒级的拷贝，而是上一个验证图执行完毕。中位成本：每步 485 微秒，而且提前运行深度每一步都被重置为零。
原因是 needs_cpu_seq_lens 标志，它由 spec-v2 涉及的每个后端做 OR 运算。trtllm_mla 在三种角色中都声明为 False；同族的线性注意力后端 GDNAttnBackend 和 Mamba2AttnBackend 都显式声明为 False。KDAAttnBackend 从未声明过它，因此继承了基类默认值 True，尽管它运行的是与其两个同族后端相同的基类元数据代码。
将 `needs_cpu_seq_lens` 设为 False 后，OR 操作被合并，逐步骤同步也被移除。其正确性论证是逐点成立的：KDA 的元数据从不读取 CPU 镜像，而回放填充数据来自 `forward_batch.num_padding`。
主机怎么敢在不知道第 k 步接受了什么的情况下就启动第 k+1 步？因为这些值从不触及 CPU。FutureMap 是一个驻留在 GPU 上的中继器：第 k 步的图将输出 token、new_seq_lens、top-k 概率和隐藏状态写入由 `req_pool_idx` 索引的设备缓冲区，第 k+1 步的图则通过相同的索引读取它们。主机只处理索引，而这些索引它早已知道。
图 6. 松弛空间所在之处。面板 A：主机循环（约 4.3 毫秒）完全容纳在 GPU 步骤（约 4.9 毫秒）之下，因此被完全隐藏。面板 B：当抖动（一次 gloo 广播或一次 GC 暂停）超过松弛空间时，主机完成时间偏晚，GPU 在下一个校验边界处等待，此时图中的第一个集合通信操作吸收了跨秩偏差。
超前运行也改变了主机开销的形态。不再是每个秩每一步都直接付出其主机时间，而只有耗尽队列松弛空间的秩才需要付出。在一次四秩追踪中，恰好有一个秩处于这种状态：其调度器段运行时间比同级秩长 5-10 倍，其草稿图启动延迟 40-80 微秒，其草稿→校验接缝比其余秩的中位数高出 165 微秒，并且周期性出现带有 GC 特征的 400-750 微秒尖峰。另外三个秩在每次会合点都自旋等待它。可推广的诊断结论是：内核的持续时间并不等于它的工作量。一个 20 KB 的嵌入向量全归约显示 150-480 微秒，并不是全归约本身慢；它是在吸收偏差，只有跨秩时间对齐才能告诉你哪个秩迟到了。
弥合接缝
随着锁步固定点被移除，图之间的接缝就值得缩小了。在 CUDA 图重放之前，需要将特定步骤的注意力元数据（kv 索引、块表、mamba 状态槽）从实时的 `req_to_token` 和 `seq_lens` 重建到图所捕获的静态缓冲区中。这种回填每一步都会急切地执行，并且是接缝内容的主要部分。在批大小为 1 时，它纯粹受主机限制：每个操作需要 5-15 微秒来派发，1-4 微秒来执行。
我们从两个层面入手。首先，融合索引链：`assign_extend_cache_locs_uniform` 在内核内部计算结束偏移量（统一的 `draft_token_num` 扩展使得跨行前缀和不再必要），而 `_fused_state_indices_kernel` 将一次 gather、一次 translate、一次填充哨兵写入和一次 `copy_` 合并为一次启动，同时小心保留所有副作用，包括在填充行上将 `req_pool_indices` 清零——该函数本身并不需要这一操作，但图中其他被捕获的内核依赖它来保证越界安全的 gather。
其次，将 refill 本身捕获到一个以 `(bs, forward_mode)` 为键的小型 CUDA 图中。这之所以可行，是因为重放契约已经保证了一个指针稳定性属性：重放时的 `ForwardBatch` 视图只向后端传递 runner 静态缓冲区和池驻留张量，因此整个准备序列的地址是固定的。围绕它设有四重安全机制：两次 eager 预热，确保 Triton JIT 和自动调优发生在捕获之外；每次重放前恢复各后端 `forward_metadata` 对象的快照（图重放设备操作，快照恢复 Python 指针）；捕获失败时永久回退到 eager 模式并发出警告；以及对 padding、TBO、pdmux 和 LoRA 的防护。该功能以 `SGLANG_ENABLE_METADATA_GLUE_GRAPH` 为开关默认关闭，并在 DFLASH 系列投机解码路径上强制禁用，因为该路径每一步都会在主机端重建注意力计划，捕获 refill 会把计划冻结在捕获时刻。
可捕获的内容存在硬性边界。判据是：由纯设备内核组成、写入持久缓冲区的 refill 可以捕获；任何经过 FlashInfer 风格 `plan()` 的路径则不可捕获。draft 侧不满足该条件：多步 draft 后端会对主 EAGLE 图已捕获的 wrapper 重新执行 `plan()`，而将这次重新计划记录到次级图中会在重放时破坏 wrapper 的内部状态。另一个相关要求是捕获必须幂等。`trtllm_mla` 的 `_init_cuda_graph_metadata` 过去每次调用都会分配新的张量并替换其 `decode_cuda_graph_metadata[bs]` 条目，这会导致第二次捕获后早期图读取到已释放的内存。
PDL：堆叠小内核的延迟下限
批量大小为 1 的步骤会在很短的时间窗口内执行数百个内核节点。在这种规模下，启动和前导开销与数学计算本身的成本相当。程序化依赖启动（PDL）允许消费内核在其生产者仍在运行时就被调度到 SM 上：消费内核会执行所有不依赖生产者输出的部分，并且仅在依赖读取之前于 gdc_wait() 处设置栅栏。
图 7. 路由器路径上的 PDL。没有 PDL 时，每个内核只能在前一个内核完全退出后才会启动，并且门控矩阵向量乘的冷 HBM 权重加载位于关键路径上。有了 PDL，权重分块的加载不依赖生产者，因此它会在 gdc_wait() 之前发出，一次 2.6 MB 的冷读取在生产者尾部下方飞过；路由器 top-k 以相同方式预取其偏置。
我们串联了三条链：MoE 主链（moe_align → up-GEMM → activation → down-GEMM → combine → all-reduce）、路由器链（norm → gate matvec → top-k）以及 KDA 链（conv1d_update → recurrent delta-rule → gated norm）。有两个设计要点值得注意。
不依赖生产者的加载放在等待之前。这就是图中的全部诀窍，也正是它让 PDL 对于延迟受限内核而言不仅仅是消除启动开销。
Inductor 内核无法携带 PDL 属性。小 M 的 MoE combine 是一个 torch.compile 生成的内核；加入该链意味着要将其替换为仓库中的 Triton 归约加 GDC。这产生了一个数值上的副作用：fp32 求和 × 缩放并只做一次最终类型转换，而旧路径会进行两次舍入。结果在精度上略有提升，但并非逐位相等，这一点已在提交信息中声明。
后来，我们在 PTX 的 griddepcontrol 中发现了一个语义问题，于是升级了相关语义：launch_dependents 只会释放依赖网格的启动，而消费者的 wait 总是会等待生产者网格完全退出。把触发点从生产者的末尾移到生产者自身 wait 之后紧接的位置，可以让消费者的前奏与生产者主体（而不仅仅是尾部）有更多重叠，但有一个前提条件：消费者必须在提前启动与每次读取生产者输出之间，仍然保留自己的 gdc_wait()。这是每个消费者各自的属性，并非普遍保证，因此我们逐个内核进行了检查，并转换了六个内核。提前触发带来的收益也并非确定性的：驱动程序可能会提前启动依赖网格，而实际能实现多少重叠，取决于当时的调度和资源压力（参见 CUDA 编程指南）。fused_moe 将这一行为限制在 M ≤ 512 的检查之后：在 prefill 形状下，提前释放大型消费者网格会从生产者那里抢占 SM，而在 decode 形状下则是纯粹的收益。
PDL 是纯粹的调度语义。累积顺序不变的改动，结果保持逐位一致；gate matvec 通过了 4/4 的 GDC 开/关位对比测试。
两个融合与一次重新调优
moe_align，在 pair 轴上进行。Triton 融合 MoE GEMM 按 block_size 的块来消费 token，其中每一行都共享同一个专家，而 moe_align_block_size 负责构建这种排列。通用路径需要两次内核启动：在每一个专家的偏移量最终确定之前，任何 token 都无法放置；这些偏移量来自一次网格级扫描，而设备级屏障只存在于内核边界处。存在单次启动的变体，但它将每线程的专家计数器暂存在共享内存中，因此仅限于 64 个或更少的专家；513 个专家的 decode 场景始终需要两次启动。
这种替换在“配对轴”上运作：一次 [NP, NP] 的成对比较，就能让每个 (token, slot) 配对在其所在桶内获得稳定排名，同时一次性得到该桶的容量；随后，每个桶的 rank-0 代表元素推导出填充计数、按桶排序的独占偏移量、已发布的总数，以及每个块的专家 ID。没有任何环节随专家数量扩展，因此专家数量上限随之消失。显而易见的替代方案——在填充后的专家轴（最多 1024 个桶）上做直方图和前缀和——虽然正确，但相比它所替换的两个内核，会在关键路径上多出约 3 倍的单 SM 工作量。这正是配对轴在此处成为承重结构的原因。
与参考实现相比，有两处刻意偏离，且均以消费者不变量为论证依据：桶内顺序按配对索引保持稳定，而非原子调度顺序（每个配对写入自己的输出行，因此消费者对顺序不敏感）；缓冲区在已发布总数之后的尾部保持未写入状态（消费者 CTA 在读取该区域之前会提前退出）。一个陡峭的边界：配对张量的复杂度为 O(NP²)。在 NP=64 时，它们完全驻留在寄存器中（约 4 微秒，与 CUDA 双内核路径相当）；在 NP=256 时，则溢出到本地内存，每次启动约耗时 230 微秒。分派门限是硬性的 numel ≤ 64；更大的批次则回退到 CUDA 路径。
在 up-GEMM 的尾声（epilogue）中实现 SwiGLU。将 silu(gate) * up 的运算折叠进 MoE up-GEMM 的尾声，可移除每个 MoE 层中独立的激活内核，以及中间缓冲区的整段“先写后读”流程。其布局技巧在于，在权重加载时对 w13 按专家进行行交错，使得 gate 和 up 落在同一输出 tile 的相邻偶/奇数列上。由于每个 GEMM 输出列都是独立的点积，交错在比特层面是中性的。
位级一致性才是这里真正花心思的地方。被替换的内核是用 `-use_fast_math` 编译的，所以收尾阶段（epilogue）要逐指令地复现它：`__expf` 对应 `mul + ex2.approx.ftz`，`div.approx.ftz`，以及乘积上最后唯一一次舍入。微妙之处在于：FlashInfer 在 float 精度下实例化激活函数子（activation functor），所以 silu 在乘法之前绝不会落到 bf16。如果在那里做舍入，结果就会发生双重舍入，并在很大一部分输入上产生偏差。这在文档里看不出来，容差检查也发现不了；必须对完整输入范围做逐元素的位级比较才能暴露。
KDA 链式校验 tile 的经济性。融合的 conv1d + 门控 delta-rule 校验内核本来就已经存在；这些提交只是对它重新调参。在带图内计时的旋转冷启动测试中，Blackwell 曲线在 T=6 时是单调的：BV=4 为 11.56 µs，8 为 12.53，16 为 12.83，32 为 14.26，64 为 20.7，128 为 38。BV=4 每次调用最多可节省 19% 的时间，因为 256 个 CTA 在 148 个 SM 上相当于 1.7 波，而把 q/k 卷积复制 32 次仍然比缩短串行链更划算。对 V 维度的 tile 切分从不影响 K 维度的归约顺序，所以在 num_warps=4 时，每个 BV 都与基线逐位一致，这次重新调参不带来任何数值风险。
bf16 路由器门控与 lm_head
结构改动之后最大的一项单一变更是一次数据类型变更。在 batch 为 1 时，路由器门控和 lm_head 是纯带宽操作：每个解码步骤都会冷读每个 MoE 层的门控权重（bf16 下为 2.6 MB）以及词表并行的 lm_head 投影，而两者都没有足够的算术运算来掩盖读取延迟。把两者从 fp32 改为 bf16 可将这些字节数减半；端到端大约带来 +10% 的提升，是主机端 run-ahead 修复之后所有单项变更中收益最大的一次。与上面其他舍入改动一样，这次变更也在提交信息中做了声明，并通过接受长度和任务指标来验证，而非位级一致性。
推测执行下的 KDA
被拒绝的推测 token 会让 KV 缓存条目无害地变陈旧，但它已经就地破坏了循环状态。线性注意力与推测执行并不能免费共存。
其运作机制如下：在验证阶段，循环以禁用状态更新的方式运行，并将链上每个位置的验证后状态写入中间缓冲区；验证结论得出后，`commit_mamba_states_after_verify` 将最后一个被接受位置对应的状态复制到持久化槽位中。先暂存，再提交。这也是紧凑型推测缓存被限制为 topk=1 的原因：对于链式结构，被接受的前缀是唯一的，状态可以按位置索引；而对于树状结构，被接受的路径只是众多路径之一，状态必须按树路径来索引。
性能分析显示，KDA 解码受带宽限制，主要瓶颈是 K×V 状态上的 HBM 流量，因此除了融合和分块重调优之外，这方面的优化空间已所剩无几。我们并未将实测带宽与 Blackwell 的峰值带宽进行对比，因此请将此视为一种形态观察，而非罗浮线（roofline）分析结果。
批大小为 1 时推测的经济性
权重带宽主导地位有一个反直觉的推论：验证更多 token 几乎是无成本的。验证 4 个 token 和验证 6 个 token 读取的权重完全相同。在批大小为 1 时加深推测深度，每增加一步只需额外执行一次廉价的草稿前向传播（草稿仅为一层），再加上 KDA 链式验证循环中递增的串行开销，而换来的是更长的接受长度。
我们通过扫描实验而非假设来确定这一参数（该扫描在融合包之前进行；此后最优值发生了变化，详见下文）：
| 步数 / 草稿 token 数 | 接受长度 | 平均 TPOT | 单步耗时（推算值） | 稳态 tok/s |
|---|---|---|---|---|
| 3 / 4 | 3.11 | 1.51 ms | 4.70 ms | 662 |
| 4 / 5 | 3.37 | 1.45 ms | 4.89 ms | 690 |
| 5 / 6 | 3.45 | 1.55 ms | 5.35 ms | 645 |
单步耗时列由 TPOT × 接受长度推算得出，属于估算值而非直接测量值。每增加一步，单步耗时增加约 4-9%，而边际接受增益则呈几何级数递减（d5 → d6 仅增加 0.08）。盈亏平衡条件大致为 Δaccept > 0.05 × accept。最优配置也会移动：在融合包落地、单步耗时下降后，(5, 6) 成为更优配置；一旦 fp8 权重进一步压缩固定基础开销，则需要再次进行扫描。
DSpark：高质量块级草稿生成
调整 NEXTN 的深度只是在固定形状的算法上拧一个一维旋钮。更大的杠杆在于改变算法本身，而本次活动的后半段正是将 DSpark 引入同一目标，并给予它与 batch-1 相同的待遇。
DSpark 的做法有何不同
DSpark 算法本身是公开的。这里的工作是将这一公开方案适配到 Ling-3.0-flash、长上下文在线知识蒸馏以及 batch-1 Blackwell 推理服务栈上。我们的适配在四个方面有所不同。
分布对齐的数据。我们主要在 Ling-3.0-flash 的后训练数据上进行蒸馏，因此草稿模型训练时所面对的数据分布与推理服务时一致。我们还在蒸馏过程中使用多种采样设置，以提升轨迹多样性，并增强在投机解码下的鲁棒性。
消融驱动的草稿设计。我们没有直接继承 Ling-3.0-flash 的架构，而是对关键的草稿设计选择进行了系统性消融实验，包括是否复用 Ling-3 的注意力结构，以及使用哪种 RoPE 变体（部分式还是交错式）。我们保留了在接受长度/延迟权衡上表现最佳的设计。
与推理服务耦合的在线训练系统。为了支持长上下文和大规模的在线训练，我们构建了 SplitServe Trainer——一个单节点 8-GPU 框架，将资源在训练与 SGLang 推理之间均分。训练期间，推理侧运行目标模型的前向传播，以产生监督信号，例如供草稿模型使用的目标隐藏状态。这使得生成-训练循环保持本地化，削减了 IO 开销，并提升了长上下文工作负载的训练效率。
图 8. SplitServe Trainer 布局。
接受感知优化。在公开的 DSpark 损失设置之上，我们增加了一个与接受长度相关的损失项，因此草稿模型不仅针对 token 级和中间目标的对齐进行训练，还针对在目标模型验证下更长的已接受前缀进行训练。
图 9. 接受感知优化。
47% 的空闲率及其背后的机制
在 batch 1 下，Blackwell TP4 上首次运行的 DSpark 追踪（覆盖 239 次稳态解码迭代）远未达到 NEXTN 路径此前达到的状态。中位步进时间为 10.62 毫秒，其中 GPU 空闲 4.99 毫秒（占 47%）。
空闲时间并不在 CUDA 图内部：图内微间隙在 3 秒中总计约 80 毫秒。它全部位于图之间的 eager 段中，即每步有 4 到 5 个 100 微秒到 2 毫秒的中等间隙。
FlashInfer 的两个 plan() 实现，即 fa2 BatchPrefillWithPagedKVCacheWrapper 和 MLA 包装器，都被喂入了设备张量，而它们在内部会对 qo_indptr、kv_indptr 和 kv_len_arr 各执行一次阻塞式的 .to("cpu")。阻塞式 D2H 会等待流上所有在途操作完成，包括仍在执行的草稿图。因此每一步，CPU 在草稿启动后就被钉死在 GPU 进度上；约 1 毫秒的 cudaGraphLaunch CPU 开销没有任何 GPU 繁忙窗口可以隐藏；而调度器尾部则在这两者之后串行化。
从结构上看，这又是 resolve_seq_lens_cpu 的钉死问题：在宿主端对设备驻留值做阻塞式读取，每一步都将预跑距离重置为零，而且发生在一个不相关的子系统中。它给出的规则是：在 batch 1 场景下，优先在宿主路径上查找对设备值的阻塞式读取，因为每一次这样的读取都会把整个宿主循环从隐藏工作变成 GPU 气泡。
宿主端提供的 plan
那三个数组从来就不需要来自设备。DFLASH 系列保证 verify 和 draft ForwardBatch 携带 seq_lens_cpu = prefix + draft_token_num，并在三个独立的调用点做了断言，而这恰好等于设备端路径计算出的 kv 长度。宿主本来就已经知道它阻塞等待读回的那个答案了。
- fa2 侧。在捕获时，将 fast_prefill_plan 安装到按 batch 大小划分的目标 verify 包装器上，并以 DFLASH verify 输入类型作为门控条件，这样 EAGLE 的目标 verify 就不会被改动；同时加一个断言，确保不存在自定义 mask（fast plan 不支持自定义 mask；DFLASH 从来不会有）。
- MLA 侧。用纯宿主算术从 seq_lens_cpu 构建 plan kwargs，零 D2H 开销，通过新的 kv_indices_buf 参数将 kv_indices 直接写入包装器的 CUDA 图缓冲区，并调用 fast_mla_decode_plan(causal=True)，从而跳过三次阻塞式 D2H 和四次设备缓冲区刷新。捕获时仍然运行真正的 plan()，正是它填充了缓存模块和包装器的缓冲区。
另外两项修复无需额外工作。`graph.replay()` 始终是纯入队操作；除了横亘在中间的 D2H 之外，没有任何因素阻止 CPU 将草稿图 → 验证准备 → 验证图连续入队。移除 D2H 后，验证元数据准备和两次图启动都在草稿图仍在执行时完成入队，两张图在 GPU 上背靠背运行。
结果
在组合任何方案之前，每个环境标志都单独进行了 A/B 测试：
| 标志（单独测量） | 接受长度 | 平均 TPOT | 结论 |
|---|---|---|---|
| 无（重叠开启，radix 缓存关闭） | 4.49 | 1.48 ms | 干净的基线 |
| SGLANG_OPT_FUSED_KDA_VERIFY=1 | 4.68 | 1.34 ms | 安全，保留 |
在固定投机配置下，标志层面的轨迹，全部为 8192 输入 / 1024 输出且并发度为 1：同步调度 1.67 ms → 重叠调度且 radix 缓存关闭 1.48 ms（调度器尾部的空闲窗口从 1118 µs 缩小到 85 µs）→ 融合 KDA 验证 1.34 ms。
部署配置在并发度为 1 的情况下对 1000 个请求进行测量：接受长度 9.95，平均 TPOT 0.78 ms，中位 TPOT 0.51 ms，输出吞吐量 1120 tok/s，峰值 1945 tok/s。9.95 的接受长度是在 block_size 16 的 DSpark 草稿模型下测得的；发布的草稿检查点使用 block_size 8。DSpark 不接受 speculative-num-steps / draft-tokens 标志；这些仅适用于 NEXTN。
将中位 TPOT 乘以平均接受长度得到 0.51 × 9.95 ≈ 5.1 ms，接近之前跟踪测得的步骤时间（在 KDA 融合之前总计约 5.3 ms）。仅将其视为粗略的一致性检查：它混合了中位数和平均数，而且面对如此宽的分布，它并不是中位步骤时间。直接从跟踪中测量步骤时间才是收尾的正确方式，而我们尚未对 DSpark 配置进行该测量。跟踪所支持的是定性结论：主机再次退出了关键路径，剩下的都在 GPU 上。
时间现在花在哪里
图 10. 活动之后的一次 NEXTN 解码步骤。
MoE 分组 GEMM 耗时 1215 微秒，router / activation / glue 小核耗时 1127 微秒，all-reduce 耗时 951 微秒，稠密 GEMM 耗时 918 微秒，KDA 耗时 488 微秒，8k 上下文下的 MLA 注意力耗时 244 微秒，另有 400 微秒的残余空闲。主机侧已完全隐藏；剩下的都是 GPU 计算，而权重带宽在其中占主导地位。
这份统计是 NEXTN 配置下的结果；DSpark 会重新分配步骤（更宽的验证窗口、第二个草稿模型图），但不会改变结论。
8k 上下文下的 MLA 注意力耗时 244 微秒。长上下文在这里不是问题，这是混合架构带来的结果。MoE 加上稠密 GEMM 大约耗时 2.1 毫秒，其中几乎全部是权重带宽。
因此路线图很短：
- fp8 权重是剩下的大杠杆。在 2.1 毫秒的带宽受限计算中把字节数减半，能带来结构性的 15-20% 提升，这远远超出了该指标的噪声带。接受长度问题的补救方案已经存在（bf16 草稿模型），而块量化的 TP 约束也是已知的：当 moe_intermediate_size = 768、块大小为 128 时，TP4 不可行；需要 --ep-size 4。
- Router 融合已经达到了合理的终点。把 gate matvec 折叠进 top-k 核会把并行度从 129×M 个 CTA 压缩到 M/BLOCK_M。PDL 链式调用是该路径的正确停止点。
- 主机环境工程。对调度进程进行核心绑定和 GC 调优，这实际上是在保护预跑余量，而不是一种内核优化。
复现
SGLANG_ENABLE_METADATA_GLUE_GRAPH=1 \
SGLANG_OPT_FUSED_KDA_VERIFY=1 \
SGLANG_ENABLE_FUSED_VERIFY_EXTEND_GRAPH=1 \
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-3.0-flash \
--tp-size 4 --trust-remote-code \
--speculative-algorithm NEXTN \
--speculative-num-steps 5 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 6 \
--attention-backend trtllm_mla \
--flashinfer-allreduce-fusion-backend auto \
--mem-fraction-static 0.85
对于 DSpark 配置，把投机标志替换为 DSpark 草稿检查点：
SGLANG_OPT_FUSED_KDA_VERIFY=1 \
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-3.0-flash \
--tp-size 4 --trust-remote-code \
--speculative-algorithm DSPARK \
--speculative-draft-model-path inclusionAI/Ling-3.0-flash-dspark \
--attention-backend trtllm_mla \
--flashinfer-allreduce-fusion-backend auto \
--disable-radix-cache
两种配置都用相同的方式进行基准测试：
python3 -m sglang.bench_serving --backend sglang \
--dataset-name random --num-prompts 1000 \
--random-input-len 8192 --random-output-len 1024 \
--random-range-ratio 1.0 --max-concurrency 1
致谢
这项工作由 RadixArk SGLang 团队与蚂蚁灵枢基础设施团队合作完成。感谢 DeepInfra 和 Novita 在 SGLang 上提供 Ling-3.0-flash 服务。
蚂蚁集团灵枢基础设施团队（按姓氏字母顺序排列）：Tiwei Bie、Yuan Luo、Dayu Qiu、Jianfeng Tan、Tongli Wang、Yue Yu、Kaihong Zhang。蚂蚁集团 inclusionAI（按姓氏字母顺序排列）：Xiang Cao、Guoshan Lu、Junbo Zhao。
Batch-1 decode keeps getting more important. Xiaomi MiMo, for example, announced MiMo-V2.5-Pro UltraSpeed in June, claiming 1,000 tok/s decode on a one-trillion-parameter MoE model.
Batch 1 gives an inference stack no room to hide overhead. There is no batch to amortize launch cost across, no concurrency to fill pipeline bubbles, and not enough arithmetic intensity for clever tiling to pay off. Every microsecond on the critical path is a microsecond the user waits.
This post is about pushing that floor down for Ling-3.0-flash, a hybrid linear-attention MoE model, on 4 NVIDIA Blackwell GPUs. It covers two speculative decoding paths. On the NEXTN/MTP path we moved single-request decode from 288 tok/s to 606 tok/s and mean TPOT from 3.33 ms to 1.53 ms. The second path is DSpark, a confidence-scheduled speculative decoder built on the same stack: a 1000-request run reaches 1120 tok/s at a mean TPOT of 0.78 ms and an accept length of 9.95. That last comparison is the controlled one: NEXTN and DSpark were measured with the same command on the same machine, and mean TPOT is 1.9x lower (1.53 ms to 0.78 ms). The rest of the post is where that time was going and what it took to get it back.
Highlights
- Final result: mean TPOT down 54% (3.33 ms to 1.53 ms), single-request throughput 2.1x higher (288 to 606 tok/s). In the controlled 1000-request comparison, DSpark reached 0.78 ms mean TPOT and 1120 tok/s.
- The optimization line was host run-ahead, then PDL chaining, then kernel optimization, then DSpark. Removing a per-step host pin let preparation hide behind GPU work; PDL then linked the MoE, router, KDA, and all-reduce path; two fusions, one KDA retune, and bf16 router/lm_head GEMMs shortened the remaining GPU critical path.
- Numerics as a bandwidth knob: moving the router gate and lm_head from fp32 to bf16 was the largest post-structural change, worth roughly +10%.
- Measurement discipline throughout: profiled-vs-unprofiled calibration before any host-side conclusion, cold-weight microbenchmarks, and A/B decisions on mean TPOT rather than single-window peaks.
- DSpark raises the tokens committed per verify step: accept length 9.95 and 1120 tok/s at concurrency 1, at a mean TPOT of 0.78 ms. Against NEXTN on the same 1000-request benchmark, that is 1.9x lower mean TPOT.
Figure 1. Headline results across the four configurations.
| Metric (8192-in / 1024-out, single concurrency, greedy, TP4 bf16) | Baseline | After the draft-extend graph fix | NEXTN, tuned | With DSpark |
|---|---|---|---|---|
| Mean output throughput | 288 tok/s | 526 tok/s | 606 tok/s | 1120 tok/s |
| Mean TPOT | 3.33 ms | 1.76 ms | 1.53 ms | 0.78 ms |
| Median TPOT | — | — | 1.56 ms | 0.51 ms |
| Peak output throughput | — | — | 1099 tok/s | 1945 tok/s |
| Accept length | 3.14 | 3.13 | 3.25 | 9.95 |
On GSM8K, the same stack scored: accuracy 0.889, invalid 0.000, latency 341.5 s, output throughput 511.1 tok/s.
All runs use Ling-3.0-flash on 4 Blackwell GPUs, TP4, bf16, concurrency 1, greedy decoding, and the same fixed 8192-input / 1024-output random workload. From left to right, the columns show the initial NEXTN baseline, NEXTN after the draft-extend graph fix, final tuned NEXTN, and DSpark. The first two are shorter campaign checkpoints; the last two are the controlled comparison, each measured over the same 1000 requests on the same machine. Peak throughput is compared only between the last two runs because it is the maximum over fixed one-second windows.
Two definitions matter here, because together they explain why output throughput is not simply the reciprocal of mean TPOT even at concurrency 1: SGLang's TPOT excludes TTFT, while output throughput divides total output tokens by total benchmark wall time (see the bench_serving guide). All headline benchmark runs in this post use a synthetic random workload; accept length in particular depends on the prompt and output distribution, so 9.95 is this workload's accept length rather than the model's.
The model
Figure 2. Ling-3.0-Flash architecture: 42 layers interleaving 35 KDA linear-attention layers with 7 MLA full-attention layers over a 512-expert MoE.
Ling-3.0-flash is a hybrid-attention MoE model (BailingMoeV3), and most of what follows comes from that word hybrid.
| Layers | 42 total: 35 KDA linear-attention + 7 MLA full-attention |
|---|---|
| MoE | 512 routed experts + 1 shared, top-8 (+1), moe_intermediate_size 768 |
| Hidden size | 2560 |
| Vocabulary | ~157k, served through a vocab-parallel lm_head |
| Weights | ~63 GB per rank in bf16 |
| Deployment | 4 NVIDIA Blackwell GPUs, TP4, bf16, NEXTN speculative decoding |
Five of every six attention layers are KDA. That is why MLA attention costs only 244 µs per step at 8k context in the final profile, and why this model is a good batch-1 target in the first place: with attention cheap and the batch tiny, what remains on the critical path is weight bandwidth and launch latency, which is exactly the regime this post is about.
The shape of a batch-1 step
We decode with NEXTN speculative decoding at steps=5, topk=1, draft_tokens=6. One decode step is three CUDA graphs in a relay.
Figure 3. Three graphs per step. The draft model proposes a 6-token chain, the target model scores all six in a single forward, and the extend graph replays the accepted prefix with the target's real hidden states to produce the next round's seed. The verdict itself (eagle_sample) happens inside the verify graph; the host learns how many tokens were accepted one step late.
The draft is a single-layer NEXTN model run autoregressively: five steps but only four forwards, because the first candidate comes from the previous round's seed and the fifth is read off the fourth forward's top-k. Verify is one forward of the full 42-layer target over all six chain positions. Extend fixes up the draft's KV cache, which only ever saw the draft's own guesses, and hands back the seed for the next round.
What crosses between the three graphs on the CPU is nothing. Fixed shapes plus padding make every accept-dependent count a GPU index rather than a host value; persistent buffers let producer graphs write straight into consumer buffers; and the decisions that genuinely need values on the CPU (EOS, stop strings, detokenization) go through a side-stream D2H and a copy_done event consumed one step late. Everything below rests on that property.
Two kinds of empty time
When we started, the GPU was busy about two-thirds of the step. Idle time at batch 1 comes in two flavors, and they need separate diagnoses because the fixes have nothing in common:
- Host-mode idle. Three graph replays per step, several hundred kernel nodes executing inside them, and Python glue in the seams between graphs. (Three replays, not three forwards: the draft graph's captured body holds all four draft forwards, so the autoregressive draft loop costs one replay rather than four.) If the host's per-step loop takes longer than the GPU's step, the GPU starves. The fix is to hide and shrink host work.
- GPU-mode idle and GPU-mode cost. Once the host is hidden, what remains is weight bandwidth (each MoE layer cold-reads roughly 94 MB of activated expert weights per step) plus the intrinsic latency floor of several hundred small kernel nodes. A batch of one amortizes neither. The fix is dtype work, fusion, and launch-dependency scheduling.
Figure 4. Two shapes of idle. Top: the host loop is longer than the GPU's work, so the holes are few and wide and land in the seams between graphs. Bottom: once the host is hidden, what remains is several hundred 1.5-6 µs kernel nodes whose launch floor rivals their arithmetic, plus the weight read itself.
These two kinds of idle describe the step-time side of TPOT. The other lever is how many tokens each step commits: mean TPOT ≈ step time / mean accept length. The rest of the post follows those levers. Host run-ahead and seam work remove host-mode idle; PDL, dtype changes, fusion, and retuning shorten the GPU critical path; speculation tuning and DSpark increase the tokens committed per target step. DSpark later revisits the first category when a blocking D2H read reintroduces a host pin.
Fixing the ruler before fixing the machine
Three properties of the measurement setup shape every number below.
The profiler inflates host-side events. CUPTI adds overhead to each host event it records. On the same configuration, a profiled step measures 5.2 ms while the real step, back-computed from TPOT × accept length on an unprofiled run, is 4.9 ms. That 0.3 ms gap is the same order as the host-side effects we wanted to reason about, so a profiled trace can show cross-rank waits that do not exist off the profiler. GPU kernel durations come from hardware timestamps and are more trustworthy than host-side timings, but not immune: tracing still perturbs launch timing, concurrency, cache state, and CUDA graph execution, and Nsight Systems documents potentially significant overhead for CUDA and graph-node tracing (user guide). So every host-side conclusion here got a profiled-vs-unprofiled calibration first.
Microbenchmarks run optimistic for cold-weight kernels. A loop calling one kernel repeatedly keeps its 2.6 MB gate weight resident in L2, while the real model flushes L2 with about 94 MB of expert traffic between consecutive calls to the same layer. Hot 7 µs, cold 11 µs: enough to reverse a ranking against the library GEMV.
Peak throughput is a single-window statistic. The benchmark's peak number is the maximum over a fixed 1-second grid, so it carries roughly a ±5% phase band: shifted TTFT/TPOT re-slices the grid, and a change that improves mean throughput by 2.3% can print as a drop from 909 to 858. Both readings reproduce exactly under a fixed seed, so reproducibility does not separate signal from phase. A/B decisions here are made on mean TPOT × mean accept length. That product is a derived estimate of step time rather than a measured one (the product of two aggregates is not the aggregate of the product), but it is stable across runs and insensitive to accept-length drift in these runs, which is what an A/B criterion needs. We report peak but never optimized against it.
Correctness had its own gate, applied to every change before it stayed: byte-exact comparison of a 256-token greedy generation, accept length unchanged within 0.05, and a greedy re-run after interleaving temperature-sampled requests to catch state pollution. Changes that legitimately alter rounding (the bf16 gate, the single-rounding combine) said so in their commit message and were validated on accept and task metrics instead of bit parity.
Letting the host run ahead
This is the structural change the rest of the campaign rests on, and it is a host-mode idle fix.
Figure 5. Lockstep to deep pipelining. Before: every step the host blocks in resolve_seq_lens_cpu waiting for the previous verify graph to finish on the GPU, so run-ahead resets to zero and each host prep segment becomes a GPU bubble. After: the queue is a full step deep, the launch of verify k+1 leads its own execution by an entire step, and the only remaining synchronization is a copy_done event consumed one step late.
cudaGraphLaunch has always been asynchronous, and the draft → verify → extend ordering on the GPU is free: same stream, FIFO. So the question was never whether verify waits for draft. It was whether the host is pinned to GPU progress every step.
It was. Under spec-v2, the scheduler does not know the accept length, so FutureMap.resolve_seq_lens_cpu() pulls new_seq_lens back from the GPU while building the next batch: gated on a publish event, copied on a private stream, then synchronize()d. The host was not waiting for a microsecond-scale copy. It was waiting for the previous verify graph to finish executing. Median cost: 485 µs per step, with the run-ahead depth reset to zero every single step.
The cause is the needs_cpu_seq_lens flag, OR-ed across every backend involved in spec-v2. trtllm_mla declares False in all three roles; the sibling linear-attention backends GDNAttnBackend and Mamba2AttnBackend both declare False explicitly. KDAAttnBackend never declared it and inherited the base-class default of True, even though it runs the same base-class metadata code as its two siblings.
Declaring needs_cpu_seq_lens = False collapsed the OR and removed the per-step synchronize. The correctness argument is pointwise: KDA's metadata never reads the CPU mirror, and replay padding comes from forward_batch.num_padding.
How does the host dare launch step k+1 without knowing what step k accepted? Because the values never touch the CPU. FutureMap is a GPU-resident relay: step k's graph writes output tokens, new_seq_lens, top-k probabilities, and hidden states into device buffers indexed by req_pool_idx, and step k+1's graph reads them by the same index. The host only handles indices, which it already knows.
Figure 6. Where the slack lives. Panel A: the host loop (~4.3 ms) fits under the GPU step (~4.9 ms), so it is fully hidden. Panel B: when jitter (a gloo broadcast or a GC pause) exceeds the slack, the host finishes late and the GPU waits at the next verify boundary, where the first collective in the graph absorbs the cross-rank skew.
Run-ahead also changes the shape of host cost. Instead of every rank paying its host time directly every step, only a rank that exhausts its queue slack pays. In one four-rank trace, exactly one rank was in that state: its scheduler segment ran 5-10x longer than its siblings, its draft graph launched 40-80 µs late, its draft→verify seam ran +165 µs above the others' median, and it showed periodic 400-750 µs spikes with a GC signature. The other three ranks spin-waited for it at every rendezvous. The diagnostic that generalizes: a kernel's duration is not its work. A 20 KB embedding all-reduce showing 150-480 µs is not a slow all-reduce; it is absorbing skew, and only cross-rank time alignment tells you which rank is late.
Closing the seams
With the lockstep pin gone, the seams between graphs became worth shrinking. Before a CUDA graph replays, step-specific attention metadata (kv indices, block tables, mamba state slots) has to be rebuilt from the live req_to_token and seq_lens into the graph's captured static buffers. That refill runs eagerly every step and is most of what a seam contains. At batch 1 it is purely host-bound: each op costs 5-15 µs to dispatch and 1-4 µs to execute.
We attacked it at two levels. First, fuse the index chains: assign_extend_cache_locs_uniform computes end offsets inside the kernel (the uniform draft_token_num expansion makes the cross-row prefix sum unnecessary), and _fused_state_indices_kernel collapses a gather, a translate, a padding-sentinel write, and a copy_ into one launch, carefully preserving both side effects, including zeroing req_pool_indices on padded rows, which nothing in that function needs but other captured kernels in the graph depend on for in-bounds gathers.
Second, capture the refill itself into a small CUDA graph keyed by (bs, forward_mode). This works because of a pointer-stability property the replay contract already guarantees: the replay ForwardBatch view hands the backend only runner-static buffers and pool-resident tensors, so the whole prep sequence has fixed addresses. Four safety mechanisms surround it: two eager warmups, so Triton JIT and autotune happen outside capture; a snapshot of each backend's forward_metadata object restored before every replay (the graph replays device ops, the snapshot restores Python pointers); permanent eager fallback with a warning if capture fails; and guards against padding, TBO, pdmux, and LoRA. It ships opt-in behind SGLANG_ENABLE_METADATA_GLUE_GRAPH and is force-disabled for DFLASH-family speculation, because that path rebuilds its attention plan on the host every step and capturing the refill would freeze the plan at capture time.
There is a hard boundary on what may be captured. The criterion: a refill made of pure device kernels writing persistent buffers is capturable; anything that goes through a FlashInfer-style plan() is not. The draft side fails it: the multi-step draft backend re-plan()s wrappers that the main EAGLE graph has already captured, and recording that re-plan into a secondary graph corrupts the wrappers' internal state on replay. A related requirement is that capture be idempotent. trtllm_mla's _init_cuda_graph_metadata used to allocate fresh tensors and replace its decode_cuda_graph_metadata[bs] entry on every call, which leaves earlier graphs reading freed memory after a second capture.
PDL: stacking the latency floors of small kernels
A batch-1 step executes several hundred kernel nodes in a short window. At that size, launch and prologue cost about as much as the math. Programmatic Dependent Launch (PDL) lets a consumer kernel be scheduled onto SMs while its producer is still running: the consumer executes everything that does not depend on the producer's output and fences at gdc_wait() only before the dependent read.
Figure 7. PDL on the router path. Without PDL, each kernel starts only after the previous one fully retires, and the gate matvec's cold-HBM weight load sits on the critical path. With PDL, the weight tile load is producer-independent, so it is issued before gdc_wait() and a 2.6 MB cold read flies under the producer's tail; the router top-k prefetches its bias the same way.
We wired three chains: the MoE main chain (moe_align → up-GEMM → activation → down-GEMM → combine → all-reduce), the router chain (norm → gate matvec → top-k), and the KDA chain (conv1d_update → recurrent delta-rule → gated norm). Two design points matter.
Producer-independent loads go before the wait. That is the whole trick in the figure, and it is what makes PDL more than launch-overhead removal for latency-bound kernels.
Inductor kernels cannot carry PDL attributes. The small-M MoE combine was a torch.compile-generated kernel; joining the chain meant swapping it for the repo's Triton reduction plus GDC. That had a numerical side effect: fp32 sum × scale with a single final cast, where the old path rounded twice. The result is slightly more accurate but not bit-equal, which the commit message declares.
Later we upgraded the semantics after a finding in PTX griddepcontrol: launch_dependents only releases the launch of dependents, while a consumer's wait always fences on the producer grid's complete retirement. Moving the trigger from the end of the producer to immediately after the producer's own wait lets a consumer's prologue overlap more of the producer's body than just its tail, subject to one precondition: the consumer must still keep its own gdc_wait() between the early launch and every read of producer output. That is a property of each consumer, not a blanket guarantee, so we checked it kernel by kernel and converted six. What the early trigger buys is also not deterministic: the driver may launch a dependent grid early, and how much overlap materializes depends on scheduling and resource pressure at the time (CUDA programming guide). fused_moe gates this behind an M ≤ 512 check: at prefill shapes, releasing a large consumer grid early steals SMs from the producer, while at decode shapes it is pure gain.
PDL is pure scheduling semantics. Changes whose accumulation order is unchanged stay bitwise identical; the gate matvec passed a 4-of-4 GDC-on/off bit comparison.
Two fusions and a retune
moe_align, on the pair axis. The Triton fused-MoE GEMM consumes tokens in block_size tiles where every row shares an expert, and moe_align_block_size builds that permutation. The generic path needs two kernel launches: no token can be placed until every expert's offset is final, those offsets come out of a grid-wide scan, and a device-wide barrier exists only at a kernel boundary. A single-launch variant exists, but it stages per-thread expert counters in shared memory, so it is limited to 64 experts or fewer; a 513-expert decode always paid two launches.
The replacement works on the pair axis: an [NP, NP] pairwise comparison gives every (token, slot) pair a stable rank within its bucket and its bucket population in one shot; the rank-0 representative of each bucket then derives padded counts, bucket-ordered exclusive offsets, the published total, and per-block expert ids. Nothing scales with the expert count, so the expert-count limit disappears. The obvious alternative, histogram and cumsum over the padded expert axis (up to 1024 buckets), is correct but puts about 3x more single-SM work on the critical path than the two kernels it replaces. That is what makes the pair axis load-bearing here.
Two deliberate deviations from the reference, both argued from consumer invariants: intra-bucket order is stable in pair index rather than atomic-scheduling order (every pair writes its own output row, so consumers are order-invariant), and the buffer tail beyond the published total is left unwritten (consumer CTAs early-exit before reading it). One cliff: the pairwise tensors are O(NP²). They live entirely in registers at NP=64 (about 4 µs, on par with the CUDA two-kernel path) and spill to local memory at NP=256, costing about 230 µs per launch. The dispatch gate is a hard numel ≤ 64; larger batches fall back to the CUDA path.
SwiGLU in the up-GEMM epilogue. Folding silu(gate) * up into the MoE up-GEMM epilogue removes a standalone activation kernel per MoE layer and the whole write-then-read of the intermediate buffer. The layout trick is a per-expert row interleave of w13 applied at weight load, which makes gate and up land in adjacent even/odd columns of the same output tile. Since each GEMM output column is an independent dot product, interleaving is bitwise neutral.
Bit parity is where the care went. The kernel being replaced is compiled with -use_fast_math, so the epilogue reproduces it instruction by instruction: mul + ex2.approx.ftz for __expf, div.approx.ftz, and a single final rounding on the product. The subtle part: FlashInfer instantiates the activation functor in float, so silu never lands in bf16 before the multiply. Round it there and the result double-rounds and diverges on a large fraction of inputs. That is invisible in the documentation and invisible to a tolerance check; it takes an element-wise bit comparison over the full input range.
KDA chain-verify tile economics. The fused conv1d + gating delta-rule verify kernel already existed; these commits retuned it. On the rotating-cold test with in-graph timing, the Blackwell curve at T=6 is monotone: BV=4 at 11.56 µs, 8 at 12.53, 16 at 12.83, 32 at 14.26, 64 at 20.7, 128 at 38. BV=4 wins by up to 19% per call because 256 CTAs is 1.7 waves over 148 SMs, and duplicating the q/k convolution 32 times is still cheaper than shortening the serial chain. Tiling the V dimension never touches the K-dimension reduction order, so at num_warps=4 every BV is bitwise identical to the baseline and the retune carries no numerical risk.
bf16 router gate and lm_head
The largest single post-structural change was a dtype change. At batch 1, the router gate and the lm_head are pure bandwidth: every decode step cold-reads each MoE layer's gate weight (2.6 MB in bf16) and the vocab-parallel lm_head projection, and neither has arithmetic to hide the read behind. Running both in bf16 instead of fp32 halves those bytes; end to end it was worth roughly +10%, the largest gain of any single change after the host run-ahead fix. Like the other rounding changes above, this one was declared in its commit message and validated on accept length and task metrics rather than bit parity.
KDA under speculation
A rejected speculative token leaves a KV cache entry harmlessly stale, but it has already corrupted a recurrent state in place. Linear attention and speculation do not coexist for free.
The scheme that makes it work: during verify, the recurrence runs with state updates disabled and writes each chain position's post-state into an intermediate buffer; after the verdict, commit_mamba_states_after_verify copies the state belonging to the last accepted position into the persistent slot. Stage, then commit. This is also why the compact spec cache is restricted to topk=1: with a chain, the accepted prefix is unique and states can be indexed by position; with a tree, the accepted path is one of many and states would have to be indexed by tree path.
Profiling shows KDA decode is bandwidth-bound, mostly HBM traffic on the K×V state, so beyond the fusion and the tile retune there is not much left there. We did not measure achieved bandwidth against the Blackwell peak, so read that as a shape observation rather than a roofline result.
The economics of speculation at batch 1
Weight-bandwidth dominance has a counter-intuitive corollary: verifying more tokens is nearly free. Verifying 4 tokens and verifying 6 tokens read exactly the same weights. Deepening speculation at batch 1 costs one extra cheap draft forward per added step (the draft is a single layer) plus the incremental serial cost in the KDA chain-verify recurrence, and buys accept length.
We swept it rather than assuming it (this sweep predates the fusion bundle; the optimum moved afterwards, as noted below):
| steps / draft tokens | accept length | mean TPOT | step time (derived) | steady-state tok/s |
|---|---|---|---|---|
| 3 / 4 | 3.11 | 1.51 ms | 4.70 ms | 662 |
| 4 / 5 | 3.37 | 1.45 ms | 4.89 ms | 690 |
| 5 / 6 | 3.45 | 1.55 ms | 5.35 ms | 645 |
The step-time column is TPOT × accept length, a derived estimate rather than a direct measurement. Each added step costs about 4-9% of step time while the marginal accept gain decays geometrically (d5 → d6 adds only 0.08). The break-even condition is roughly Δaccept > 0.05 × accept. The optimum also moves: after the fusion bundle landed and step time dropped, (5, 6) became the better configuration; once fp8 weights shrink the fixed base further, it will need another sweep.
DSpark: high-quality block drafting
Tuning NEXTN's depth is a one-dimensional knob on a fixed-shape algorithm. The larger lever is changing the algorithm, and the second half of the campaign went into bringing DSpark onto the same target and giving it the same batch-1 treatment.
What DSpark does differently
The DSpark algorithm itself is public. The work here is adapting that public recipe to Ling-3.0-flash, long-context online distillation, and the batch-1 Blackwell serving stack. Our adaptation differs in four ways.
Distribution-aligned data. We distill mainly on Ling-3.0-flash post-training data, so the draft trains on the distribution it will face at serving time. We also use multiple sampling settings during distillation to improve trajectory diversity and robustness under speculative decoding.
An ablation-driven draft design. Instead of directly inheriting the Ling-3.0-flash architecture, we ran systematic ablations over key draft choices, including whether to reuse the Ling-3 attention structure and which RoPE variant to use (partial or interleaved). We kept the design with the best acceptance-length/latency tradeoff.
A serving-coupled online training system. For long-context and large-scale online training we built SplitServe Trainer, a single-node 8-GPU framework that splits resources evenly between training and SGLang inference. During training, the inference side runs target forwards to produce supervision signals such as target hidden states for the draft. This keeps the generation-training loop local, cuts IO overhead, and improves training efficiency for long-context workloads.
Figure 8. SplitServe Trainer layout.
Acceptance-aware optimization. On top of the public DSpark loss setup, we added an acceptance-length-related loss, so the draft is trained not only for token-level and intermediate-target alignment but also for longer accepted prefixes under target verification.
Figure 9. Acceptance-aware optimization.
47% idle, and the mechanism behind it
The first DSpark trace on Blackwell TP4 at batch 1, over 239 steady-state decode iterations, was nowhere near the state the NEXTN path had reached. Median step time was 10.62 ms with 4.99 ms of GPU idle (47%).
The idle was not inside the CUDA graphs: in-graph micro-gaps totaled about 80 ms out of 3 seconds. It was all in the eager segments between graphs, as four or five medium gaps of 100 µs to 2 ms per step.
Both FlashInfer plan() implementations, the fa2 BatchPrefillWithPagedKVCacheWrapper and the MLA wrapper, were being fed device tensors, and internally they do a blocking .to("cpu") on each of qo_indptr, kv_indptr, and kv_len_arr. A blocking D2H waits for everything in flight on the stream, including the draft graph that is still executing. So every step, the CPU was pinned to GPU progress right after the draft launch; the roughly 1 ms of cudaGraphLaunch CPU cost had no GPU-busy window left to hide in; and the scheduler tail serialized behind both.
Structurally this is the resolve_seq_lens_cpu pin again: a host-side blocking read of a device-resident value that resets run-ahead to zero every step, in an unrelated subsystem. The rule it suggests: at batch 1, look for blocking reads of device values on the host path before anything else, because each one converts the entire host loop from hidden work into a GPU bubble.
Host-fed plans
Those three arrays never needed to come from the device. The DFLASH family guarantees that the verify and draft ForwardBatch carry seq_lens_cpu = prefix + draft_token_num, asserted at three independent call sites, and that equals exactly the kv length the device-side path computes. The host already knew the answer it was stalling to read back.
- fa2 side. Install
fast_prefill_planon the per-batch-size target-verify wrappers at capture time, gated on the DFLASH verify input type so EAGLE's target-verify is untouched, plus an assertion that no custom mask is present (fast plans do not support one; DFLASH never has one). - MLA side. Build the plan kwargs from
seq_lens_cpuin pure host arithmetic with zero D2H, writekv_indicesdirectly into the wrapper's CUDA-graph buffer through a newkv_indices_bufparameter, and callfast_mla_decode_plan(causal=True), skipping three blocking D2Hs and four device buffer refreshes. Capture still runs the realplan(), which is what populates the cached module and the wrapper's buffers.
The other two fixes required no extra work. graph.replay() was always a pure enqueue; nothing prevented the CPU from queueing draft graph → verify prep → verify graph back to back except the D2H standing in the middle. With it removed, verify metadata prep and both graph launches are enqueued while the draft graph is still executing, and the two graphs run back to back on the GPU.
Results
Each environment flag was A/B'd on its own before anything was combined:
| Flag (measured individually) | Accept length | Mean TPOT | Verdict |
|---|---|---|---|
| none (overlap on, radix cache off) | 4.49 | 1.48 ms | clean baseline |
SGLANG_OPT_FUSED_KDA_VERIFY=1 | 4.68 | 1.34 ms | safe, kept |
The flag-level trajectory, at a fixed speculation configuration, all at 8192-in / 1024-out and concurrency 1: synchronous scheduling 1.67 ms → overlap scheduling with the radix cache off 1.48 ms (the scheduler tail's idle window collapsed from 1118 µs to 85 µs) → fused KDA verify 1.34 ms.
The deployed configuration, measured over 1000 requests at concurrency 1: accept 9.95, mean TPOT 0.78 ms, median TPOT 0.51 ms, 1120 tok/s output throughput, 1945 tok/s peak. The 9.95 accept length was measured with a block_size 16 DSpark draft; the released draft checkpoint uses block_size 8. DSpark takes no speculative-num-steps / draft-tokens flags; those are NEXTN-only.
Multiplying median TPOT by mean accept length gives 0.51 × 9.95 ≈ 5.1 ms, close to the step time the earlier trace measured (about 5.3 ms in total, before the KDA fusion). Read that only as a rough consistency check: it mixes a median with a mean, and against a distribution this wide it is not the median step time. Measuring step time directly from the trace is the way to close it, and we have not done that for the DSpark configuration. What the trace does support is the qualitative conclusion: the host is out of the way again, and what remains is on the GPU.
Where the time goes now
Figure 10. One NEXTN decode step after the campaign.
MoE grouped GEMM 1215 µs, router / activation / glue small kernels 1127 µs, all-reduce 951 µs, dense GEMM 918 µs, KDA 488 µs, MLA attention at 8k context 244 µs, plus 400 µs of residual idle. The host is fully hidden; what remains is GPU work, and weight bandwidth dominates it.
This census is the NEXTN configuration; DSpark redistributes the step (a wider verify window, a second draft-model graph) but not the conclusion.
MLA attention at 8k context is 244 µs. Long context is not the problem here, a consequence of the hybrid architecture. MoE plus dense GEMM is about 2.1 ms, nearly all of it weight bandwidth.
So the roadmap is short:
- fp8 weights are the remaining large lever. Halving the bytes on 2.1 ms of bandwidth-bound work is a structural 15-20%, far outside the metric's noise band. The accept-length remedy already exists (bf16 draft), and the block-quantization TP constraint is known: with
moe_intermediate_size = 768and block 128, TP4 is not feasible; it requires--ep-size 4. - Router fusion has reached its sensible endpoint. Folding the gate matvec into the top-k kernel would collapse parallelism from 129×M CTAs to M/BLOCK_M. PDL chaining is the right stopping point for that path.
- Host environment engineering. Core pinning and GC tuning for the scheduler processes, which is really a way of defending run-ahead slack rather than a kernel optimization.
Reproducing
SGLANG_ENABLE_METADATA_GLUE_GRAPH=1 \
SGLANG_OPT_FUSED_KDA_VERIFY=1 \
SGLANG_ENABLE_FUSED_VERIFY_EXTEND_GRAPH=1 \
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-3.0-flash \
--tp-size 4 --trust-remote-code \
--speculative-algorithm NEXTN \
--speculative-num-steps 5 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 6 \
--attention-backend trtllm_mla \
--flashinfer-allreduce-fusion-backend auto \
--mem-fraction-static 0.85
For the DSpark configuration, swap the speculation flags for a DSpark draft checkpoint:
SGLANG_OPT_FUSED_KDA_VERIFY=1 \
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-3.0-flash \
--tp-size 4 --trust-remote-code \
--speculative-algorithm DSPARK \
--speculative-draft-model-path inclusionAI/Ling-3.0-flash-dspark \
--attention-backend trtllm_mla \
--flashinfer-allreduce-fusion-backend auto \
--disable-radix-cache
Either configuration is benchmarked the same way:
python3 -m sglang.bench_serving --backend sglang \
--dataset-name random --num-prompts 1000 \
--random-input-len 8192 --random-output-len 1024 \
--random-range-ratio 1.0 --max-concurrency 1
Acknowledgments
This work was a collaboration between the RadixArk SGLang Team and the Ant Ling Infra Team. Thanks to DeepInfra and Novita for serving Ling-3.0-flash on SGLang.
Ant Ling Infra Team, Ant Group (sorted alphabetically by last name): Tiwei Bie, Yuan Luo, Dayu Qiu, Jianfeng Tan, Tongli Wang, Yue Yu, Kaihong Zhang. inclusionAI, Ant Group (sorted alphabetically by last name): Xiang Cao, Guoshan Lu, Junbo Zhao.