推测解码通过增加计算量来减少解码步数，但这种权衡会随着负载增加而恶化：在批次大小为 B、推测 token 数为 K 的情况下，目标模型每步需验证 B * K 个 token，超过某个临界点后，其成本将超过收益。DSpark 从两端入手——采用半自回归块级草稿模型（每次草稿前向生成整个块，因此接受率保持较高水平），以及基于草稿模型自身置信度的、每个请求可变的验证长度，从而停止验证那些工作负载不太可能接受的 token。该算法及其收益均来自 DSpark 论文。
SGLang 现已支持在稠密模型和稀疏模型（例如 Qwen3 和 DeepSeek-V4）上运行 DSpark。本文介绍的是集成工作（sgl-project/sglang#30261）。我们在一个开源推理引擎上复现了论文中收益的形态——即单用户加速比，以及验证预算随负载增加而缩减的趋势——并描述了将这种调度策略转化为实际运行时间所需的工程实现：在参差不齐、按请求独立验证的场景下使用完整的 CUDA 图（因此修剪后的批次实际重放的是一个更小的图，而非填充后的图）；一种具有重叠感知能力的推测路径，可将调度器隐藏在前向计算之后；一个成本表分析器，使调度器能够在线调整每个请求的验证预算；以及用于观测接受率上限的可观测性，否则修剪操作会掩盖这一上限。由于硬件、推理引擎和流量模式均与论文不同，我们复现的是其机制和曲线趋势，而非精确到个位数的数值；下文所有“更快”的表述，均是与我们自己的对照组（除推测配置外完全相同）对比得出的。
相较于 MTP 和无推测解码的加速效果
图 1. 总吞吐量（y 轴）与单用户解码速度（x 轴）的关系；每条曲线对应从批次大小 1 到 256 的并发度扫描，每种方案一条曲线。越靠右上方表示性能越好。
在整个并发度扫描范围内，DSpark 实现了最佳的吞吐量/延迟权衡，在图 1 示例中明显优于 MTP 和无推测解码的基线。三种方案均使用 DeepSeek-V4-Flash 在 H200 上运行，采用 DP-attention 分布在四个 rank 上，除推测配置外完全相同——包括无推测解码基线、MTP（EAGLE 风格基线，取 1-1-2 和 3-1-4 配置中每个批次大小的最佳结果）以及 DSpark。
在 SGLang 中采用 DSpark
DSpark 算法源自该论文，其实现位于三个草稿侧组件中：
- 块草稿器——一条密集线（例如 Qwen3）和一条稀疏线（例如 DeepSeek-V4）；一次前向传播生成一个 gamma token 块，通过一个轻量级顺序头（马尔可夫或 RNN）使每一步都依赖前一个 token，因此该块是半自回归的。
- 置信度头——为每个草稿 token 评分，评估其通过验证的概率；整个块的乘积即为该块的存活概率。
- 顺序温度缩放（STS）——对这些分数进行校准，使得存活率能够真实反映调度器所预算的接受率。
围绕这些核心组件，SGLang 增加了服务支持层：
- 置信度调度器——在每一步将每个块的存活概率转换为每个请求的验证预算。
- 单请求非对齐验证——在一个批次内，每个请求的验证长度可变（静态/紧凑/上限接受）。
- 完整 CUDA 图——针对非对齐、可变长度的验证过程进行捕获。
- 可观测性——修剪操作下的接受上限及其他指标。
- 增量 SPS 成本表——一个离线分析的步时模型，由调度器在线读取。
- 数据并行注意力——与其他并行维度一同支持。
- 零开销调度——集成到 SGLang 的重叠调度器中，几乎无需为 DSpark 添加特殊处理逻辑。
- 性能优化——融合的 Triton 内核以及分片的块草稿器矩阵乘法。
验证模式
三种验证模式是本文后续讨论的核心轴。静态模式在每一步验证整个草稿块（基线方案）。紧凑模式仅验证调度器选定的每个请求窗口——即生产路径。上限接受模式验证整个块，但仅提交到该窗口为止：输出结果与紧凑模式相同，同时能揭示完整验证本应接受的内容——这是我们衡量修剪操作下接受上限的方式。
在完整 CUDA 图下的非对齐验证
按请求划分的窗口无法适配固定形状的 CUDA 图：在一个批次中，一个请求验证两个 token，另一个请求验证六个 token，此时不存在单一的查询长度，而将所有请求填充至完整块宽度只会把裁剪部分重新填充回来。因此，我们保持批次的不规整性，并根据总 token 数量对图进行键控——将变长请求前端打包到一个紧凑缓冲区中，并向上取整到最近的已捕获层级。当预算缩减时，打包后的总 token 数会降至更小的层级，DSpark 会重放一个真正更轻量的图（更少的注意力与 MLP 行，而非掩码后的全宽度前向传播）；在 DP 注意力机制下，各 rank 共享同一个层级（即任一 rank 所需的最大层级），并同步降级。
打包后的缓冲区是一种 cu_seqlens 风格的变长输入，因此紧凑验证复用了后端已有的注意力核——在 DeepSeek-V4 上，使用的是模型自身的稀疏 MLA 路径（flash_mla），无需新核；每个受支持的后端只需在图重放时根据打包布局重建其变长元数据。
图 2. 将每个请求验证长度可变的批次适配到已捕获的 CUDA 图中。固定形状的图会将每个请求填充至完整块宽度（N x W）；不规整路径则将已调度的 token 前端打包，仅将总 token 数向上取整至最近的已捕获层级，从而为相同的已接受 token 计算远更少的填充单元。
可观测性
裁剪机制限制了上限：紧凑模式仅验证一个块的前几个位置——即调度器的窗口——因此，完整块验证在该步骤本应接受多少个 token 这一信息永远无法被观测到；而缺少这一信息，就无法区分良好的裁剪与有损的裁剪。上限接受运行可以恢复这一信息：它验证完整块，但仅提交至窗口范围，因此它提交的内容与紧凑模式完全相同，同时暴露了上限。我们还提供每个请求的置信度与校准指标（例如 ECE），用于事后分析。
估算裁剪下的上限
一种专为生产运行或其他不希望额外伴随运行的场景设计的块接受估计器，可直接在紧凑型运行内部恢复估计的截断上限。该估计器利用未来步骤中目标 token 及其对数概率，并假设修剪轨迹与未修剪轨迹中锚定 token 的属性相似性，从而计算反事实尾部的估计区间。
动态调度与固定调度的初步对比
置信度调度器是首个基础版本，我们将其视为一个端到端机制可行的证明，而非高度调优的结果。我们在两个接受率不同的示例工作负载上，将紧凑型（每步 SPS-argmax 预算）与无修剪型（通过相同不规则路径运行的静态全块调度）进行了对比。
图 3. 紧凑型（动态修剪）与无修剪型（全块）对比，在 DP4 下批次从 1 到 256，基于两个接受率不同的示例。越靠右上方表示效果越好。
动态预算的优势主要体现在大批次场景。在批次大小为 1 时，目标验证不会因 token 增多而显著变慢，因此修剪节省的时间很少，两种方案表现相当。随着并发度增加、吞吐量开始趋于平稳，修剪缩短了步骤时间，紧凑型方案开始领先。在接受率较低的示例上，差距更大且更早出现——接受率越低意味着需要修剪的尾部越长，这与成本模型的预测完全一致。
每个面板都是一个干净的紧凑型与无修剪型 A/B 对比（面板内设置完全相同），但两个示例并非严格意义上的单变量对比：除了接受率之外，它们在设置上也有细微差异（提示词格式和每轮运行次数），因此我们观察的是跨面板的趋势，而非面板间的绝对数值。
这些预算的有效性也取决于其背后的成本表。我们当前的 SPS（及校准）拟合是初步近似，可能尚未完全考虑步骤成本随上下文长度变化的情况——因此调度器最终确定的精确工作点很可能还有改进空间，我们在此展示的是机制本身，而非调优后的具体数值。
混合流量下的逐请求差异化处理
同质化扫描掩盖了置信度调度的真正意义。如果批次中的两个请求中一个的可预测性远高于另一个，它们就不应该获得相同的验证窗口。混合流量才是这一机制真正发挥作用的地方。
图4. 按工作负载划分的预算（左）与每步验证长度分布（右）。
举例来说，我们按接受难度混合了三种工作负载：gsm8k（高）、arena-hard（中）和poetry（低）。窗口随难度收缩——分别为5.24、3.78、2.91个token——而对上限（即区块未经修剪时能接受的长度）的利用率仍然很高（0.88–0.97）。调度器是在为每个请求单独确定大小，而不是应用一个批次平均值。右侧面板逐步展示了这一过程：约55%的gsm8k步骤填满了完整的六个窗口，而约80%的poetry步骤只使用了三个或更少的窗口。
性能优化与零开销调度（ZOS）
两种工程手段将调度转化为实际时间：降低每一步的成本，以及将调度器隐藏在前向传播之后。两者结合，在DeepSeek-V4-Pro上，TP=8，B300，接受长度约5、批次大小为1时，达到了383.7 tok/s的速度。
我们将一系列微操作集群重写为融合的Triton内核，例如紧凑型散射、SWA页面索引、验证长度top-k调度以及不规则窗口打包。区块草稿器的采样路径被整合到融合内核中，其矩阵乘法也进行了分片。在一个示例性能分析中，目标验证之外的操作耗时减少了1.7毫秒，而验证本身耗时7.3毫秒。
DSpark直接嵌入SGLang的零开销（重叠）调度器，几乎无需特殊处理，仅增加了本文提出的两步回溯置信度中继。这其中很少有DSpark特有的管道。SGLang的spec-v2运行时已经实现了在独立流上，将下一步的调度与当前前向传播重叠执行；DSpark作为一等公民工作节点加入：前向传播输出以异步future形式返回，跨迭代顺序由运行时的设备端屏障控制，而设备端页表则意味着无需每步进行主机同步。置信度中继使用相同的通道，读取两步之前的数据。解码循环随后运行，不再有每步气泡——比关闭调度器时紧凑约1.5倍。
图 5. 批大小为 1 时的解码，重叠调度器关闭（上）与开启（下）。开启后，run_batch 迭代之间或一个步骤内的块草稿生成与目标验证阶段之间不再有空泡。
对成本表进行性能分析
图 6. 加性成本模型——原始数据与拟合结果（a）及吞吐量（b）——以及预测步时与实测步时（c）。
我们将调度器对步时 T(bs, K) 的估计——其中 K 为批次额外的验证 token 数——表示为一个加性模型：T(bs, K) = bias + alpha(bs) + theta(M)，M = bs + K，其中 alpha(bs) 是请求扩展下限（草稿通过加上部分注意力），不受裁剪影响；theta(M) 是目标的验证 token 成本，也是裁剪能回收的唯一项。调度器的 argmax 在预期接受 token 数与实际边际成本之间进行权衡，因此裁剪空间仅在 theta 较大时才会显现。图 6(c) 验证了该模型对实时服务器的预测效果。
后续计划
DSpark 目前已集成在 SGLang 中；我们在 sgl-project/sglang#30344 跟踪路线图。后续计划包括：
- 成本模型与调度——一个更强、日益在线/自适应的成本模型，以及对动态调度器的进一步改进。
- 模型覆盖——更多稠密和稀疏模型。
- 并行化——更广泛地覆盖各种并行模式和服务拓扑。
- 可观测性——将块接受率估计器和跨检查点的置信度校准等指标投入生产化。
- 鲁棒性——强化全 CUDA 图路径，并进行更广泛的压力/回归测试。
感谢 DSpark 的作者们，以及感谢深度求索提供的算法和模型。
附录：复现
以下所有命令均在预构建镜像（docker pull lmsysorg/sglang:dev-dspark）内运行，或从 sgl-project/sglang#30261 的源代码构建，锁定在提交 692c5f7d。
图 1、图 3 和图 6——前沿服务器（DeepSeek-V4-Flash，H200，DP4）。启动 DSpark 分支：
SGLANG_ENABLE_METRICS_DEVICE_TIMER=1 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Flash-DSpark \
--speculative-algorithm DSPARK \
--tp 4 --dp-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend none --moe-runner-backend flashinfer_mxfp4 --disable-flashinfer-autotune \
--swa-full-tokens-ratio 0.1 --chunked-prefill-size 1024 \
--mem-fraction-static 0.8 --cuda-graph-max-bs 192 --max-running-requests 1024 \
--disable-radix-cache --trust-remote-code --host 0.0.0.0 --port 30000
其中 `--disable-radix-cache` 是为了避免基准测试脚本命中缓存。其他分支仅改变推测配置：非推测模式（non-spec）去掉 `--speculative-*` 参数，加载 `--model-path deepseek-ai/DeepSeek-V4-Flash`；MTP 使用相同目标模型，搭配 `--speculative-algorithm EAGLE --speculative-num-steps {1,3} --speculative-eagle-topk 1 --speculative-num-draft-tokens {2,4}`（取每批次大小中两者的最佳值）；DSpark 紧凑模式或静态模式设置 `SGLANG_RAGGED_VERIFY_MODE=compact|static`；使用 SPS 表执行紧凑模式时，添加 `--speculative-dspark-sps-table-path sps_table.json`；图 3 的 no-trim 分支为 `SGLANG_RAGGED_VERIFY_MODE=compact` 且不加载 SPS 表（即全窗口下的不规则路径）。使用固定提示词驱动任意分支，并在不同批次大小下进行扫描：
python3 -m sglang.benchmark.one_batch_server \
--model None --base-url http://127.0.0.1:30000 \
--batch-size 1 8 16 32 64 96 128 160 192 256 --output-len 1024 --temperature 0.7 \
--fixed-prompt-file frontier_prompt.txt --fixed-prompt-apply-chat-template --show-report
固定提示词在此处（frontier_prompt.txt），由 16 道拼接的 GSM8K 问题组成，以确保生成内容为真实文本。用户可基于自身数据测试，因为不同数据集的推测解码接受长度各不相同。
图 6 的成本表来自性能分析运行：以 `SGLANG_DSPARK_ENABLE_SPS_RECORD=1 SGLANG_SIMULATE_ACC_LEN=1.0` 启动紧凑模式，然后通过 `python3 -m sglang.benchmark.dspark_sps_profiler all` 拟合加性模型（在输入长度为 512 时，扫描批次大小 × 验证比例网格）。
图 4——混合流量。服务器配置与图 1 相同，`--mem-fraction-static 0.7`，块大小为 6；通过 `SGLANG_RAGGED_VERIFY_MODE` 运行所有三种模式（静态/紧凑/接受上限），并驱动包含 gsm8k + arena-hard + poetry 的混合请求集，以非流式完成时间吞吐量进行测量。
图 5——零开销（DeepSeek-V4-Pro，B300，TP8）。
SGLANG_RAGGED_VERIFY_MODE=compact SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_TORCH_PROFILER_DIR=./trace \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-DSpark --speculative-algorithm DSPARK \
--tp 8 --moe-runner-backend flashinfer_mxfp4 --disable-flashinfer-autotune \
--mem-fraction-static 0.82 --chunked-prefill-size 4096 --cuda-graph-max-bs 4 \
--trust-remote-code --host 127.0.0.1 --port 30000
捕获批次大小为 1 的解码轨迹，然后读取仅 GPU 的通道：
python3 -m sglang.benchmark.one_batch_server \
--model None --base-url http://127.0.0.1:30000 \
--batch-size 1 --input-len 256 --output-len 256 \
--profile --profile-activities GPU --profile-steps 20
Speculative decoding trades extra compute for fewer decode steps, and the trade sours as load grows: at batch size B with K speculative tokens the target verifies B * K tokens every step, and past a point that costs more than it saves. DSpark attacks both ends — a semi-autoregressive block drafter (a whole block per draft forward, so acceptance stays high) and a variable per-request verify length driven by the draft model's own confidence, which stops verifying tokens the workload is unlikely to accept. The algorithm and its gains are from the DSpark paper.
SGLang now supports DSpark on both dense and sparse models (e.g. Qwen3 and DeepSeek-V4). This post is about the integration (sgl-project/sglang#30261). We reproduce the shape of the paper's gains on an open serving engine — the per-user speedup, and the verify budget shrinking as load rises — and describe the engineering that turns that schedule into wall-clock time: full CUDA graphs over a ragged, per-request verify (so a trimmed batch replays a genuinely smaller graph, not a padded one); an overlap-aware speculative path that hides the scheduler behind the forward; a cost-table profiler that lets the scheduler size each request's verify budget online; and observability for the acceptance ceiling that trimming would otherwise hide. Hardware, engine, and traffic all differ from the paper, so we reproduce the mechanism and the curve rather than its numbers to the digit, and every "faster" below is measured against our own controls — identical except for the speculation config.
The speedup over MTP and non-spec
Figure 1. Aggregate throughput (y) vs. per-user decode speed (x); each curve sweeps concurrency from batch 1 to 256, one curve per arm. Higher and to the right is better.
DSpark delivers the best throughput/latency trade-off across the whole concurrency sweep, clearly ahead of both MTP and the non-spec floor in the Figure 1 example. All three arms run DeepSeek-V4-Flash on H200 with DP-attention over four ranks, identical except for the speculation config — a non-speculative floor, MTP (the EAGLE-style baseline, the per-batch-size best of the 1-1-2 and 3-1-4 configs), and DSpark.
Adopting DSpark in SGLang
The DSpark algorithm, adopted from the paper, lives in three draft-side pieces:
- Block drafter — a dense line (e.g. Qwen3) and a sparse line (e.g. DeepSeek-V4); one forward emits a
gamma-token block, with a lightweight sequential head (Markov or RNN) conditioning each step on the previous token, so the block is semi-autoregressive. - Confidence head — scores each drafted token's chance of surviving verification; the product across the block is the block's survival probability.
- Sequential Temperature Scaling (STS) — calibrates those scores so survival reflects the true acceptance rate the scheduler budgets against.
Around that, SGLang adds the serving support surface:
- Confidence scheduler — converts per-block survival into a per-request verify budget each step.
- Per-request ragged verify — a variable verify length per request within one batch (
static/compact/cap-accept). - Full CUDA graph — captured over the ragged, variable-length verify.
- Observability — acceptance ceiling under trimming and other metrics.
- Additive SPS cost table — an offline-profiled step-time model, read online by the scheduler.
- Data-parallel attention — supported alongside the other parallelism dimensions.
- Zero-overhead scheduling — integrated into SGLang's overlap scheduler with almost no DSpark-specific special-casing.
- Performance optimizations — fused Triton kernels and a sharded block-drafter matmul.
Verify modes
The three verify modes are the axis the rest of this post turns on. static verifies the full drafted block every step (the baseline). compact verifies only the per-request window the scheduler picked — the production path. cap-accept verifies the full block but commits only up to that window: same output as compact, while exposing what a full verify would have accepted — how we measure the ceiling under trimming.
Ragged verify under full CUDA graphs
Per-request windows don't fit a fixed-shape CUDA graph: a batch where one request verifies two tokens and another six has no single query length, and padding everyone up to the full block width just pads the trim back in. So we keep the batch ragged and key the graph on the total token count — front-pack the variable-length requests into one compact buffer and round up to the nearest captured tier. When budgets trim, the packed total drops to a smaller tier and DSpark replays a genuinely cheaper graph (fewer attention and MLP rows, not a masked full-width forward); under DP attention the ranks share one tier (the largest any rank needs) and step down together.
The packed buffer is a cu_seqlens-style varlen input, so the compact verify reuses attention kernels the backend already has — on DeepSeek-V4 the model's own sparse-MLA path (flash_mla), with no new kernel; each supported backend just rebuilds its varlen metadata from the packed layout on graph replay.
Figure 2. Fitting a batch with per-request-variable verify lengths into a captured CUDA graph. A fixed-shape graph pads every request to the full block width (N x W); the ragged path front-packs the scheduled tokens and rounds only the total up to the nearest captured tier, computing far fewer padded cells for the same accepted tokens.
Observability
Trimming censors the ceiling: compact mode only verifies a block's first few positions — the scheduler's window — so how many tokens a full-block verify would have accepted at that step is never observed — and without it you cannot tell a good trim from a lossy one. A cap-accept run recovers it: it verifies the full block but commits only up to the window, so it commits exactly what compact commits while exposing the ceiling. We also surface per-request confidence and calibration metrics (e.g. ECE) for post-hoc analysis.
Estimating the ceiling under trimming
A block-accept estimator, designed for production runs or other scenarios where an extra companion run is unwanted, recovers the estimated censored ceiling directly inside a compact run. It is implemented with the utilization of the target tokens in the future steps with its logprobs, and computes estimation intervals for the counterfactual tail, assuming property similarity of anchor tokens in the trimmed versus untrimmed trajectory.
A preliminary look at dynamic vs. fixed scheduling
The confidence scheduler is a first, vanilla version, and we treat it that way — a proof that the mechanism works end to end, not a highly tuned result. We compare compact (the per-step SPS-argmax budget) against no-trim — the static full-block schedule run through the same ragged path — on two example workloads that differ in acceptance.
Figure 3. compact (dynamic trim) vs. no-trim (full block), batch 1 to 256 at DP4, on two examples that differ in acceptance. Higher and to the right is better.
The dynamic budget's win is primarily a high-batch effect. At batch size 1 the target verify does not slow down much with more tokens, so trimming saves little and the two arms tie. As concurrency grows and throughput starts to plateau, trimming shortens the step and compact pulls ahead. The gap is larger, and opens earlier, on the lower-accept example — lower acceptance leaves more tail to trim, exactly as the cost model predicts.
Each panel is a clean compact-vs-no-trim A/B (identical setup within a panel), but the two examples are not a strict single-variable pair: beyond acceptance they also differ slightly in setup (prompt formatting and per-arm round count), so we read the trend across them, not absolute cross-panel numbers.
These budgets are also only as good as the cost tables behind them. Our current SPS (and calibration) fit is a first approximation, and it may not yet fully account for how step cost varies with context length — so the exact operating point the scheduler lands on is likely improvable, and we present the mechanism here rather than a tuned number.
Per-request differentiation on mixed traffic
Homogeneous sweeps hide the real point of confidence scheduling. Two requests in the same batch should not get the same verify window if one is far more predictable than the other. Mixed traffic is where that matters.
Figure 4. Budget by workload (left) and per-step verify-length distribution (right).
As an example, we mix three workloads by acceptance difficulty: gsm8k (high), arena-hard (mid), and poetry (low). The window contracts with difficulty — 5.24, 3.78, 2.91 tokens — while utilization against the ceiling (what the block would accept untrimmed) stays high (0.88–0.97). The scheduler is sizing each request, not applying one batch average. The right panel shows it step by step: about 55% of gsm8k steps fill the full window of six, while about 80% of poetry steps use three or fewer.
Performance optimizations and zero-overhead scheduling (ZOS)
Two kinds of engineering turn the schedule into wall-clock time: cutting the cost of each step, and hiding the scheduler behind the forward. Together they reach 383.7 tok/s at accept length ~5 at batch size 1 on DeepSeek-V4-Pro, TP=8, B300.
We rewrote the clusters of tiny ops as fused Triton kernels, such as the compact scatter, the SWA page-index, the verify-length top-k schedule, and the ragged-window packing. The block drafter's sampling path folds into fused kernels, and its matrix multiplication is sharded. In one example profile, things outside the target verify shrinks by 1.7 ms, against a 7.3 ms verify.
DSpark drops straight into SGLang's zero-overhead (overlap) scheduler with almost no special-casing, adding the paper's two-step-back confidence relay. Little of this is DSpark-specific plumbing. SGLang's spec-v2 runtime already overlaps the next step's scheduling with the current forward on separate streams, and DSpark joins as a first-class worker: forward outputs come back as async futures, cross-iteration ordering rides the runtime's device-side barrier, and on-device page tables mean no per-step host sync. The confidence relay uses the same channel, read two steps back. The decode loop then runs with no per-step bubble — about 1.5x tighter than with the scheduler off.
Figure 5. Decode at batch size 1, overlap scheduler off (top) vs. on (bottom). With it on, there is no bubble between run_batch iterations or between the block-draft-generate and target-verify phases inside a step.
Profiling the cost table
Figure 6. Additive cost model — raw vs. fit (a) and throughput (b) — and predicted vs. measured step time (c).
We express the scheduler's estimate of step time T(bs, K) — K the batch's extra verify tokens — with an additive model: T(bs, K) = bias + alpha(bs) + theta(M), M = bs + K, where alpha(bs) is the request-scaling floor (draft pass plus part of attention), unmoved by trimming; theta(M) is the target's verify-token cost, the only term trimming recovers. The scheduler's argmax trades expected accepted tokens against real marginal cost, so trim headroom shows up only where theta is large. Figure 6(c) validates the model's predictions against a live server.
What's next
DSpark is in SGLang today; we track the roadmap in sgl-project/sglang#30344. What's next:
- Cost model and scheduling — a stronger, increasingly online/adaptive cost model and further improvements to the dynamic scheduler.
- Model coverage — more dense and sparse models.
- Parallelism — broader coverage across parallelism modes and serving topologies.
- Observability — productionizing metrics like the block-accept estimator and confidence calibration across checkpoints.
- Robustness — hardening the full-CUDA-graph path and broader stress / regression testing.
Thanks to the DSpark authors and to DeepSeek for the algorithm and the models.
Appendix: Reproduction
All commands below run inside the prebuilt image (docker pull lmsysorg/sglang:dev-dspark), or build from source at sgl-project/sglang#30261, pinned at commit 692c5f7d.
Figures 1, 3, and 6 — the frontier server (DeepSeek-V4-Flash, H200, DP4). Launch the DSpark arm:
SGLANG_ENABLE_METRICS_DEVICE_TIMER=1 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Flash-DSpark \
--speculative-algorithm DSPARK \
--tp 4 --dp-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend none --moe-runner-backend flashinfer_mxfp4 --disable-flashinfer-autotune \
--swa-full-tokens-ratio 0.1 --chunked-prefill-size 1024 \
--mem-fraction-static 0.8 --cuda-graph-max-bs 192 --max-running-requests 1024 \
--disable-radix-cache --trust-remote-code --host 0.0.0.0 --port 30000
where the --disable-radix-cache is to avoid bench scripts hitting the cache. The other arms change only the speculation config: non-spec drops --speculative-* and loads --model-path deepseek-ai/DeepSeek-V4-Flash; MTP uses that same target with --speculative-algorithm EAGLE --speculative-num-steps {1,3} --speculative-eagle-topk 1 --speculative-num-draft-tokens {2,4} (per-batch-size best of the two); DSpark compact or static sets SGLANG_RAGGED_VERIFY_MODE=compact|static; use --speculative-dspark-sps-table-path sps_table.json when executing compact mode with SPS table; and Figure 3's no-trim arm is SGLANG_RAGGED_VERIFY_MODE=compact with no SPS table (the ragged path at the full window). Drive any arm with a fixed prompt swept across batch sizes:
python3 -m sglang.benchmark.one_batch_server \
--model None --base-url http://127.0.0.1:30000 \
--batch-size 1 8 16 32 64 96 128 160 192 256 --output-len 1024 --temperature 0.7 \
--fixed-prompt-file frontier_prompt.txt --fixed-prompt-apply-chat-template --show-report
The fixed prompt is here (frontier_prompt.txt), 16 concatenated GSM8K questions to allow the generation be real content. Users may test on their own data given speculative decoding has different accept lengths for different datasets.
Figure 6's cost table comes from a profiling run: launch compact with SGLANG_DSPARK_ENABLE_SPS_RECORD=1 SGLANG_SIMULATE_ACC_LEN=1.0, then fit the additive model with python3 -m sglang.benchmark.dspark_sps_profiler all (sweeping a batch × verify-fraction grid at input-len 512).
Figure 4 — mixed traffic. The same server as Figure 1, at --mem-fraction-static 0.7 with block size six; run all three modes (static / compact / cap-accept) via SGLANG_RAGGED_VERIFY_MODE, and drive a mixed gsm8k + arena-hard + poetry request set, measured as non-streaming makespan throughput.
Figure 5 — zero-overhead (DeepSeek-V4-Pro, B300, TP8).
SGLANG_RAGGED_VERIFY_MODE=compact SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_TORCH_PROFILER_DIR=./trace \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-DSpark --speculative-algorithm DSPARK \
--tp 8 --moe-runner-backend flashinfer_mxfp4 --disable-flashinfer-autotune \
--mem-fraction-static 0.82 --chunked-prefill-size 4096 --cuda-graph-max-bs 4 \
--trust-remote-code --host 127.0.0.1 --port 30000
Capture a batch-1 decode trace, then read the GPU-only lane:
python3 -m sglang.benchmark.one_batch_server \
--model None --base-url http://127.0.0.1:30000 \
--batch-size 1 --input-len 256 --output-len 256 \
--profile --profile-activities GPU --profile-steps 20