摘要
如今，最先进的（SOTA）模型规模越来越大，崩溃后重新加载模型服务的成本极其高昂。因此，我们推出了 Weight Cache Daemon（权重缓存守护进程），这是一个常驻 GPU 进程，将量化后的模型权重保存在 GPU 内存中，并通过 CUDA IPC 零拷贝映射将其提供给新的 SGLang 引擎实例。这使权重加载时间从数分钟缩短到数秒。
Weight Cache Daemon 是我们快速引擎恢复框架的第一阶段，该框架的目标是为生产环境中的 LLM 服务实现冷启动小于 10 秒、热备切换小于 1 秒。
关键成果：
- 权重加载：约 495 秒 → 约 0.63 秒——基于 Ling-2.6-1T FP8 模型，实现了约 785 倍的加速。
- 总启动时间：8.8 分钟 → 0.528 分钟——端到端引擎启动时间减少了 93.9%。
- 多实例权重共享——同一 GPU 上的多个引擎实例映射到相同的 IPC 句柄，消除了重复的磁盘 I/O 和量化后变换。
- 主备故障切换小于 1 秒——备用引擎通过零拷贝共享权重，无需将整块 GPU 专用于空闲副本，即可实现近乎零停机的故障切换。
- 多节点实例权重共享——支持大模型的多节点模式。
背景
随着 LLM 模型规模不断增大——Qwen3-235B、Ling-2.6-1T，以及新发布的 2.8T Kimi K3——服务引擎的冷启动时间已成为生产效率的关键瓶颈。一个部署在 8×H20-3e GPU 上的 Ling-2.6-1T FP8 实例仅准备就绪就需要约 8.52 分钟，权重存储在 3.5T NVME SSD 上。在生产环境中，这意味着：
- 重启期间 P99 尾延迟飙升——所有进行中的请求要么失败，要么无限期排队。
- 可用性降低——数分钟的恢复窗口违反了 SLA 目标。
- 运维摩擦——滚动更新、配置变更和故障恢复都受制于重启周期。
- GPU 资源浪费——传统的主备部署会将整组 GPU 专用于空闲副本，使故障切换的硬件成本翻倍。
时间都花在哪里了？我们对 Ling-2.6-1T FP8 的完整 SGLang 引擎启动过程进行了剖析：
| 阶段 | 时间（秒） | 占比 | 说明 |
|---|---|---|---|
| 预初始化与 ServerArgs | 约 1 | 0.2% | 预初始化与 ServerArgs 解析 |
| Tokenizer 初始化 | 约 13 | 2.4% | 加载并初始化 tokenizer |
| 初始化 torch 分布式 | 约 5 | 0.9% | NCCL 2.28.9，8 卡 H20，NVLink mesh 370.8 GB/s，P2P/IPC；最慢 rank TP1=5.19s |
| 加载权重（磁盘） | 约 495 | 93.9% | 161 个分片，W8A8 FP8（CompressedTensorsW8A8Fp8MoE），最慢 rank=495.3s，每卡 120GB；磁盘 I/O 瓶颈 |
| 缓存分配（KV+Mamba） | 约 1 | 0.2% | KV：553,599 个 token/5.94GB bf16；Mamba SSM 状态：5.33GB，max_mamba_cache_size=155 |
| 捕获 CUDA 图 | 约 7.7 | 1.5% | 仅 3 个解码 BS [1,2,4] |
| 服务器就绪 | 约 4 | 0.8% | Unified RadixTree 初始化，HTTP/uvicorn 启动，预热请求 |
| 总计 | 约 527 | 约 8.8 分钟 |
瓶颈很明显：从磁盘加载权重占启动时间的 93.2%。对于 Ling-2.6-1T FP8 模型，每个 TP rank 从磁盘读取约 120GB 的 safetensors，进行反序列化、应用 TP 分片，并执行量化后变换（FP8 量化、权重重打包）。每次重启都会重复完全相同的工作，尽管最终得到的 GPU 张量是确定性的，而且往往已经存在于 GPU 内存中。
我们能否避免每次从磁盘重新加载？答案是肯定的——方法是在引擎重启期间将权重保留在 GPU 内存中。
设计
核心思路：通过 CUDA IPC 实现持久化权重缓存
权重缓存守护进程是一个持久的 GPU 进程，将量化后、TP 分片后的权重保存在 GPU 内存中。引擎重启时，新的引擎进程通过 CUDA IPC 零拷贝从守护进程映射权重——无需磁盘 I/O、无需反序列化、无需量化。
每个 GPU 为其 TP rank 运行一个守护进程。该守护进程：
- 从磁盘加载模型权重（完整流水线：磁盘 → TP 分片 → 量化 → 重打包）。
- 将 model.state_dict() 中的每个参数和缓冲区导出为 CUDA IPC 句柄。
- 记录 CacheConfig 指纹（模型路径、TP/DP 大小、量化配置哈希、dtype）。
- 通过 Unix socket 向请求的引擎进程提供 IPC 句柄。
引擎连接到守护进程，验证配置兼容性，并将权重直接映射到其地址空间中——引擎和守护进程通过 CUDA IPC 共享同一块物理 GPU 内存。
通过 Meta 设备实现零拷贝加载
亚秒级加载的关键在于零拷贝：引擎的 param.data 指针直接指向 IPC 映射的 GPU 张量。不复制任何数据。
为实现这一目标，引擎在 meta 设备上初始化模型（不分配 GPU/CPU 内存），然后将每个参数的数据指针替换为 IPC 映射的张量。
由 process_weights_after_loading() 创建的后量化参数（例如 FP8 量化产生的 weight_scale）也会被守护进程缓存并直接映射——无需重新量化。
配置校验：安全第一
引擎配置与守护进程缓存配置之间的任何不匹配都会触发完整的磁盘重新加载，以确保正确性：
| 字段 | 不匹配示例 | 后果 |
|---|---|---|
| model_path + model_arch + revision | 不同的模型或版本 | 权重完全错误 |
| tp_size + tp_rank | 不同的 TP 分片方式 | 该 rank 拿到错误的分片 |
| pp_size + pp_rank | 不同的 PP 划分方式 | 该流水线阶段拿到错误的层 |
| dp_size + ep_size | 不同的 DP/EP 策略 | 权重分布不正确 |
| quant_method + quant_config_hash | 不同的量化方式 | 未量化与 FP8 不匹配 |
| dtype | float16 与 bfloat16 | 类型不匹配 |
| device_capability + torch_version | 不同的 GPU 架构或 torch 版本 | 权重映射正常但数值结果错误 |
最后两个字段构成一个环境戳记：如果守护进程和客户端运行了不同的后处理分支（不同的计算能力或 torch/内核版本），它们产出的权重虽然能通过 IPC 正常映射，却可能输出垃圾结果——将环境信息戳记写入 CacheConfig，就能把这种情况变成一个清晰的不匹配。
这对生产环境的安全性至关重要：如果运维人员更改了模型或量化配置，引擎会检测到不匹配并回退到磁盘加载，而不是映射不兼容的权重。
在配置校验之上，量化方法还受 IPC 白名单约束。CUDA IPC 零拷贝只导出原始张量数据，因此只有当 process_weights_after_loading() 的全部效果都被该数据完整覆盖时，这种方式才是正确的。那些会在 Python 侧写入元数据或重新打包/转置权重的方法（per-tensor FP8、Marlin、AWQ/GPTQ）会静默地产生错误的数值结果——它们会直接抛出硬错误。目前已验证通过：未量化和 block-wise FP8（已设置 weight_block_size）；更多方法将在端到端验证后陆续加入。
三种模式：daemon、client 和 off
| 模式 | 流程 | 权重加载时间 | GPU 显存 | 使用场景 |
|---|---|---|---|---|
| 守护进程 | 引擎启动守护进程 → 守护进程从磁盘加载 → 引擎映射 IPC | < 1 秒（守护进程就绪后） | 1×（共享） | 首次启动；引擎管理守护进程生命周期 |
| 客户端 | 连接至已运行的守护进程 → 映射 IPC | < 1 秒 | 1×（共享） | 引擎重启；守护进程已在运行 |
| 关闭 | 正常磁盘加载 | 405–411 秒（Ling-2.6-1T FP8） | 1× | 默认；无缓存 |
在守护进程模式下，引擎在启动时生成守护进程，并等待它们从磁盘加载权重。首次启动仍然较慢（守护进程必须从磁盘加载），但后续重启是即时的。
在客户端模式下，引擎连接到已在运行的守护进程。这是快速重启路径——守护进程提前启动，且已在 GPU 显存中持有权重。
安全性与健壮性
权重缓存守护进程的设计目标是非侵入性和安全性：
- 最小侵入性：该功能自包含在 python/sglang/srt/weight_cache/ 中，对核心引擎的改动极小（仅 load_model() 分发和一个 CLI 标志）。
- 崩溃安全：如果守护进程崩溃，现有引擎实例继续运行——它们已通过 CUDA 引用计数持有 IPC 映射张量的引用。只有当守护进程和引擎都退出时，GPU 显存才会被释放。
- 守护进程恢复：如果守护进程重启，它会从磁盘重新加载权重并重新导出 IPC 句柄。新的引擎实例随后可以连接到重启后的守护进程。
- 配置不匹配时的回退：配置不匹配会自动回退到磁盘加载（客户端模式），或抛出错误（守护进程模式，因为两个进程共享同一 GPU，回退会导致 OOM）。
超越重启：生产场景
权重缓存守护进程解锁了传统基于磁盘的加载方式难以实现的生产模式：
多实例权重共享
每 GPU 一个守护进程在内存中持有权重；多个引擎实例（例如独立服务）通过零拷贝映射到相同的 IPC 句柄。无论有多少实例消费权重，权重都仅从磁盘加载并量化一次。
优先级协同服务
在同一块 GPU 上运行高优先级在线服务和低优先级批处理任务，两者由同一个权重缓存守护进程提供支持。低优先级实例可以在亚秒级时间内被驱逐并重新启动，无需从磁盘重新加载权重——从而实现灵活的 GPU 分时共享，且没有通常的启动开销。
主备故障切换
在主引擎旁边部署一个备用引擎，两者由同一个权重缓存守护进程提供支持。备用引擎通过零拷贝映射权重并保持热备状态。当主引擎发生故障时，备用引擎在 < 1 秒内接管——无需加载权重，无需磁盘 I/O。
这实现了近乎零宕机的故障切换，而无需将整组 GPU 专门用于空闲副本，避免了传统热备部署中昂贵的 GPU 资源浪费。
性能
权重加载：磁盘 vs IPC 零拷贝
单节点
| 模型 | 权重大小 | 磁盘加载（秒） | IPC 零拷贝（秒） | 加速比 |
|---|---|---|---|---|
| Qwen3-235B FP8 | ~235 GB | ~306–327 | <1 | ~500× |
| Ling-2.6-1T | ~1 TB | ~405–411 | <1 | ~780× |
性能图表
使用方法
启动权重缓存守护进程 - 单节点
一条命令即可启动所有 TP 秩守护进程：
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 4 \
--load-format auto --dtype auto --quantization fp8
等待守护进程就绪（它们会为每个秩写入一个 .ready 文件）：
ls /tmp/sglang_weight_cache_rank*.ready
使用权重缓存启动引擎
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 4 \
--weight-cache-mode client
启动权重缓存守护进程 - 多节点
在多节点部署中，每个节点为其本地 TP 秩运行自己的守护进程。所有守护进程加入同一个分布式组，因此 --nnodes、--node-rank 和 --dist-init-method 必须在各节点间保持一致，$MASTER_ADDR 指向节点 0：
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 2 \
--load-format auto --dtype auto --quantization fp8 \
--nnodes 2 --node-rank 0 \
--dist-init-method tcp://$MASTER_ADDR:29500
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 2 \
--load-format auto --dtype auto --quantization fp8 \
--nnodes 2 --node-rank 1 \
--dist-init-method tcp://$MASTER_ADDR:29500
当每个节点都报告其守护进程就绪后，启动引擎客户端。它们使用与守护进程（29500）不同的独立 rendezvous 端口（29600）：
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 2 \
--weight-cache-mode client \
--nnodes 2 --node-rank 0 \
--dist-init-addr $MASTER_ADDR:29600 --port 34000
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 2 \
--weight-cache-mode client \
--nnodes 2 --node-rank 1 \
--dist-init-addr $MASTER_ADDR:29600
快速引擎恢复框架：路线图
权重缓存守护进程是更广泛的快速恢复框架的第一阶段，该框架的目标是 < 10 秒冷重启和 < 1 秒热备切换：
| 阶段 | 当前（秒） | 目标（秒） | 方案 | 状态 |
|---|---|---|---|---|
| 加载权重 | ~306–327 | < 1 | 权重缓存守护进程（CUDA IPC） | 已完成（本 PR） |
| 捕获 CUDA 图 | ~34.9 | < 3 | CUDA 图序列化 + 重放 | 计划中 |
| DeepGEMM JIT 预热 | ~23.1 | < 2 | 内核缓存持久化，并行预热 | 计划中 |
| 服务器初始化与分词器 | ~17.3 | < 3 | 延迟分词器初始化，配置缓存 | 计划中 |
| 初始化 torch 分布式 | ~4.7 | < 2 | NCCL 会话复用，持久化进程组 | 计划中 |
| KV 缓存分配 | ~0.5 | < 0.5 | kvcache 复用 | 计划中 |
| 服务器就绪 | ~3.4 | < 1 | 重启时跳过预热请求 | 计划中 |
| 总计（单节点） | 约 390 | < 10 |
对更多模型的支持也即将推出。
公开路线图
权重缓存守护进程只是第一步——还有很多工作要做，我们对未来的路线充满期待。目前的第一阶段涵盖 TP + PP、单节点和多节点启动、每 GPU 零拷贝 CUDA IPC，以及非量化加块级 FP8。除此之外，还有许多高影响力的方向仍然开放：
- 更多模型与量化：将 IPC 允许列表扩展到块级 FP8 之外（逐张量 FP8、INT8、MXFP8、NVFP4、AWQ/GPTQ 等），并覆盖更多架构，包括多模态和 LoRA 基础权重。
- DP/EP 与多节点：DP/EP 分片键控和跨节点守护进程协调、生命周期管理和故障转移。
- 无需重新加载的权重更新：用于 RL / 在线更新的就地权重刷新，以守护进程作为交付代理。
- 跨 GPU 与集群共享：对等复制和集群填充，使集群冷启动时每个分片组大约只需一次磁盘读取。
- KV 缓存恢复：在重启 / 故障转移期间保留并重新映射 KV 缓存（KV 复用、交接给备用节点），使进行中的上下文在恢复后得以保留，而不是从头重新计算。
- 启动路径的其余部分：CUDA 图序列化、内核预热持久化，以及更快的服务器 / 分布式初始化，以实现 < 10 秒的冷重启目标。
- 其他硬件后端：将此功能扩展到提供类似功能的其他加速器（AMD 和 Intel 都有可比的 IPC 机制）。
- 运维与可靠性：指标、状态工具、安全加固和 CI 覆盖。
这很大程度上是一项社区工作。完整计划在 sgl-project/sglang#33522 中公开跟踪——非常欢迎贡献和反馈，而且有很多有影响力的工作可以接手。
致谢
蚂蚁集团 Ant Ling 基础设施团队：Michael Qiu qiudayu.qdy@antgroup.com
阿里巴巴：Siyu Liu liusy58@smail.nju.edu.cn
SGLang 团队：Alex Nails
TL;DR
Nowadays, State-of-the-Art (SOTA) models are getting much bigger and reloading the model service after a crash is very expensive. Therefore, we are introducing the Weight Cache Daemon, a persistent GPU process that holds post-quantized model weights in GPU memory and serves them to new SGLang engine instances via CUDA IPC zero-copy mapping. This reduces weight loading from minutes to seconds.
The Weight Cache Daemon is the first phase of our Fast Engine Recovery Framework, which targets < 10 second cold restarts and < 1 second warm standby switches for production LLM serving.
Key results:
- Weight loading: ~495s → ~0.63s — a ~785× speedup, based on the Ling-2.6-1T FP8 model.
- Total startup: 8.8min → 0.528min — an 93.9% reduction in end-to-end engine boot time.
- Multi-instance weight sharing — multiple engine instances on the same GPU map to the same IPC handles, eliminating redundant disk I/O and post-quantization transforms.
- Active-standby failover in < 1 second — standby engines share weights via zero-copy, enabling near-zero-downtime failover without dedicating full GPUs to idle replicas.
- Multi-node-instance weight sharing - support multi-node mode for large models
Background
As LLM models grow larger — Qwen3-235B, Ling-2.6-1T, and the newly released 2.8T Kimi K3 — the cold-start time of serving engines has become a critical bottleneck for production efficiency. A Ling-2.6-1T FP8 instance on 8×H20-3e GPUs takes ~8.52 minutes just to become ready to serve, weights stay in 3.5T NVME SSD. In production, this means:
- P99 tail latency spikes during restarts — all in-flight requests fail or queue indefinitely.
- Reduced availability — multi-minute recovery windows violate SLA targets.
- Operational friction — rolling updates, config changes, and failure recovery are all bottlenecked by the restart cycle.
- GPU resource waste — traditional active-standby deployments dedicate a full set of GPUs to idle replicas, doubling hardware cost for failover.
Where does the time go? We profiled a complete SGLang engine startup for Ling-2.6-1T FP8:
| Phase | Time (s) | Percentage | Notes |
|---|---|---|---|
| Pre-init & ServerArgs | ~1 | 0.2% | Pre-init and ServerArgs parsing |
| Tokenizer init | ~13 | 2.4% | load and init tokenizer |
| Init torch distributed | ~5 | 0.9% | NCCL 2.28.9,8 卡 H20,NVLink mesh 370.8 GB/s,P2P/IPC;slowest rank TP1=5.19s |
| Load weight (disk) | ~495 | 93.9% | 161 shard,W8A8 FP8 (CompressedTensorsW8A8Fp8MoE),slowest rank=495.3s, 120GB per card; Disk I/O bound |
| Cache allocation (KV+Mamba) | ~1 | 0.2% | KV:553,599 tokens/5.94GB bf16;Mamba SSM state:5.33GB,max_mamba_cache_size=155 |
| Capture CUDA graph | ~7.7 | 1.5% | only 3 decode BS [1,2,4] |
| Server ready | ~4 | 0.8% | Unified RadixTree init, HTTP/uvicorn startup, warmup requests |
| Total | ~527 | ~8.8 minutes |
The bottleneck is clear: weight loading from disk accounts for 93.2% of startup time. For Ling-2.6-1T FP8 model, each TP rank reads ~120GB of safetensors from disk, deserializes, applies TP sharding, and runs post-quantization transforms (FP8 quantization, weight repacking). This work is repeated identically on every restart, even though the resulting GPU tensors are deterministic and often already present in GPU memory.
Can we avoid reloading from disk every time? The answer is yes — by keeping weights in GPU memory across engine restarts.
Design
Core Idea: Persistent Weight Cache via CUDA IPC
The Weight Cache Daemon is a persistent GPU process that holds post-quantized, TP-sharded weights in GPU memory. On engine restart, the new engine process maps weights from the daemon via CUDA IPC zero-copy — no disk I/O, no deserialization, no quantization.
Each GPU runs one daemon process for its TP rank. The daemon:
- Loads model weights from disk (full pipeline: disk → TP shard → quantize → repack).
- Exports every parameter and buffer in
model.state_dict()as CUDA IPC handles. - Records a
CacheConfigfingerprint (model path, TP/DP size, quant config hash, dtype). - Serves IPC handles over a Unix socket to requesting engine processes.
The engine connects to the daemon, validates config compatibility, and maps weights directly into its address space — the engine and daemon share the same physical GPU memory via CUDA IPC.
Zero-Copy Loading via Meta Device
The key to sub-second loading is zero-copy: the engine's param.data pointer is set directly to the IPC-mapped GPU tensor. No data is copied.
To achieve this, the engine initializes the model on the meta device (no GPU/CPU memory allocation), then replaces each parameter's data pointer with the IPC-mapped tensor.
Post-quantization parameters (e.g., weight_scale from FP8 quantization) that were created by process_weights_after_loading() are also cached by the daemon and mapped directly — no re-quantization needed.
Config Validation: Safety First
Any mismatch between the engine's config and the daemon's cached config triggers a full disk reload, ensuring correctness:
| Field | Mismatch Example | Consequence |
|---|---|---|
model_path + model_arch + revision | Different model or revision | Wrong weights entirely |
tp_size + tp_rank | Different TP sharding | Wrong shard for this rank |
pp_size + pp_rank | Different PP partitioning | Wrong layers for this pipeline stage |
dp_size + ep_size | Different DP/EP strategy | Incorrect weight distribution |
quant_method + quant_config_hash | Different quantization | Unquantized vs FP8 mismatch |
dtype | float16 vs bfloat16 | Type mismatch |
device_capability + torch_version | Different GPU arch or torch version | Weights map cleanly but serve wrong numerics |
The last two fields form an environment stamp: a daemon and a client that ran different post-processing branches (different compute capability or torch/kernel version) can produce weights that map cleanly over IPC yet serve garbage — stamping the environment into CacheConfig turns that into a clean mismatch.
This is critical for production safety: if an operator changes the model or quantization config, the engine will detect the mismatch and fall back to disk loading rather than mapping incompatible weights.
On top of config validation, quantization methods are gated by an IPC allowlist. CUDA IPC zero-copy exports only raw tensor data, so it is correct only when the entire effect of process_weights_after_loading() is captured by that data. Methods that stamp Python-side metadata or repack/transpose weights (per-tensor FP8, Marlin, AWQ/GPTQ) would silently serve wrong numerics — they raise a hard error instead. Currently verified: unquantized and block-wise FP8 (weight_block_size set); more methods will be added after end-to-end verification.
Three Modes: daemon, client, and off
| Mode | Flow | Weight Load Time | GPU Memory | Use Case |
|---|---|---|---|---|
| daemon | Engine launches daemon → daemon loads from disk → engine maps IPC | < 1s (after daemon ready) | 1× (shared) | First start; engine manages daemon lifecycle |
| client | Connect to pre-running daemon → map IPC | < 1s | 1× (shared) | Engine restart; daemon pre-running |
| off | Normal disk loading | 405–411s (Ling-2.6-1T FP8) | 1× | Default; no cache |
In daemon mode, the engine spawns daemon processes during startup and waits for them to load weights from disk. The first start is still slow (daemons must load from disk), but subsequent restarts are instant.
In client mode, the engine connects to already-running daemons. This is the fast-restart path — the daemon was started earlier and already holds weights in GPU memory.
Safety and Robustness
The Weight Cache Daemon is designed to be non-intrusive and safe:
- Minimal invasiveness: The feature is self-contained in
python/sglang/srt/weight_cache/with minimal changes to the core engine (onlyload_model()dispatch and a CLI flag). - Crash-safe: If the daemon crashes, existing engine instances continue running — they already hold references to the IPC-mapped tensors via CUDA reference counting. GPU memory is only freed when both the daemon and the engine exit.
- Daemon recovery: If the daemon is restarted, it reloads weights from disk and re-export IPC handles. New engine instances can then connect to the restarted daemon.
- Fallback on mismatch: Config mismatches automatically fall back to disk loading (in client mode) or raise an error (in daemon mode, where fallback would cause OOM since both processes share the same GPU).
Beyond Restart: Production Scenarios
The Weight Cache Daemon unlocks production patterns that are impractical with traditional disk-based loading:
Multi-Instance Weight Sharing
A single daemon per GPU holds weights in memory; multiple engine instances (e.g., independent services) map to the same IPC handles via zero-copy. Weights are loaded from disk and quantized exactly once per GPU, regardless of how many instances consume them.
Priority Co-Serving
Run a high-priority online service and a low-priority batch job on the same GPU, backed by the same weight cache daemon. The low-priority instance can be evicted and re-spawned in sub-second time without reloading weights from disk — enabling flexible GPU time-sharing without the usual startup penalty.
Active-Standby Failover
Deploy a standby engine alongside the primary, both backed by the same weight cache daemon. The standby maps weights via zero-copy and stays warm. When the primary fails, the standby takes over in < 1 second — no weight loading, no disk I/O.
This achieves near-zero-downtime failover without dedicating a full set of GPUs to an idle replica, avoiding the expensive GPU resource waste of traditional hot-standby deployments.
Performance
Weight Loading: Disk vs IPC Zero-Copy
Single Node
| Model | Weight Size | Disk Load (s) | IPC Zero-copy (s) | Speedup |
|---|---|---|---|---|
| Qwen3-235B FP8 | ~235 GB | ~306–327 | <1 | ~500× |
| Ling-2.6-1T | ~1 TB | ~405–411 | <1 | ~780× |
Performance Chart
How to Use
Launch Weight Cache Daemons - single-node
One command launches all TP rank daemons:
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 4 \
--load-format auto --dtype auto --quantization fp8
Wait for daemons to become ready (they write a .ready file per rank):
ls /tmp/sglang_weight_cache_rank*.ready
Start Engine with Weight Cache
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 4 \
--weight-cache-mode client
Launch Weight Cache Daemons - multi-node
In a multi-node deployment, each node runs its own daemon for its local TP ranks. All daemons join the same distributed group, so --nnodes, --node-rank, and --dist-init-method must be consistent across nodes, with $MASTER_ADDR pointing at node 0:
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 2 \
--load-format auto --dtype auto --quantization fp8 \
--nnodes 2 --node-rank 0 \
--dist-init-method tcp://$MASTER_ADDR:29500
python -m sglang.srt.weight_cache.daemon \
--model-path /path/to/model --tp-size 2 \
--load-format auto --dtype auto --quantization fp8 \
--nnodes 2 --node-rank 1 \
--dist-init-method tcp://$MASTER_ADDR:29500
Once every node reports its daemons ready, start the engine clients. They use a separate rendezvous port (29600) from the daemons (29500):
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 2 \
--weight-cache-mode client \
--nnodes 2 --node-rank 0 \
--dist-init-addr $MASTER_ADDR:29600 --port 34000
python -m sglang.launch_server \
--model-path /path/to/model --tp-size 2 \
--weight-cache-mode client \
--nnodes 2 --node-rank 1 \
--dist-init-addr $MASTER_ADDR:29600
Fast Engine Recovery Framework: Roadmap
The Weight Cache Daemon is Phase 1 of a broader Fast Recovery Framework targeting < 10s cold restarts and < 1s warm standby switches:
| Phase | Current (s) | Target (s) | Approach | Status |
|---|---|---|---|---|
| Load weight | ~306–327 | < 1 | Weight Cache Daemon (CUDA IPC) | Done (this PR) |
| Capture CUDA graph | ~34.9 | < 3 | CUDA graph serialization + replay | Planned |
| DeepGEMM JIT warmup | ~23.1 | < 2 | Kernel cache persistence, parallel warmup | Planned |
| Server init & Tokenizer | ~17.3 | < 3 | Lazy tokenizer init, config caching | Planned |
| Init torch distributed | ~4.7 | < 2 | NCCL session reuse, persistent process groups | Planned |
| KV Cache allocation | ~0.5 | < 0.5 | kvcache reuse | Planned |
| Server ready | ~3.4 | < 1 | Skip warmup requests on restart | Planned |
| Total (single-node) | ~390 | < 10 |
Support for more models is also on the way.
Public Roadmap
The Weight Cache Daemon is just the first step — there is still a lot to build, and we are excited about the road ahead. Phase 1 today covers TP + PP, single- and multi-node launch, per-GPU zero-copy CUDA IPC, and unquantized plus block-wise FP8. Beyond that, many high-impact directions remain open:
- More models & quantization: extend the IPC allowlist beyond block-wise FP8 (per-tensor FP8, INT8, MXFP8, NVFP4, AWQ/GPTQ, ...) and cover more architectures, including multimodal and LoRA base weights.
- DP/EP & multi-node: DP/EP shard keying and cross-node daemon coordination, lifecycle management, and failover.
- Weight update without reload: in-place weight refresh for RL / online updates, with the daemon as the delivery agent.
- Cross-GPU & fleet sharing: peer-copy and fleet-fill so a cluster cold start pays roughly one disk read per shard group.
- KV cache restore: preserve and remap KV cache across restarts / failover (KV reuse, handoff to standby) so in-flight context survives recovery instead of being recomputed from scratch.
- Rest of the startup path: CUDA graph serialization, kernel-warmup persistence, and faster server / distributed init to reach the < 10s cold-restart goal.
- Other hardware backends: extend this feature to other accelerators that expose similar functionality (AMD and Intel both have comparable IPC mechanisms).
- Ops & reliability: metrics, status tooling, security hardening, and CI coverage.
This is very much a community effort. The full plan is tracked publicly in sgl-project/sglang#33522 — contributions and feedback are very welcome, and there is plenty of impactful work to pick up.
Acknowledgements
Ant Ling Infra Team, Ant Group: Michael Qiu qiudayu.qdy@antgroup.com
Alibaba: Siyu Liu liusy58@smail.nju.edu.cn
SGLang Team: Alex Nails