Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork
TL;DR
We introduce Tensor R-Fork (stands for Tensor Remote Fork), a novel weight loading methodology that leverages efficient inter-node device-to-device interconnect to load tensors from a running SGLang instance to a new instance with zero-copy.
We introduce Tensor R-Fork (stands for Tensor Remote Fork), a novel weight loading methodology that leverages efficient inter-node device-to-device interconnect to load tensors from a running SGLang instance to a new instance with zero-copy.
Eliminates redundant model weight storage on local disk and/or DRAM;
Ensures non-disturbing operation for inference services.
For instance, when applied to Deepseek-R1 model, the loading time is reduced from several minutes to mere seconds, while local disk and/or DRAM storage usage is reduced by ~600GB and inference service quality maintains during model transfers.
Background
As the scale LLM services and the size of model weights continue to expand, the cold-start time of SGLang instances has become a critical bottleneck for production efficiency. Among the cold-start phases, weight loading remains the most time-consuming task.
Taking Deepseek-R1 as an example, loading weights from local disk typically takes several minutes, while loading from remote storage systems can take up to tens of minutes. As model sizes continue to grow exponentially, the time required for initialization and data transfer will likely worsen.
How can we optimize weight loading performance? The most straightforward approach is to maximize the bottleneck bandwidth in the weight data flow. The data flow of commonly-used model loading approaches in the industry and their associated bottleneck bandwidths are as follows:
Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork
TL;DR
We introduce Tensor R-Fork (stands for Tensor Remote Fork), a novel weight loading methodology that leverages efficient inter-node device-to-device interconnect to load tensors from a running SGLang instance to a new instance with zero-copy.
We introduce Tensor R-Fork (stands for Tensor Remote Fork), a novel weight loading methodology that leverages efficient inter-node device-to-device interconnect to load tensors from a running SGLang instance to a new instance with zero-copy.
Eliminates redundant model weight storage on local disk and/or DRAM;
Load weights fromData FlowBottleneckremote storage centerremote storage -> remote Ethernet NIC -> Ethernet -> local Ethernet NIC -> local DRAM -> local GPU memoryNVMe/Ethernet NIClocal diskdisk -> DRAM -> GPU memoryNVMelocal DRAMDRAM -> GPU memoryPCIe
Can we exploit higher-bandwidth data flows for transferring tensors? The answer is YES — inter-node device-to-device interconnect offers hundreds of gigabytes per second of throughput. However, the critical question remains: How can we fully leverage this interconnect's bandwidth for efficient weight loading in SGLang?
To address this challenge, we have developed a novel weight-loading framework called Tensor R-Fork (stands for Tensor Remote Fork), which reduces Deepseek-R1 model loading time to mere seconds and is already production-ready.
Design
The core concept of Tensor R-Fork[0] is to leverage GPU-Direct RDMA for constructing a peer-to-peer (P2P) weight storage architecture.
The performance of data transfer using traditional method is low, because there is always bottleneck in the entire path, whose bandwidth is much smaller than inter-node device-to-device interconnect. From the data flow analysis, we observe that weight tensors are stored on each GPU and can be transmitted directly between nodes via GPU-direct RDMA.
To maximize the utilization of RDMA-capable NIC's bandwidth, we design a per GPU-pair data transfer strategy: a local GPU directly transfers data to/from its paired remote GPU. This design effectively bypasses the PCIe bottleneck between GPU and CPU, enabling high-throughput communication without relying on CPU or host memory. The data flow of loading weights from remote SGLang instance is the following:
Implementation
To make every running instances act as the source of model weights for any new instance requiring the same model—while minimizing (or ideally eliminating) disruption to the inference services of running instances—we implemented the framework with two backend options: NCCL and TransferEngine. Consider a running instance A (referred to as the source instance) and a new instance B to be booted (destination instance). Below, we will explain the implementation of weight transfer mechanisms using these two backends in detail.
NCCL backend
When using NCCL as the backend[1], the process involves two stages:
Establishing communication groups between source and destination instances.
Transferring weights from the source instance to the destination instance via these groups.
During destination instance initialization, it sends an HTTP request to the designated source instance to initiate communication group creation. Each TPWorker of destination instance establishes a NCCL communication group with its corresponding TPWorker of source instance (i.e.source rank 0 pairs with destination rank 0, etc.). Each communication group consists of exactly two members: the source TPWorker and destination TPWorker.
Once communication groups are established, each source TPWorker broadcasts its weights tensor located on GPU memory through the group using NCCL broadcast. The destination TPWorker receives the weights directly into its GPU memory without any intermediate memory copies.
While NCCL serves as Tensor R-Fork backend by leveraging GPU-Direct RDMA, it does have a critical limitation: weight transfer disrupts the source instance's inference service, due to two key factors:
Communication group establishment: The source instance must actively participate in creating communication groups.
CUDA kernel interference: The NCCL broadcast mechanism triggers CUDA kernel execution, which competes for GPU resources and introduces latency spikes during generation tasks.
TransferEngine backend
To achieve non-disturbing weight transfer, we introduce an alternative backend: TransferEngine, which leverages GPU-Direct RDMA for efficient data movement[2]. TransferEngine (TE) is a lightweight RDMA-based transfer runtime that runs alongside each TPWorker on the source instance and exposes GPU-resident weight tensors to remote readers without invoking CUDA kernels on the source.
During source SGLang instance initialization:
Each TPWorker (tensor parallel worker) spawns a TransferEngine instance.
TransferEngine registers the GPU memory addresses of its weights with the RDMA channel.
When initializing the destination instance:
It sends an HTTP request to retrieve the source instance's TransferEngine metadata, including RDMA keys mapped to the corresponding GPU memory addresses.
Using these RDMA keys, the destination instance directly loads weights from the source's GPU memory without interrupting the source instance's ongoing services.
*Want to learn more about TransferEngine? You are more than welcome to check TransferEngine in Appendix session 🚀
NCCL vs. TransferEngine
NCCLTransferEngineDeployment Complexity✅ No additional dependency.❌ Additional library mooncake is needed.Overhead of Transfer Setup✅ Building communication groups takes hundreds of milliseconds➖ Registering memory regions to RDMA channel may take several seconds, but can be overlapped with other initialization phases.Non-disturbing to GPU workload❌ Tensor transfer will launch CUDA kernels.✅ No CUDA kernels launched for transferring weights.
We evaluated the performance of launching a new SGLang instance equipped with eight NVIDIA H20 GPUs, while loading the DeepSeek-R1 model from different sources.
Registering the memory region can be overlapped with other initialization phases to further optimize total boot-up time.
Industrial Practice
In the previous sections, we demonstrated how to manually configure seed instances for Tensor R-Fork within SGLang server arguments. However, this manual approach is impractical in real-world industrial deployment, where identifying available seed instances requires significant operational overhead.
To address this challenge, we propose Tensor R-Fork Planner[4], a cluster scheduler designed to orchestrate source instance metadata. The Planner tracks critical information, including:
Model compatibility: Which model is currently running on the instance.
Service health status: Whether the instance is healthy and suitable as a seed instance.
Upon completion of its initialization, each instance registers itself with the Planner, providing its model metadata and parallelism configuration. When a new instance boots up, it first queries the Planner to identify an eligible seed instance that matches both its model and parallelism strategy. If a compatible seed instance is found, the new instance loads weights directly from the seed; otherwise, it falls back to the default load format.
Future Work
The practice of R-Fork opens up more imaginative possibilities: the key concept of R-Fork is that it enables all SGLang instances to act as data storage centers for other instances. Starting from weight tensors, we will manage additional tensors through Tensor R-Fork mechanism in the future, allowing GPU clusters to function not only as computing centers but also as storage centers.
Acknowledgements
Ant Group DeepXPU Team: Anqi Shen, Tianyu Zhou, Zehuan Li, Tiwei Bie, Mingliang Gong, Jianfeng Tan
Multi-backend support: TE supports multiple backends, including RDMA (with GPUDirect), NVLink, GDS, and TCP. It can intelligently identify the best backend per request, so that the highest performance could be reached.
Direct RDMA reads: using the published addresses and rkeys, the destination performs RDMA operations (typically RDMA READ) directly into its own pre-allocated GPU buffers, leveraging GPU-Direct RDMA so that no host-device or device-host intermediate copies are required.
Non-disturbing: TE performs pure NIC-driven transfers that avoid launching CUDA kernels on the source GPU.
Lifecycle & housekeeping: TE maintains the lifetime of registrations until tensors are evicted or the process exits.
Concurrency & flow control: TE coordinates concurrent reads (from one or many destinations) and can apply throttling or rate limits to avoid saturating instance’s NIC or impacting inference latency.
Known limitation in the current TransferEngine implementation:
Memory registration (register_mr) is slow: This is due to the RDMA driver. If you have any insights or solutions to this issue, we would be truly grateful to hear from you. We value diverse perspectives and are keen to explore innovative approaches together.
Ensures non-disturbing operation for inference services.
For instance, when applied to Deepseek-R1 model, the loading time is reduced from several minutes to mere seconds, while local disk and/or DRAM storage usage is reduced by ~600GB and inference service quality maintains during model transfers.
Background
As the scale LLM services and the size of model weights continue to expand, the cold-start time of SGLang instances has become a critical bottleneck for production efficiency. Among the cold-start phases, weight loading remains the most time-consuming task.
Taking Deepseek-R1 as an example, loading weights from local disk typically takes several minutes, while loading from remote storage systems can take up to tens of minutes. As model sizes continue to grow exponentially, the time required for initialization and data transfer will likely worsen.
How can we optimize weight loading performance? The most straightforward approach is to maximize the bottleneck bandwidth in the weight data flow. The data flow of commonly-used model loading approaches in the industry and their associated bottleneck bandwidths are as follows:
Load weights fromData FlowBottleneckremote storage centerremote storage -> remote Ethernet NIC -> Ethernet -> local Ethernet NIC -> local DRAM -> local GPU memoryNVMe/Ethernet NIClocal diskdisk -> DRAM -> GPU memoryNVMelocal DRAMDRAM -> GPU memoryPCIe
Can we exploit higher-bandwidth data flows for transferring tensors? The answer is YES — inter-node device-to-device interconnect offers hundreds of gigabytes per second of throughput. However, the critical question remains: How can we fully leverage this interconnect's bandwidth for efficient weight loading in SGLang?
To address this challenge, we have developed a novel weight-loading framework called Tensor R-Fork (stands for Tensor Remote Fork), which reduces Deepseek-R1 model loading time to mere seconds and is already production-ready.
Design
The core concept of Tensor R-Fork[0] is to leverage GPU-Direct RDMA for constructing a peer-to-peer (P2P) weight storage architecture.
The performance of data transfer using traditional method is low, because there is always bottleneck in the entire path, whose bandwidth is much smaller than inter-node device-to-device interconnect. From the data flow analysis, we observe that weight tensors are stored on each GPU and can be transmitted directly between nodes via GPU-direct RDMA.
To maximize the utilization of RDMA-capable NIC's bandwidth, we design a per GPU-pair data transfer strategy: a local GPU directly transfers data to/from its paired remote GPU. This design effectively bypasses the PCIe bottleneck between GPU and CPU, enabling high-throughput communication without relying on CPU or host memory. The data flow of loading weights from remote SGLang instance is the following:
Implementation
To make every running instances act as the source of model weights for any new instance requiring the same model—while minimizing (or ideally eliminating) disruption to the inference services of running instances—we implemented the framework with two backend options: NCCL and TransferEngine. Consider a running instance A (referred to as the source instance) and a new instance B to be booted (destination instance). Below, we will explain the implementation of weight transfer mechanisms using these two backends in detail.
NCCL backend
When using NCCL as the backend[1], the process involves two stages:
Establishing communication groups between source and destination instances.
Transferring weights from the source instance to the destination instance via these groups.
During destination instance initialization, it sends an HTTP request to the designated source instance to initiate communication group creation. Each TPWorker of destination instance establishes a NCCL communication group with its corresponding TPWorker of source instance (i.e.source rank 0 pairs with destination rank 0, etc.). Each communication group consists of exactly two members: the source TPWorker and destination TPWorker.
Once communication groups are established, each source TPWorker broadcasts its weights tensor located on GPU memory through the group using NCCL broadcast. The destination TPWorker receives the weights directly into its GPU memory without any intermediate memory copies.
While NCCL serves as Tensor R-Fork backend by leveraging GPU-Direct RDMA, it does have a critical limitation: weight transfer disrupts the source instance's inference service, due to two key factors:
Communication group establishment: The source instance must actively participate in creating communication groups.
CUDA kernel interference: The NCCL broadcast mechanism triggers CUDA kernel execution, which competes for GPU resources and introduces latency spikes during generation tasks.
TransferEngine backend
To achieve non-disturbing weight transfer, we introduce an alternative backend: TransferEngine, which leverages GPU-Direct RDMA for efficient data movement[2]. TransferEngine (TE) is a lightweight RDMA-based transfer runtime that runs alongside each TPWorker on the source instance and exposes GPU-resident weight tensors to remote readers without invoking CUDA kernels on the source.
During source SGLang instance initialization:
Each TPWorker (tensor parallel worker) spawns a TransferEngine instance.
TransferEngine registers the GPU memory addresses of its weights with the RDMA channel.
When initializing the destination instance:
It sends an HTTP request to retrieve the source instance's TransferEngine metadata, including RDMA keys mapped to the corresponding GPU memory addresses.
Using these RDMA keys, the destination instance directly loads weights from the source's GPU memory without interrupting the source instance's ongoing services.
*Want to learn more about TransferEngine? You are more than welcome to check TransferEngine in Appendix session 🚀
NCCL vs. TransferEngine
NCCLTransferEngineDeployment Complexity✅ No additional dependency.❌ Additional library mooncake is needed.Overhead of Transfer Setup✅ Building communication groups takes hundreds of milliseconds➖ Registering memory regions to RDMA channel may take several seconds, but can be overlapped with other initialization phases.Non-disturbing to GPU workload❌ Tensor transfer will launch CUDA kernels.✅ No CUDA kernels launched for transferring weights.
We evaluated the performance of launching a new SGLang instance equipped with eight NVIDIA H20 GPUs, while loading the DeepSeek-R1 model from different sources.
Registering the memory region can be overlapped with other initialization phases to further optimize total boot-up time.
Industrial Practice
In the previous sections, we demonstrated how to manually configure seed instances for Tensor R-Fork within SGLang server arguments. However, this manual approach is impractical in real-world industrial deployment, where identifying available seed instances requires significant operational overhead.
To address this challenge, we propose Tensor R-Fork Planner[4], a cluster scheduler designed to orchestrate source instance metadata. The Planner tracks critical information, including:
Model compatibility: Which model is currently running on the instance.
Service health status: Whether the instance is healthy and suitable as a seed instance.
Upon completion of its initialization, each instance registers itself with the Planner, providing its model metadata and parallelism configuration. When a new instance boots up, it first queries the Planner to identify an eligible seed instance that matches both its model and parallelism strategy. If a compatible seed instance is found, the new instance loads weights directly from the seed; otherwise, it falls back to the default load format.
Future Work
The practice of R-Fork opens up more imaginative possibilities: the key concept of R-Fork is that it enables all SGLang instances to act as data storage centers for other instances. Starting from weight tensors, we will manage additional tensors through Tensor R-Fork mechanism in the future, allowing GPU clusters to function not only as computing centers but also as storage centers.
Acknowledgements
Ant Group DeepXPU Team: Anqi Shen, Tianyu Zhou, Zehuan Li, Tiwei Bie, Mingliang Gong, Jianfeng Tan
Multi-backend support: TE supports multiple backends, including RDMA (with GPUDirect), NVLink, GDS, and TCP. It can intelligently identify the best backend per request, so that the highest performance could be reached.
Direct RDMA reads: using the published addresses and rkeys, the destination performs RDMA operations (typically RDMA READ) directly into its own pre-allocated GPU buffers, leveraging GPU-Direct RDMA so that no host-device or device-host intermediate copies are required.
Non-disturbing: TE performs pure NIC-driven transfers that avoid launching CUDA kernels on the source GPU.
Lifecycle & housekeeping: TE maintains the lifetime of registrations until tensors are evicted or the process exits.
Concurrency & flow control: TE coordinates concurrent reads (from one or many destinations) and can apply throttling or rate limits to avoid saturating instance’s NIC or impacting inference latency.
Known limitation in the current TransferEngine implementation:
Memory registration (register_mr) is slow: This is due to the RDMA driver. If you have any insights or solutions to this issue, we would be truly grateful to hear from you. We value diverse perspectives and are keen to explore innovative approaches together.