Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs
1️⃣ Introduction: Deploying the Most Advanced Open-Source MoE Model
Kimi K2 is currently the most advanced open-source Mixture-of-Experts (MoE) model available.
Released by Moonshot AI in 2025, it features:
1 trillion total parameters
32 billion activated parameters per token
384 experts with dynamic routing
Multi-head Latent Attention (MLA) for long context support
Kimi K2 achieves strong performance in frontier knowledge, math, and coding, and is optimized for agentic tasks—not just answering questions but taking multi-step actions.
Moonshot AI open-sourced two versions:
Kimi-K2-Base: The foundation model for research and fine-tuning
Kimi-K2-Instruct: A post-trained model for general-purpose chat and agentic applications
For more details, please refer to the official Kimi K2 release.
Why Large-Scale Deployment Matters
Large-scale deployment fully leverages hardware capabilities and reduces costs given the model’s architecture.
Serve More Requests, Faster: Higher throughput, lower latency, more concurrent sessions, and shorter queues.
Lower $/Token: Saturate hardware and amortize model load; efficiency improves at scale.
However, the large-scale deployment of trillion-scale MoE models present unique challenges:
Computational sparsity in MoE layers necessitates large batch sizes to make matrix operations compute-intensive. Large-scale Expert Parallelism (EP) scales parallelism strategies across more GPUs, aggregates requests from multiple devices, reduces per-GPU memory pressure, and frees up VRAM for larger KV caches—effectively increasing batch size.
Cross-node communication takes a large amount of time and requires optimizations
Sparse expert activation leads to load imbalance
Efficient deployment of Kimi K2 on 128 H200 GPUs requires rethinking both system design and deployment workflows.
In this blog, we explain how we solved this problem using OME and SGLang.
2️⃣ Background: From DeepSeek R1 to Kimi K2
In May 2025, we published Deploying DeepSeek R1 with PD Disaggregation and Large-Scale EP, where we demonstrated:
Prefill-Decode (PD) Disaggregation to separate compute-heavy and latency-sensitive tasks
Large-Scale Expert Parallelism (EP) to handle MoE routing across 96 GPUs
5× throughput improvement compared to vanilla tensor parallelism on H100s
At the same time, our OME blog introduced model-driven deployment, solving the operational gap between:
ML Engineers, who design complex serving strategies
Production Engineers, who need simple and reliable deployments
The OME insight—the model should drive deployment, not vice-versa—proved productive for scaling to Kimi K2’s 1T-parameter architecture. This transition required adapting DeepSeek’s PD Disaggregation and EP to Kimi K2’s 384 experts while maintaining high performance.
For Kimi K2, we combined the strengths of OME and SGLang to create an optimized, scalable deployment pipeline.
Model-Driven Deployment with OME
OME (Open Model Engine) simplifies the deployment of advanced models like Kimi K2 by abstracting away the complexity of parallelism, sharding, scaling, and runtime configuration. With a declarative configuration model, OME enables production teams to deploy and manage large models without manual tuning or custom scripting.
OME Installation
Install OME directly from the OCI registry using the following commands:
Note: You may download the YAML file and customize the path field to specify where the model should be stored locally. OME will download the model directly from Hugging Face with optimized parallelism and automatically verify the artifact checksum to ensure integrity.
Installing the Kimi K2 latest SGLang Serving Runtime
With these declarative resources in place, OME will automatically handle model downloading, runtime orchestration, and endpoint provisioning—enabling scalable, production-grade inference for the Kimi K2 model family.
Interacting with the Model
This command forwards local port 8080 to model on port 80:
Leave this running in one terminal. It will route your local http://localhost:8080 to the SGlang router. After the port-forward is active, run this in a second terminal:
curl -s -X POST http://localhost:8080/generate \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer None' \ -d '{ "text": "The future of AI is", "max_new_tokens": 50, "temperature": 0.7 }'
curl -s -X POST http://localhost:8080/generate \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer None' \ -d '{ "text": "The future of AI is", "max_new_tokens": 50, "temperature": 0.7 }'
OME Advantages & PD + DeepEP + Router Insights
OME (Open Model Engine) offers a declarative, production-ready framework for deploying large models like Kimi K2. It abstracts the complexities of GPU topology, distributed configuration, and runtime tuning—eliminating the need for custom orchestration logic. With a single ClusterServingRuntime definition, teams can launch optimized multi-node inference workloads at scale.
This configuration demonstrates a powerful setup leveraging Prefill-Decode (PD) disaggregation and Large Scale EP, enabling:
Disaggregated scaling of prefill and decode workloads with independent resource control
Low-latency decode via deepep-mode=low_latency and token-aware dispatch tuning
Advanced expert routing with ep-dispatch-algorithm=dynamic and enable-eplb
RDMA acceleration for high-throughput kv-cache transfer
The deployment is orchestrated by a lightweight SGLang Router, which provides:
Dynamic service discovery for prefill and decode nodes via label selectors
Auto-scaling capabilities independent of engine and decoder workloads
Least-privilege routing model—ideal for secure production environments
Optimized load balancing tailored for disaggregated serving patterns
Together, OME and the SGLang Router form a robust foundation for large-scale, low-latency, and maintainable inference infrastructure.
Prefill-Decode Disaggregation
We separate inference into two independent components:
StageRolePrefillHandles large prompt ingestion (e.g., 2000-token inputs). This is compute-bound and benefits from large batch parallelism.DecodeHandles autoregressive generation (e.g., 100-token outputs). This is latency-sensitive and optimized for high-throughput outputs.
Prefill and Decode are deployed as independent services, each scaled and optimized separately.
Large-Scale Expert Parallelism (EP)
Kimi K2 activates a subset of 384 experts per token. We implemented:
96 redundant experts on decode nodes to balance MoE routing
NUMA-aware GPU grouping for optimal NVLink and PCIe utilization on H200 clusters
This design minimizes load imbalance and ensures even GPU utilization across the 128-card cluster.
4️⃣ Performance: 2000-Input, 100-Output Benchmark
We benchmarked Kimi K2 using a typical LLM serving workload on 128 H200 GPUs with 1P1D (4 nodes/P and 12 nodes/D):
We use the same benchmark setup as in the DeepSeek R1 deployment blog as an example. Longer output for agentic scenarios will be future work.
Note: The prefill-to-decode ratio is workload-dependent. We prioritized decode nodes to maximize the KV Cache pool size, which is critical for scaling batch size to 480.
Cluster-Level Performance (128 × H200 GPUs)
MetricValuePrefill Throughput224k tokens/sec (4 P Nodes)Decode Throughput288k tokens/sec (12 D Nodes)Cost per 1M Output Tokens~$0.21(H200 $2.3/hour)
Despite Kimi K2’s larger MoE and more complex routing, our deployment achieves:
Balanced expert activation, using expert-parallel load balancer (EPLB)
High throughput per GPU by applying SGLang’s specific optimizations for DeepSeek V3 architecture to H200
The next step involves evaluating and optimizing long-context scenarios. As K2 is a model designed for agentic tasks, it has been reported that the average input length in such scenarios can range from 30,000 to 50,000 tokens.
5️⃣ Conclusion: Trillion-Scale Inference at Scale
By combining OME, SGLang, PD Disaggregation, and Large-Scale Expert Parallelism, we deployed Kimi K2 on 128 H200 GPUs, achieving:
Cost-effective large-scale inference (~$0.21 per 1M output tokens on H200) is available for short-context scenarios, with ongoing efforts to optimize the long-context scenarios.
Simplified deployment workflows with model-driven configuration
All components of this deployment are fully open-source and reproducible. We welcome the community to build on this work.
This deployment was made possible not only by open collaboration between Mooncake and the SGLang community, but also through the generous infrastructure support from NVIDIA DGX Cloud. NVIDIA provided the SGLang team with access to 128 H200 GPUs via DGX Cloud, enabling us to accelerate the deployment of Kimi K2 from model release to production-grade inference very quickly. As a result, organizations can now leverage SGLang to serve Kimi K2 at scale, unlocking advanced reasoning capabilities with state-of-the-art performance.
Acknowledgments
We would like to express our heartfelt gratitude to the following teams and collaborators:
Mooncake Team: Boxin Zhang, Shangming Cai, Mingxing Zhang, and colleagues.
SGLang Team and community: Simo Lin, Jingyi Chen, Qiaolin Yu, Yanbo Yang, Yineng Zhang, and many others.
We extend our thanks to the MoonshotAI Team—including Shaowei Liu, Zhengtao Wang, Weiran He, Xinran Xu, and others—for their support in tuning the big beautiful model K2.
Further Reading
Deploying DeepSeek R1 with PD Disaggregation and Large-Scale EP
Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs
1️⃣ Introduction: Deploying the Most Advanced Open-Source MoE Model
Kimi K2 is currently the most advanced open-source Mixture-of-Experts (MoE) model available.
Released by Moonshot AI in 2025, it features:
1 trillion total parameters
32 billion activated parameters per token
384 experts with dynamic routing
Multi-head Latent Attention (MLA) for long context support
Kimi K2 achieves strong performance in frontier knowledge, math, and coding, and is optimized for agentic tasks—not just answering questions but taking multi-step actions.
Moonshot AI open-sourced two versions:
Kimi-K2-Base: The foundation model for research and fine-tuning
Kimi-K2-Instruct: A post-trained model for general-purpose chat and agentic applications
For more details, please refer to the official Kimi K2 release.
Why Large-Scale Deployment Matters
Large-scale deployment fully leverages hardware capabilities and reduces costs given the model’s architecture.
Serve More Requests, Faster: Higher throughput, lower latency, more concurrent sessions, and shorter queues.
Lower $/Token: Saturate hardware and amortize model load; efficiency improves at scale.
However, the large-scale deployment of trillion-scale MoE models present unique challenges:
Computational sparsity in MoE layers necessitates large batch sizes to make matrix operations compute-intensive. Large-scale Expert Parallelism (EP) scales parallelism strategies across more GPUs, aggregates requests from multiple devices, reduces per-GPU memory pressure, and frees up VRAM for larger KV caches—effectively increasing batch size.
Cross-node communication takes a large amount of time and requires optimizations
Sparse expert activation leads to load imbalance
Efficient deployment of Kimi K2 on 128 H200 GPUs requires rethinking both system design and deployment workflows.
In this blog, we explain how we solved this problem using OME and SGLang.
2️⃣ Background: From DeepSeek R1 to Kimi K2
In May 2025, we published Deploying DeepSeek R1 with PD Disaggregation and Large-Scale EP, where we demonstrated:
Prefill-Decode (PD) Disaggregation to separate compute-heavy and latency-sensitive tasks
Large-Scale Expert Parallelism (EP) to handle MoE routing across 96 GPUs
5× throughput improvement compared to vanilla tensor parallelism on H100s
At the same time, our OME blog introduced model-driven deployment, solving the operational gap between:
ML Engineers, who design complex serving strategies
Production Engineers, who need simple and reliable deployments
The OME insight—the model should drive deployment, not vice-versa—proved productive for scaling to Kimi K2’s 1T-parameter architecture. This transition required adapting DeepSeek’s PD Disaggregation and EP to Kimi K2’s 384 experts while maintaining high performance.
For Kimi K2, we combined the strengths of OME and SGLang to create an optimized, scalable deployment pipeline.
Model-Driven Deployment with OME
OME (Open Model Engine) simplifies the deployment of advanced models like Kimi K2 by abstracting away the complexity of parallelism, sharding, scaling, and runtime configuration. With a declarative configuration model, OME enables production teams to deploy and manage large models without manual tuning or custom scripting.
OME Installation
Install OME directly from the OCI registry using the following commands:
Note: You may download the YAML file and customize the path field to specify where the model should be stored locally. OME will download the model directly from Hugging Face with optimized parallelism and automatically verify the artifact checksum to ensure integrity.
Installing the Kimi K2 latest SGLang Serving Runtime
With these declarative resources in place, OME will automatically handle model downloading, runtime orchestration, and endpoint provisioning—enabling scalable, production-grade inference for the Kimi K2 model family.
Interacting with the Model
This command forwards local port 8080 to model on port 80:
Leave this running in one terminal. It will route your local http://localhost:8080 to the SGlang router. After the port-forward is active, run this in a second terminal:
curl -s -X POST http://localhost:8080/generate \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer None' \ -d '{ "text": "The future of AI is", "max_new_tokens": 50, "temperature": 0.7 }'
curl -s -X POST http://localhost:8080/generate \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer None' \ -d '{ "text": "The future of AI is", "max_new_tokens": 50, "temperature": 0.7 }'
OME Advantages & PD + DeepEP + Router Insights
OME (Open Model Engine) offers a declarative, production-ready framework for deploying large models like Kimi K2. It abstracts the complexities of GPU topology, distributed configuration, and runtime tuning—eliminating the need for custom orchestration logic. With a single ClusterServingRuntime definition, teams can launch optimized multi-node inference workloads at scale.
This configuration demonstrates a powerful setup leveraging Prefill-Decode (PD) disaggregation and Large Scale EP, enabling:
Disaggregated scaling of prefill and decode workloads with independent resource control
Low-latency decode via deepep-mode=low_latency and token-aware dispatch tuning
Advanced expert routing with ep-dispatch-algorithm=dynamic and enable-eplb
RDMA acceleration for high-throughput kv-cache transfer
The deployment is orchestrated by a lightweight SGLang Router, which provides:
Dynamic service discovery for prefill and decode nodes via label selectors
Auto-scaling capabilities independent of engine and decoder workloads
Least-privilege routing model—ideal for secure production environments
Optimized load balancing tailored for disaggregated serving patterns
Together, OME and the SGLang Router form a robust foundation for large-scale, low-latency, and maintainable inference infrastructure.
Prefill-Decode Disaggregation
We separate inference into two independent components:
StageRolePrefillHandles large prompt ingestion (e.g., 2000-token inputs). This is compute-bound and benefits from large batch parallelism.DecodeHandles autoregressive generation (e.g., 100-token outputs). This is latency-sensitive and optimized for high-throughput outputs.
Prefill and Decode are deployed as independent services, each scaled and optimized separately.
Large-Scale Expert Parallelism (EP)
Kimi K2 activates a subset of 384 experts per token. We implemented:
96 redundant experts on decode nodes to balance MoE routing
NUMA-aware GPU grouping for optimal NVLink and PCIe utilization on H200 clusters
This design minimizes load imbalance and ensures even GPU utilization across the 128-card cluster.
4️⃣ Performance: 2000-Input, 100-Output Benchmark
We benchmarked Kimi K2 using a typical LLM serving workload on 128 H200 GPUs with 1P1D (4 nodes/P and 12 nodes/D):
We use the same benchmark setup as in the DeepSeek R1 deployment blog as an example. Longer output for agentic scenarios will be future work.
Note: The prefill-to-decode ratio is workload-dependent. We prioritized decode nodes to maximize the KV Cache pool size, which is critical for scaling batch size to 480.
Cluster-Level Performance (128 × H200 GPUs)
MetricValuePrefill Throughput224k tokens/sec (4 P Nodes)Decode Throughput288k tokens/sec (12 D Nodes)Cost per 1M Output Tokens~$0.21(H200 $2.3/hour)
Despite Kimi K2’s larger MoE and more complex routing, our deployment achieves:
Balanced expert activation, using expert-parallel load balancer (EPLB)
High throughput per GPU by applying SGLang’s specific optimizations for DeepSeek V3 architecture to H200
The next step involves evaluating and optimizing long-context scenarios. As K2 is a model designed for agentic tasks, it has been reported that the average input length in such scenarios can range from 30,000 to 50,000 tokens.
5️⃣ Conclusion: Trillion-Scale Inference at Scale
By combining OME, SGLang, PD Disaggregation, and Large-Scale Expert Parallelism, we deployed Kimi K2 on 128 H200 GPUs, achieving:
Cost-effective large-scale inference (~$0.21 per 1M output tokens on H200) is available for short-context scenarios, with ongoing efforts to optimize the long-context scenarios.
Simplified deployment workflows with model-driven configuration
All components of this deployment are fully open-source and reproducible. We welcome the community to build on this work.
This deployment was made possible not only by open collaboration between Mooncake and the SGLang community, but also through the generous infrastructure support from NVIDIA DGX Cloud. NVIDIA provided the SGLang team with access to 128 H200 GPUs via DGX Cloud, enabling us to accelerate the deployment of Kimi K2 from model release to production-grade inference very quickly. As a result, organizations can now leverage SGLang to serve Kimi K2 at scale, unlocking advanced reasoning capabilities with state-of-the-art performance.
Acknowledgments
We would like to express our heartfelt gratitude to the following teams and collaborators:
Mooncake Team: Boxin Zhang, Shangming Cai, Mingxing Zhang, and colleagues.
SGLang Team and community: Simo Lin, Jingyi Chen, Qiaolin Yu, Yanbo Yang, Yineng Zhang, and many others.
We extend our thanks to the MoonshotAI Team—including Shaowei Liu, Zhengtao Wang, Weiran He, Xinran Xu, and others—for their support in tuning the big beautiful model K2.
Further Reading
Deploying DeepSeek R1 with PD Disaggregation and Large-Scale EP