TurboServe:高效经济地服务流式视频生成
阅读原文· arxiv.org流式视频生成需在用户会话中逐块渐进生成视频,面临会话持续时间异质性和用户需求时间异质性两大挑战。TurboServe是首个专为此设计的服务系统,将服务形式化为在线调度问题,联合协调会话放置与GPU资源调配。其闭环调度算法包含迁移感知放置控制器(跨GPU重平衡会话以降低每块最大延迟)和负载驱动自动缩放控制器(根据工作负载调整GPU预算)。运行时通过合并块处理、GPU-CPU卸载和NCCL迁移实现决策。在生数科技生产轨迹上,最多64块NVIDIA B300 GPU的评估显示,相比基线,最坏情况每块延迟降低37.5%,总GPU运营成本平均降低37.2%。
Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.