Yicheng Xiao

Wenxun Dai

Xinran Qin

Lin Song

Maoquan Zhang

Hang Xu

Yitong Li

Guohui Zhang

Yuan Zhang

Xuying Zhang

Tommy Zhang

Jianlong Yuan

Peihao Li

Shuai Lu

Siming Fu

Chuyang Zhao

Xin Han

Jie Huang

Wenbo Li

Guoqing Ma

Xiaojuan Qi

Joy Future Academy, JD

Abstract

Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train–inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.

Corresponding author. See Sec.

for the full author list.

1 Introduction

Instruction-guided video editing is evolving from an offline post-production tool into a continuously available visual capability. Recent diffusion-based editors support increasingly precise and preference-aligned image manipulation [24, 18], as well as diverse video transformations including object manipulation, appearance modification, background replacement, and stylization [23, 15, 34, 9, 19]. Meanwhile, efficient latent representations and selective token processing have reduced the cost of high-resolution video synthesis and localized video editing [25, 31]. This progress opens up applications in live broadcasting, video communication, interactive entertainment, and real-time content creation, where edited frames must be produced as the source video arrives. Unlike editing a predefined short clip, however, streaming video editing must jointly provide causal output, low response latency, bounded computation and memory, and stable editing quality over an unknown duration.

Most high-quality video editors are designed for offline, fixed-length inputs. They process an entire clip using bidirectional or global temporal interactions and emit the result only after iterative denoising has completed [36, 29, 39]. Although full temporal context benefits short-clip coherence, dependence on future frames prevents causal output. Moreover, temporal tokens, attention states, and activation memory grow with the input length, while bidirectional computation limits the reuse of cached states [33, 37]. Applying an offline editor independently to consecutive clips is not a satisfactory alternative: it repeatedly processes overlapping context and can introduce visible discontinuities at clip boundaries. A practical streaming editor must instead reuse a bounded temporal state and maintain nearly constant incremental cost as the stream grows.

Causalizing an offline editor alone does not solve this problem. During training, an autoregressive model typically conditions on clean target history, whereas at inference time it consumes its own imperfect predictions. This train–inference mismatch causes reconstruction errors, color deviations, and appearance changes to propagate through the generated history and accumulate into long-term drift [14, 11]. The challenge is particularly acute for video editing: in addition to temporal coherence, every output chunk must remain aligned with the current source chunk, preserve untargeted identity, geometry, motion, and background content, and consistently apply the requested transformation. Generated history promotes continuity but may propagate errors, whereas the source condition preserves fidelity but may weaken editing consistency if not properly balanced. Furthermore, the few-step generation required for real-time throughput introduces an additional gap between the iterative, classifier-free-guided training target and the single-branch model used at deployment. Streaming video editing therefore requires causal adaptation, source-faithful few-step distillation, and explicit optimization against long-horizon error accumulation.

We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing. The model consists of an MLLM-based condition encoder, a causal video VAE, and a multimodal diffusion transformer, and supports both instruction-guided video-to-video editing and reference-conditioned image-and-video-to-video editing. Starting from a strong bidirectional editor trained through a progressive image and video generation-and-editing curriculum, we convert the model into a chunk-wise causal editor. Attention is bidirectional within each chunk and causal across chunks. A sliding temporal window retains a fixed number of recent chunks together with the first chunk as a global sink, thereby bounding the temporal state and per-chunk computation independently of stream duration. We first train the causal editor with clean-history teacher forcing and subsequently replace the clean history with detached, model-generated estimates through resampling forcing [11]. This adaptation exposes the model to the history distribution encountered during deployment and reduces the discrepancy between training and autoregressive inference.

To enable real-time inference without sacrificing editing quality, we introduce Source-Anchored Distribution Matching Distillation (SA-DMD), which distills the iterative diffusion process into a two-step generator. SA-DMD guides the real-score teacher independently along the text-conditioning and source-fidelity axes, using each temporally aligned source chunk to counteract drift from imperfect autoregressive history. The source-aware guidance is applied only to the distillation target and absorbed into the generator, allowing the deployed model to preserve source fidelity with a single conditional branch. We further introduce Long-Horizon Autoregressive Distillation, which performs segmented optimization over extended rollouts and directly supervises states affected by accumulated autoregressive errors while keeping training memory bounded. Together, chunk-wise autoregressive adaptation, SA-DMD, and long-horizon optimization form a unified training framework that addresses causal generation, source drift, few-step acceleration, and long-term stability. To support systematic evaluation under sustained streaming, we also construct LongV2VBench, a long-video editing benchmark covering representative global and local editing tasks.

As illustrated in Figure 1, JoyAI-Video-Edit supports diverse instruction-guided transformations while preserving subject identity, motion, spatial structure, and regions unrelated to the requested edit. The edited appearance remains coherent as new source frames continuously arrive, demonstrating the model’s ability to maintain both editing consistency and source fidelity over extended streams. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short- and long-video editing. By applying SA-DMD, we reduce the diffusion denoising process to only two steps while retaining high editing quality and temporal consistency. Together with bounded-history KV caching, FP8 quantization, and an optimized VAE pipeline, this two-step generator enables the complete system to perform end-to-end video editing at approximately 30 FPS on a single Nvidia B200 GPU.

2 Related Work

2.1 Video Editing

Image editing foundations. Recent image editors have extended instruction-guided manipulation toward continuous photographic control and human-preference alignment. CamEdit enables continuous control over camera parameters, while HP-Edit introduces preference-oriented post-training for real-world editing [24, 18]. These advances strengthen spatial controllability and perceptual quality, but do not address the causal temporal modeling required for streaming video editing.

Offline video editing. Instruction-guided diffusion editors combine text, source videos, and optional visual references to support diverse spatial and semantic transformations [7, 20, 12, 9, 19, 1, 17, 4, 3]. Many high-quality systems are formulated for fixed clips and coordinate edits through bidirectional or global temporal interactions. This formulation is effective for offline processing, whereas streaming deployment requires causal emission and reusable temporal state [36, 29, 39].

Efficiency-oriented work reduces video diffusion cost through compressed latent representations or sparse computation. Turbo2K targets efficient high-resolution video synthesis through a highly compressed latent space, whereas YOSE selects mask-relevant tokens for efficient video object removal [25, 31]. These techniques address complementary computational bottlenecks but do not provide a general framework for open-ended instruction-guided streaming editing.

Streaming video editing. Streaming methods replace full-clip processing with causal or incremental computation. StreamDiffusionV2 develops a continuous framework for interactive video generation [10]. SANA-Streaming subsequently combines a streaming V2V architecture, long-video training, and system optimization [39], while LiveEdit transfers an offline editor to a causal model and reuses computation through an autoregressive mask cache [29]. Xmax X2.0 further targets real-time manipulation of live camera input [32]. Despite enabling practical streaming, these methods often trade model capacity and computational complexity for real-time latency, limiting editing quality and versatility. Maintaining editing consistency and instruction following over long streams also remains challenging.

2.2 Long Video Generation

Diffusion Forcing combines sequence prediction with diffusion by assigning independent noise levels to temporal elements [5], and subsequent systems generate videos causally from reusable or bounded histories [13, 28, 6]. Self Forcing reduces the train–inference discrepancy by explicitly unrolling model-generated autoregressive trajectories [14]. Because this rollout is sequential, it is formulated as post-training and relies on a few-step generator and truncated gradients to control cost, which complicates scaling to larger backbones and longer training horizons. Resampling Forcing instead constructs model-induced histories through sequential resampling within causal diffusion training [11]. Nevertheless, objectives evaluated predominantly on short trajectories may underrepresent errors that emerge only after repeated history reuse.

Few-step distillation addresses the complementary problem of inference efficiency. Distribution Matching Distillation (DMD) compresses iterative diffusion sampling [35] and has been adapted from bidirectional video diffusion to causal autoregressive generation [36]. Causal Forcing further emphasizes that distillation should account for the architectural and state-distribution mismatch between bidirectional teachers and causal students [40]. For long-duration generation, LongLive-Series combines bounded causal attention with long-horizon tuning [33, 8], while Helios and Vidu S1 explore alternative routes to long-duration stability and real-time throughput [37, 38]. These methods cannot be directly applied to streaming video editing, which requires balancing conditioning on the source video, generated history, and current chunk to maintain long-term editing consistency. Without such a balance, errors in previously edited chunks are repeatedly propagated and accumulated throughout the autoregressive rollout.

3 Model

3.1 Model Architecture

As illustrated in Figure 3, JoyAI-Video-Edit is a unified autoregressive diffusion model designed for video editing. It consists of a multimodal large language model (MLLM), a causal video variational autoencoder (VAE), and a multimodal diffusion transformer (MM-DiT). Our model supports both V2V and IV2V editing under different input conditions.

MLLM. Given the first frame of the source video and its corresponding editing instruction, the MLLM jointly processes the visual and textual inputs to extract condition tokens. The first frame provides the appearance and semantic context of the source video, while the instruction specifies the intended transformation. The resulting condition tokens encode both the source-aware visual information and the editing intent, providing semantic guidance to the MM-DiT throughout the denoising process.

Causal VAE. The causal video VAE encodes the source video into latent sequences and maps the optional reference image into the same latent space. It uses a spatiotemporal compression ratio of , corresponding to a temporal compression factor of and spatial compression factors of . Thus, each latent frame represents eight video frames.

MLLM. Given the first frame of the source video and its corresponding editing instruction, the MLLM extracts condition tokens that capture the source content and the intended edit.

3.2 Foundation Training

T2V training. We first conduct progressive T2I pretraining at resolutions of and to establish text–visual alignment, scene composition, and high-resolution appearance priors. The resulting T2I checkpoint is then used to initialize T2V training, during which temporal modeling is introduced while the T2I objective is retained to prevent degradation of the learned spatial generation capability.

Training begins with low-resolution videos at 12 and 24 fps and subsequently increases both the spatial resolution and frame rate. This progressive curriculum allows the model to acquire motion dynamics and long-range temporal composition without abruptly disrupting the spatial priors inherited from T2I pretraining. After large-scale pretraining, we perform supervised fine-tuning (SFT) on higher-quality data at the target resolution of , followed by continual training (CT) to further consolidate visual quality, motion coherence, and prompt alignment.

Bidirectional V2V training. Starting from the T2V checkpoint, we first introduce I2I supervision to learn instruction-conditioned transformation and content preservation while retaining the T2V and T2I objectives. The resulting image-editing checkpoint then initializes bidirectional video-editing training. We jointly optimize video-to-video (V2V) and image-and-video-to-video (IV2V) supervision together with the preceding tasks [15]. V2V extends instruction-conditioned editing over time, whereas IV2V additionally propagates appearance or identity cues from reference images.

Specifically, given a source video , an editing instruction represented by the conditional tokens , an optional reference image, and the edited target , the causal video VAE produces the source, reference, and target latents , , and defined above. The reference latent is omitted for V2V samples without a reference image. For a sampled noise level and Gaussian noise , we construct the noisy target and its flow target as

(1)

The bidirectional editor is trained with

(2)

Bidirectional target attention exposes the model to the complete edited clip, providing a quality-oriented editing initialization. Bidirectional target attention establishes source preservation and temporally coherent editing before the streaming constraint is introduced. It therefore serves as the starting point for the causal autoregressive adaptation described next.

媒体内容 · 前往原文查看

Phase	Task	Resolution	fps	LR	Volume
Stage 1: Text-to-Image Pretraining
Pretrain	T2I		–		4.3B
Pretrain	T2I		–		793M
Stage 2: Text-to-Video Training
Pretrain	T2V/T2I		12/24		370M
Pretrain	T2V/T2I		24		108M
SFT	T2V/T2I		24		30M
CT	T2V/T2I		24		6.5M
Stage 3: Image-Editing Training
SFT	I2I/T2V/T2I		-		3.2M
Stage 4: Bidirectional Video-Editing Training
SFT	V2V/IV2V/I2I/T2V/T2I		24		5.3M
CT	V2V/IV2V/I2I/T2V/T2I		24		1.1M

4 JoyAI-Video-Edit

4.1 Chunk-wise Autoregressive Adaptation

We adapt the bidirectional model into a causal paradigm that generates the edited latent sequence chunk by chunk. Each video editing pair is split along the temporal axis into aligned source and target chunks of a fixed size, with one latent frame per chunk in our implementation. We adopt a chunk-wise attention mechanism with bidirectional attention within each chunk and causal attention across chunks. Consequently, each target chunk can only leverage information from the current and preceding chunks, allowing the model to decode streams online without seeing future frames. To bound the per-step computation and memory, we restrict cross-chunk attention to a sliding window. An active chunk attends to a fixed number of recent history chunks and to the first chunk, which is retained as a global sink. The window keeps the attention context constant regardless of video length, while the sink serves as a persistent anchor for long-horizon generation. During training, we pack the noised active chunk together with the source tokens , the condition tokens , the optional reference tokens , and the clean history tokens into a single sequence, and apply an attention mask that governs the visibility of each group, as illustrated in Figure 3. The active chunk attends bidirectionally to its source and condition tokens, and causally to the in-window history and the global sink; future chunks are masked, and remains globally visible. Following Diffusion Forcing [5], each target chunk is assigned an independent noise level during training with a masked flow-matching objective. Teacher forcing on clean history is stable to optimize but mismatches streaming inference, where the model consumes its own imperfect and drifting predictions [14]. Following Resampling Forcing [11], we replace the clean history with an on-policy estimate, where each historical chunk is regenerated via a single-step denoising rollout and detached from gradient computation. As a result, training is performed under a history distribution closer to that encountered during inference, alleviating the train–test distribution mismatch to some extent.

4.2 Source-Anchored Distribution Matching Distillation

To achieve real-time throughput, we distill the model into a few-step generator using Distribution Matching Distillation (DMD) [35] within an autoregressive framework. The system comprises a causal generator , a trainable fake-score model , and a frozen real-score model . Operating under a shared-backbone LoRA configuration, all three models are initialized from the weights of the previous stage. This standard DMD objective pulls the student toward the teacher via a mode-seeking reverse-KL divergence, guided by the discrepancy between real and fake scores. However, during extended rollouts, the generator increasingly relies on its own imperfect history, causing errors to compound into source drift and hallucinations. To mitigate this, we introduce Source-Anchored DMD (SA-DMD), which anchors the teacher to the temporally aligned source chunk. Specifically, we apply classifier-free guidance (CFG) to the real score along the independent axes of text condition and source fidelity:

(3)

where the source-free prediction omits the aligned source latent , and omits the text condition. This formulation renders the guided teacher a source-sharpened posterior, with acting as a hyperparameter to balance history continuity against source fidelity. Following the DMD formulation, we perturb the rollout to noise level and convert the guided real and fake velocities into clean data predictions. The generator is then updated to minimize the normalized difference between these predictions, effectively pulling the generated distribution toward the source-anchored target. Concurrently, the fake score is trained via flow-matching regression on the detached generator samples. By restricting source-anchoring strictly to the training target, the source-fidelity control is distilled directly into the generator. Consequently, the deployed model achieves high fidelity via a single conditional forward pass.

4.3 Long-Horizon Autoregressive Distillation

Short rollouts fail to capture the compounded errors typical of long-horizon inference, which relies heavily on self-generated history [33]. To expose the distillation to these deep states without incurring out-of-memory (OOM) errors from retaining the full computational graph, we perform segmented optimization over extended -chunk rollouts. Specifically, we divide the sequence into shorter consecutive clips, compute the SA-DMD backward pass per clip, and clear the graph before generating the next. Gradients are accumulated across clips for a single optimizer step, successfully reflecting the full horizon’s gradient within a bounded memory footprint. When target rollouts exceed the available source video length, we extend the conditioning through a dynamic mirror looping strategy (alternating forward and reversed sequences). This method helps preserve temporal continuity without materializing duplicated tensors, thereby avoiding the abrupt semantic shifts often caused by simple cyclic repetition. Furthermore, bounded-window causal attention ensures that KV-cache usage and per-chunk compute remain constrained during deployment, allowing the model to scale efficiently to longer sequences.

5 Real-Time Deployment

At deployment, the incoming video stream is divided into consecutive eight-frame chunks . To minimize inference overhead, we employ FP8 quantization, operator fusion, and computation-graph compilation throughout the deployment pipeline. Each chunk is encoded by the causal VAE, edited by the few-step DiT using the current condition and cached history, and immediately decoded without waiting for future frames. After generation, the clean key-value (KV) states are cached for subsequent chunks, while pseudo encoder provides the single context frame in the VAE’s -frame formulation. The cache retains the first chunk as a global sink and a sliding window of recent chunks, bounding both memory and per-chunk computation for open-ended streams.

Figure 5 shows the stage-wise runtime on a single Nvidia B200 GPU. VAE encoding, DiT denoising, and VAE decoding take 22, 185, and 19 ms per chunk, respectively, yielding a request-to-response latency of 226 ms. Clean KV-cache construction and pseudo encoding add 31 and 9 ms, making the complete 266-ms cycle equivalent to 30.1 FPS. Reduced-precision execution, compiled and autotuned VAE paths, startup warm-up, and memory reuse further reduce runtime overhead. Together with few-step distillation and bounded KV reuse, the 16B model supports real-time editing at approximately 30 FPS on a single Nvidia B200 GPU.

6 Data

Text-to-Video Data. We construct the T2V training corpus from diverse image and video sources covering people, lifestyle, entertainment, nature, objects, and urban scenes, as illustrated in Figure 7. For images, we apply sharpness and quality assessment, black-border detection, and both perceptual-hash- and embedding-based deduplication. For videos, we further filter samples according to visual quality, aesthetic score, motion magnitude, camera stability, and temporal validity, while removing blurred, rotated, presentation-style, and otherwise corrupted videos.

To avoid overrepresenting frequent concepts, we cluster the filtered data and downsample dominant categories while retaining long-tail concepts as much as possible. Since aesthetic filtering may favor static videos, motion-based filtering is additionally applied to preserve samples with meaningful and stable dynamics. For high-quality fine-tuning data, automatic filtering is followed by manual inspection to remove videos containing prominent text, broken visual structures, implausible motion, or violations of basic physical consistency.

Video Editing Data. High-quality paired video-editing data are substantially more difficult to collect at scale. We therefore transfer mature image-editing supervision to video editing using image-to-image (I2I) and reference-to-image (R2I) data from JoyAI-Image [27].

As illustrated in Figure 6, paired editing videos are constructed through two complementary pipelines. First, a representative keyframe is selected from the source video and edited according to the instruction. The edited keyframe and source video are then provided to an image-and-video-to-video model, which propagates the edit across the video while preserving the original motion and unedited content. Second, paired videos can be generated from an original image and its edited counterpart using latent-shared I2V generation. The two branches share early denoising latents to maintain consistent motion and composition, and are conditioned on different images during later denoising to introduce the desired edit.

The generated pairs are filtered according to visual quality, editing correctness, content preservation, and temporal consistency. An MLLM subsequently compares the source and edited videos and refines their editing instructions. As shown in Figure 7, the resulting data cover local and global edits, as well as subject addition, replacement, and removal. Local edits mainly involve subjects, backgrounds, and specific regions, whereas global edits include style, tone, and motion transformations.

7 Experiments

7.1 Evaluation Setup

We compare JoyAI-Video-Edit with both streaming and offline video editors. The streaming baselines include StreamDiffusionV2 [10], SANA-Streaming [39], LiveEdit [29], and the closed-source XMax-X2.0 [32]. The offline baselines comprise the open-source VACE [15], OpenVE-Edit [12], UniVideo [30], OmniWeaving [21], Kiwi-Edit [20], VInO [7], and Bernini-R [3], together with the commercial systems PixVerse V6 [22], Runway Aleph [26], Kling-3.0 Omni [17], and Kling-O1 [16].

7.2 Automatic Evaluation

Short-video editing. We first evaluate short-video editing on the open-source OpenVE-Bench [12]. Following its five-category V2V protocol, a Gemini multimodal judge scores each edited video from 1 to 5 in terms of global style, local change, background change, local removal, and local addition. We compare against both streaming editors and strong offline systems, with results summarized in Table 2.

媒体内容 · 前往原文查看

Method	Params.	Resolution	Overall	Global Style	Local Change	Background Change	Local Remove	Local Add
Closed-source Methods
PixVerse V6 [22]	—		3.05	3.02	4.10	2.23	2.82	3.09
Runway-Aleph [26]	—		3.45	2.62	4.18	4.16	2.78	3.49
Kling-3.0 Omni [17]	—		3.64	4.03	4.15	3.20	3.46	3.36
Kling-O1 [16]	—		3.62	3.38	4.44	3.23	3.32	3.74
Open-source Methods
VACE [15]	14B		1.57	1.49	1.55	2.07	1.46	1.26
OpenVE-Edit [12]	5B		2.49	3.16	2.36	2.98	1.85	2.15
Lucy-Edit [9]	5B		2.22	2.27	1.57	3.20	1.75	2.30
ICVE [19]	13B		2.18	2.22	1.62	2.57	2.51	1.97
DITTO [2]	14B		2.13	4.01	1.68	2.03	1.53	1.41
Kiwi-Edit [20]	5B		3.02	3.64	2.64	3.83	2.63	2.36
VInO [7]	8B		3.32	4.34	2.54	3.73	3.22	2.77
Bernini-R [3]	27B		3.72	4.16	4.47	3.25	3.88	2.89
Streaming Video Editing
StreamDiffusionV2 [10]	1.3B		1.23	1.48	1.35	1.01	1.27	1.05
SANA-Streaming [39]	2B		2.62	3.48	2.29	3.20	2.27	1.88
LiveEdit [29]	1.3B		2.00	2.18	2.73	2.05	1.55	1.51
Xmax-X2.0 [32]	—		1.87	2.47	2.09	1.63	1.73	1.41
JoyAI-Video-Edit (Ours)	16B		3.60	3.62	4.47	2.90	4.06	2.97

JoyAI-Video-Edit achieves an overall score of 3.60, outperforming SANA-Streaming [39], LiveEdit [29], XMax-X2.0 [32], and StreamDiffusionV2 [10] by 0.98, 1.60, 1.73, and 2.37 points, respectively. It ranks first among streaming methods in four of the five categories, with particularly large gains in local change and local removal. Despite causal inference, its overall performance is comparable to strong offline editors such as Kiwi-Edit [20], Bernini-R [3], Kling-3.0 Omni [17], and Kling-O1 [16]. It also matches the best local-change score and achieves the highest local-removal score among all compared methods, substantially narrowing the quality gap between streaming and offline video editing.

Long-video editing. Existing video-editing benchmarks predominantly consist of clips shorter than 10 seconds, making them insufficient for assessing error accumulation, temporal degradation, and processing efficiency over sustained input streams. We therefore construct LongV2VBench, a one-minute video-editing benchmark comprising 229 tasks across five representative categories: background change, global style editing, local addition, local modification, and local removal. Figure 8 presents the benchmark composition and representative examples. We evaluate JoyAI-Video-Edit against existing streaming editors on LongV2VBench and report both editing quality and full-pipeline throughput.

媒体内容 · 前往原文查看

Method	Resolution	Throughput (FPS)	Overall	Background Change	Global Style	Local Add	Local Change	Local Remove
StreamDiffusionV2 [10]		18.07	1.21	1.08	1.71	1.18	1.11	1.03
SANA-Streaming [39]		14.51	1.64	1.19	2.02	1.50	1.72	1.85
LiveEdit [29]		15.45	1.23	1.10	1.34	1.33	1.34	1.04
XMax-X2.0 [32]		20.90	1.71	1.36	2.07	1.64	2.08	1.40
JoyAI-Video-Edit (Ours)		30.19	3.30	2.49	3.85	3.10	4.09	2.99

As shown in Table 3, JoyAI-Video-Edit achieves an overall score of 3.30 and ranks first in all five editing categories. It exceeds the strongest baseline, XMax-X2.0 [32], by 1.59 points overall, demonstrating that the quality advantage of our method remains consistent over one-minute videos.

Full Throughput measures the end-to-end processing speed of the complete editing pipeline. JoyAI-Video-Edit reaches 30.19 FPS at , which is 44.4% faster than XMax-X2.0 at a substantially lower resolution and more than twice as fast as SANA-Streaming [39] at a comparable resolution. This efficiency benefits from bounded-history causal inference, which reuses a fixed temporal state and maintains stable per-chunk computation as the input stream grows.

7.3 Human Evaluation

We conduct pairwise human evaluation between anonymized outputs from JoyAI-Video-Edit and each competing method. Evaluators select JoyAI-Video-Edit, the competing method, or a tie according to overall editing quality. All evaluation examples are single-shot videos shorter than 10 seconds.

As shown in Figure 9, JoyAI-Video-Edit receives 90%, 87%, 81%, and 87% of the preference votes against LiveEdit [29], SANA-Streaming [39], XMax-X2.0 [32], and StreamDiffusionV2 [10], respectively. Against Bernini-R [3], JoyAI-Video-Edit receives 48% of the votes, while Bernini-R receives 44%. Kling-3.0 Omni [17] and Seedance 2.0 [4] each receive 56% of the preference votes in their corresponding comparisons. Overall, JoyAI-Video-Edit is substantially preferred over existing streaming editors and achieves human-evaluated quality competitive with strong offline systems.

7.4 Deployment Efficiency

We further evaluate the latency and throughput of the complete deployment pipeline and its two principal neural components, the diffusion transformer and the video autoencoder. Full latency includes the end-to-end execution time of the entire editing pipeline. FPS is computed over the complete evaluated sequence and reports the mean per-frame throughput rather than instantaneous peak performance. All values are reported after deployment optimization and rounded to two decimal places.

媒体内容 · 前往原文查看

Method	Resolution	Full Pipeline		DiT		VAE
		Latency (s)	FPS	Latency (s)	FPS	Latency (s)	FPS
StreamDiffusionV2 [10]		4.48	18.07	1.99	40.64	2.19	37.06
LiveEdit [29]		5.24	15.45	2.98	27.16	2.17	37.26
SANA-Streaming [39]		5.58	14.51	1.49	54.36	2.99	27.12
JoyAI-Video-Edit (Ours)		2.68	30.19	2.18	37.21	0.405	200.00

As shown in Table 4, JoyAI-Video-Edit achieves the lowest full-pipeline latency of 2.68 seconds and the highest end-to-end throughput of 30.19 FPS. Its full throughput is 1.67 times that of StreamDiffusionV2, 1.95 times that of LiveEdit, and 2.08 times that of SANA-Streaming. For SANA-Streaming, we benchmark its publicly available open-source deployment, which is slower than the implementation reported in the original paper [39]. Although SANA-Streaming has a faster standalone DiT in our evaluation, JoyAI-Video-Edit provides the best complete-system performance and reaches 200.00 FPS for VAE processing, substantially reducing the autoencoding bottleneck.

The end-to-end improvement results from the combined quantization and deployment design described in Section 5. Reduced-precision execution lowers the cost of both the transformer and autoencoder, while compiled VAE paths with autotuning accelerate encoding and decoding. Pipelined execution overlaps host-side processing with device computation, and startup warm-up, persisted compilation artifacts, and retained memory pools eliminate repeated compilation and allocation overhead. Together with causal KV caching that avoids recomputing previously generated content, these optimizations translate component-level acceleration into higher full-pipeline throughput.

7.5 The Analysis of Proposed Strategies

媒体内容 · 前往原文查看

SA-DMD	LHAD	Overall	Background Change	Global Style	Local Add	Local Change	Local Remove
		2.81	2.45	3.61	1.97	3.43	2.58
✓		3.23	2.49	4.24	2.74	4.00	2.67
	✓	3.06	2.60	3.56	2.49	3.94	2.70
✓	✓	3.30	2.49	3.85	3.10	4.09	2.99

As shown in Table 5, SA-DMD drives the largest single performance boost, raising global style from 3.61 to 4.24 and local change from 3.43 to 4.00. This confirms that anchoring the teacher to the aligned source chunk successfully curbs the source drift that otherwise erodes appearance fidelity. Meanwhile, LHAD offers a complementary increase to an overall score of 3.06, particularly benefiting background change and local removal by stabilizing the late, error-accumulated states of extended rollouts. Ultimately, enabling both strategies yields the best overall score of 3.30 and dominates all three local-editing tasks, demonstrating that source anchoring and long-horizon optimization are highly complementary.

7.6 Visualization Results

JoyAI-Video-Edit transforms video editing from a fixed-clip post-production process into a continuous and interactive workflow. Figure 10 illustrates its capabilities across global style transfer, scene and background replacement, localized object manipulation, appearance modification, and motion-aware editing. Across these diverse cases, the model follows the editing instruction while preserving subject identity, spatial layout, motion trajectories, and regions unrelated to the requested change. More importantly, these properties remain stable as new frames continuously arrive, allowing an edit to persist coherently over an unbounded video stream. The causal pipeline also provides immediate visual feedback, enabling users to apply or revise instructions during playback without waiting for the complete video to be processed. This capability extends video editing beyond conventional offline production to live broadcasting, real-time digital humans, interactive entertainment, game content creation, telepresence, and embodied simulation (See Figure 11). For creators, continuous preview and instruction-level control can reduce repeated rendering and shorten the iteration cycle; for platforms and enterprises, persistent stream processing makes personalized visual effects and adaptive content transformation practical at scale. JoyAI-Video-Edit therefore provides not only a faster editing tool, but also a new interaction paradigm in which video content can be modified continuously as it is captured, transmitted, or consumed.

8 Authors

Core Contributors
Yicheng Xiao¹¹1Equal contribution., Wenxun Dai¹¹1Equal contribution., Xinran Qin¹¹1Equal contribution., Lin Song¹¹1Equal contribution.²²2Project leader., Maoquan Zhang, Hang Xu, Yukang Chen, Yitong Li, Guohui Zhang, Yuan Zhang, Xuying Zhang, Tommy Zhang, Jianlong Yuan, Peihao Li, Shuai Lu, Siming Fu, Chuyang Zhao, Xin Han, Jie Huang, Wenbo Li, Guoqing Ma, Wei Huang, Xiaojuan Qi, Haoyang Huang³³3Corresponding authors., Nan Duan³³3Corresponding authors.

Contributors⁴⁴4Contributors are listed in alphabetical order.
Anson Li, Bi Cheng, Bin Li, Bo Wang, Boyang Li, Dongyan Yang, Feice Huang, Fengyuan Shi, Fan Lin, Haoran Li, Haoyu Wu, Hu Yu, Jia Shi, Jiachen Liu, Jiaqi Wang, Jiawei Li, Jianhui Liu, Jiayi Deng, Jiangmiao Pang, Junhao Zhuang, Kangliang Chen, Libing Fang, Lichen Ma, Liang Lin, Lingjie Li, Lixin Wang, Nan Jiang, Nanhua Lai, Nick, Pan Wang, Qingyi Si, Qiushi Yang, Ruofan Lv, Shaonan Wu, Tong He, Wanyan Yu, Wei Tang, Xiaoxiao Huo, Xing Pan, Xi Yang, Xuan Yang, Yan Li, Yanfei Tang, Yichen Wang, Yijun Yang, Yipeng Sun, Yuhang Li, Yujia Liang, Yue Ma, Zeyue Xue, Zuopeng Dong

References

• Alibaba ATH Team [2026] Alibaba ATH Team. Happyhorse 1.0. https://happy-horse.ai/docs/basic-usage, 2026. Accessed: 2026-07-28.
• Bai et al. [2026] Bai, Q., Wang, Q., Ouyang, H., Yu, Y., Wang, H., Wang, W., Cheng, K. L., Ma, S., Zeng, Y., Liu, Z., Xu, Y., Shen, Y., and Chen, Q. Scaling instruction-based video editing with a high-quality synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 37971–37981, 2026.
• Bernini Team [2026] Bernini Team. Bernini: Latent semantic planning for video diffusion. arXiv preprint arXiv:2605.22344, 2026.
• ByteDance Seed [2026] ByteDance Seed. Seedance 2.0: Advancing video generation for world complexity. arXiv preprint arXiv:2604.14148, 2026.
• Chen et al. [2024] Chen, B., Martí Monsó, D., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024.
• Chen et al. [2025] Chen, G., Lin, D., Yang, J., Lin, C., Zhu, J., Fan, M., Zhang, H., Chen, S., Chen, Z., Ma, C., et al. Skyreels-v2: Infinite-length film generative model. arXiv preprint arXiv:2504.13074, 2025.
• Chen et al. [2026a] Chen, J., He, T., Fu, Z., Wan, P., Gai, K., and Ye, W. Vino: A unified visual generator with interleaved omnimodal context. arXiv preprint arXiv:2601.02358, 2026a.
• Chen et al. [2026b] Chen, Y., Wang, L., Huang, W., Yang, S., Zhang, B., Xiao, Y., Chu, R., Mao, W., Hu, Q., Liu, S., et al. Longlive-2.0: An nvfp4 parallel infrastructure for long video generation. arXiv preprint arXiv:2605.18739, 2026b.
• DecartAI Team [2025] DecartAI Team. Lucy edit: Open-weight text-guided video editing. Technical report, DecartAI, 2025. URL https://d2drjpuinn46lb.cloudfront.net/Lucy_Edit__High_Fidelity_Text_Guided_Video_Editing.pdf.
• Feng et al. [2025] Feng, T., Li, Z., Yang, S., Xi, H., Li, M., Li, X., Zhang, L., Yang, K., Peng, K., Han, S., et al. Streamdiffusionv2: A streaming system for dynamic and interactive video generation. arXiv preprint arXiv:2511.07399, 2025.
• Guo et al. [2025] Guo, Y., Yang, C., He, H., Zhao, Y., Wei, M., Yang, Z., Huang, W., and Lin, D. End-to-end training for autoregressive video diffusion via self-resampling. arXiv preprint arXiv:2512.15702, 2025.
• He et al. [2025] He, H., Wang, J., Zhang, J., Xue, Z., Bu, X., Yang, Q., Wen, S., and Xie, L. Openve-3m: A large-scale high-quality dataset for instruction-guided video editing. arXiv preprint arXiv:2512.07826, 2025.
• Henschel et al. [2025] Henschel, R., Khachatryan, L., Poghosyan, H., Hayrapetyan, D., Tadevosyan, V., Wang, Z., Navasardyan, S., and Shi, H. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2568–2577, 2025.
• Huang et al. [2025] Huang, X., Li, Z., He, G., Zhou, M., and Shechtman, E. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025.
• Jiang et al. [2025] Jiang, Z., Han, Z., Mao, C., Zhang, J., Pan, Y., and Liu, Y. Vace: All-in-one video creation and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17191–17202, 2025.
• Kuaishou Technology [2025] Kuaishou Technology. Kling o1 launches as the world’s first unified multimodal video model. https://ir.kuaishou.com/news-releases/news-release-details/kling-o1-launches-worlds-first-unified-multimodal-video-model-0/, 2025. Accessed: 2026-08-03.
• Kuaishou Technology [2026] Kuaishou Technology. Kling ai 3.0 and video 3.0 omni. https://ir.kuaishou.com/news-releases/news-release-details/kling-ai-launches-30-model-ushering-era-where-everyone-can-be, 2026. Accessed: 2026-07-28.
• Li et al. [2026] Li, F., Wang, C., Lei, L., Qiu, Y., Xu, J., Jiang, J., Qin, X., Chen, Z., Song, F., Wang, Z., Pei, R., and Zuo, W. HP-Edit: A human-preference post-training framework for image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 43113–43123, 2026.
• Liao et al. [2025] Liao, X., Zeng, X., Song, Z., Fu, Z., Yu, G., and Lin, G. In-context learning with unpaired clips for instruction-based video editing. arXiv preprint arXiv:2510.14648, 2025.
• Lin et al. [2026] Lin, Y., Liang, G., Zeng, Z., Bai, Z., Chen, Y., and Shou, M. Z. Kiwi-edit: Versatile video editing via instruction and reference guidance. arXiv preprint arXiv:2603.02175, 2026.
• Pan et al. [2026] Pan, K., Tian, Q., Zhang, J., Kong, W., Xiong, J., Long, Y., Zhang, S., Qiu, H., Wang, T., Lv, Z., Wu, Y., Bo, L., Tang, S., and Zhong, Z. Omniweaving: Towards unified video generation with free-form composition and reasoning. arXiv preprint arXiv:2603.24458, 2026.
• PixVerse [2026] PixVerse. Pixverse launches v6, advancing ai video generation across creative and agentic workflows. https://pixverse.ai/en/blog/pixverse-launches-v6-advancing-ai-video-generation, 2026. Accessed: 2026-08-03.
• Qi et al. [2023] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., and Chen, Q. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15932–15942, 2023.
• Qin et al. [2025] Qin, X., Wang, Z., Li, F., Chen, H., Pei, R., Li, W., and Cao, X. CamEdit: Continuous camera parameter control for photorealistic image editing. In Advances in Neural Information Processing Systems, volume 38, pp. 114152–114171, 2025.
• Ren et al. [2025] Ren, J., Li, W., Wang, Z., Sun, H., Liu, B., Chen, H., Xu, J., Li, A., Zhang, S., Shao, B., Guo, Y., and Zhu, L. Turbo2K: Towards ultra-efficient and high-quality 2K video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18155–18165, 2025.
• Runway [2025] Runway. Runway aleph: A new way to edit, transform and generate video. https://runwayml.com/research/introducing-runway-aleph, 2025.
• Song et al. [2026] Song, L., Li, W., Ma, G., Tang, W., Wang, B., Zhang, Y., Yang, Y., Xiao, Y., Liu, J., Zhang, Y., Zhang, G., Zhang, W., Xu, H., Jiang, N., Han, X., Sun, H., Zhang, M., Huang, H., and Duan, N. Awaking spatial intelligence in unified multimodal understanding and generation. arXiv preprint arXiv:2605.04128, 2026.
• Teng et al. [2025] Teng, H., Jia, H., Sun, L., Li, L., Li, M., Tang, M., Han, S., Zhang, T., Zhang, W., Luo, W., et al. Magi-1: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211, 2025.
• Wang et al. [2026] Wang, X., Zhao, C., Zhan, F., and Ma, Y. Liveedit: Towards real-time diffusion-based streaming video editing. arXiv preprint arXiv:2606.26740, 2026.
• Wei et al. [2025] Wei, C., Liu, Q., Ye, Z., Wang, Q., Wang, X., Wan, P., Gai, K., and Chen, W. Univideo: Unified understanding, generation, and editing for videos. arXiv preprint arXiv:2510.08377, 2025.
• Wu et al. [2026] Wu, C., Lei, L., Li, F., Guo, C., Kong, D., Qin, X., Wang, Z., Cheng, M., and Li, C. YOSE: You only select essential tokens for efficient DiT-based video object removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 32926–32935, 2026.
• Xmax AI [2026] Xmax AI. Xmax x2.0: Real-time interactive video model. https://xmax.ai/, 2026. Accessed: 2026-07-28.
• Yang et al. [2025] Yang, S., Huang, W., Chu, R., Xiao, Y., Zhao, Y., Wang, X., Li, M., Xie, E., Chen, Y., Lu, Y., et al. Longlive: Real-time interactive long video generation. arXiv preprint arXiv:2509.22622, 2025.
• Ye et al. [2025] Ye, Z., He, X., Liu, Q., Wang, Q., Wang, X., Wan, P., Zhang, D., Gai, K., Chen, Q., and Luo, W. Unic: Unified in-context video editing. arXiv preprint arXiv:2506.04216, 2025.
• Yin et al. [2024] Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., and Freeman, B. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455–47487, 2024.
• Yin et al. [2025] Yin, T., Zhang, Q., Zhang, R., Freeman, W. T., Durand, F., Shechtman, E., and Huang, X. From slow bidirectional to fast autoregressive video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22963–22974, 2025.
• Yuan et al. [2026] Yuan, S., Yin, Y., Li, Z., Huang, X., Yang, X., and Yuan, L. Helios: Real real-time long video generation model. arXiv preprint arXiv:2603.04379, 2026.
• Zhang et al. [2026] Zhang, J., Jiang, K., Chen, J., Wang, X., Luo, Y., Wang, Y., Chen, D., Li, J., Ye, C., Chen, M., Zhu, H., Zhao, M., Jiang, Y., Huang, Z., Xiang, C., Zheng, K., Wang, H., Wang, X., Jia, Q., Chen, X., Chen, Y., Jiang, Y., Fu, F., Deng, Z., Bao, F., Chen, J., and Zhu, J. Vidu s1: A real-time interactive video generation model. arXiv preprint arXiv:2607.03118, 2026.
• Zhao et al. [2026] Zhao, Y., Pan, Y., He, Q., Yu, J., Chen, J., Ye, T., Liu, H., Xie, E., and Han, S. Sana-streaming: Real-time streaming video editing with hybrid diffusion transformer. arXiv preprint arXiv:2605.30409, 2026.
• Zhu et al. [2026] Zhu, H., Zhao, M., He, G., Su, H., Li, C., and Zhu, J. Causal forcing: Autoregressive diffusion distillation done right for high-quality real-time interactive video generation. arXiv preprint arXiv:2602.02214, 2026.