Chongjian Ge

Hanwen Jiang

Tianyu Wang

Jiuxiang Gu

Yiran Xu

Ziwen Chen

Shaoteng Liu

Jing Shi

Yicong Hong

Zefan Cai

Hailin Jin

Hao Tan

July 25th, 2026

Abstract

Visual generation is entering a token-intensive regime: high-resolution images, long videos, and multimodal context make the quadratic cost of full attention in Diffusion Transformers increasingly prohibitive. Language models underwent a similar transition, but solutions developed for them cannot be transferred directly to visual diffusion models, which must preserve spatiotemporal locality and support bidirectional interactions across modalities. We present Chimera, a hybrid visual diffusion backbone co-designed with a principled scaling recipe. Chimera processes text, images, and video tokens in a single stream with three complementary mechanisms: Kimi Delta Attention (KDA) provides long-context state tracking with efficient computation complexity, interleaved Multi-head Latent Attention (MLA) enables direct global interaction, and modality-aware short convolutions capture local spatiotemporal context. Together, these components enable processing of a unified raster-ordered token sequence free of positional embeddings. Sparse Mixture-of-Experts (MoE) layers further expand model capacity while keeping activated compute under control. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scaling scheme that transfers hyperparameters across width and depth based on each tensor’s functional fan-in and the model’s depth. HeteroP produces a consistently tuned model family, enabling us to fit Chinchilla-style compute-optimal laws for activated model size, training-token count and image–video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments yield three main findings. First, measured by pretraining diffusion loss, our dense backbone achieves 1.7 the compute efficiency of a matched full-attention Wan-2.1 (2B) baseline, while the complete system raises this factor to 7.3. Second, without length-specific fine-tuning, Chimera exhibits zero-shot length extrapolation from 5-second training clips to 30-second videos, with only a degradation in FID over the final five seconds. Third, the fitted scaling laws indicate that compute-optimal image pretraining allocates compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher compute budgets. Together, we hope these findings provide a principled foundation for designing and scaling efficient long-context diffusion architectures.

1 Introduction

Visual generation is becoming a long-sequence problem. A image patchified at already contains roughly 65K visual tokens, while videos, dense conditions, and multi-turn interactions increase the sequence length further [67, 63, 7]. We refer to this as the token-extensive regime, in which sequence length becomes a primary determinant of memory, latency, and training cost. Yet progress in this regime is constrained by two coupled gaps. Architecturally, the dominant Diffusion Transformer still relies on full self-attention [63], whose pairwise interactions scale quadratically with sequence length [81]. Methodologically, visual generation lacks a systematic framework for scaling such architectures: models and optimization recipes are often designed at a particular scale, without a controlled model family for establishing scaling laws that guide how diffusion-pretraining performance depends on model size, token budget, and data composition [50, 103].

Language models entered this regime earlier and offer useful precedents on both fronts. Architecturally, language models have increasingly moved beyond dense attention [81] toward sparse attention [14, 97, 54], compressed key-value representations [69, 1, 52], recurrent or linear sequence operators [45, 75, 29, 94, 47], and sparsely activated capacity [48, 25, 17]. Methodologically, hyperparameter transfer across model scales [93, 18] and compute-optimal scaling laws that relate model capacity, data, and training compute [44, 36] have turned model scaling into a systematic and increasingly predictable process.

However, neither component of the language-model recipe can be transferred directly to visual diffusion training. On the architectural side, visual diffusion must preserve local structure over two- and three-dimensional grids, enable direct interaction among heterogeneous text, image, and video contexts, and remain effective under a denoising objective [33, 35, 43, 8]. On the scaling side, hybrid visual backbones contain tensors whose functional input dimensions grow differently, or do not grow at all, as nominal model width increases, so standard rules based on a single global scaling ratio do not preserve comparable optimization dynamics [93, 18]. Moreover, visual training compute depends not only on the number of tokens, but also on data composition. The challenge is therefore twofold yet coupled: to design an efficient context-modeling architecture tailored to the visual data structure, and to develop a systematic framework for scaling it and deriving its scaling laws.

We present Chimera, a visual diffusion backbone co-designed with a principled scaling framework for token-extensive generation. As shown in Fig. 1, Chimera represents text, images, and videos as a unified token sequence for modeling. At the sequence scale, most layers use Kimi Delta Attention (KDA) [47] as a linear-complexity mechanism for content-adaptive long-range state tracking, while periodic Multi-head Latent Attention (MLA) layers [52] provide direct global token-to-token interaction with compressed key-value representations. At the local scale, we propose modality-aware short convolutions aggregate neighbors along the native axes of each modality before every KDA state update. KDA therefore writes locally enriched features, rather than isolated tokens from the flattened sequence, into its recurrent state. The local relative-offset cues supplied by the convolutions, together with the ordering inherent in KDA’s causal recurrence, make a simple temporal-major raster scan sufficient and allow us to dispense with explicit positional embeddings. Orthogonally, Chimera combines sparse Mixture-of-Experts (MoE), identity hyper-connections (iHC), and sandwich normalization for greater capacity at controlled activated compute and stable signal propagation.

On the training side, to address the challenge of transferring hyperparameters across scales in our heterogeneous architecture [93, 18], we introduce HeteroP, a hyperparameter-transfer parameterization that derives a separate scaling ratio for each tensor from its functional fan-in and model depth. HeteroP translates these tensor-wise ratios into module-specific transfer rules for learning rates, initialization, and residual scaling, allowing a recipe tuned on a small proxy model to transfer across model widths and depths. Together, these rules produce a consistently optimized and mutually comparable family of Chimera models (59M to 19.3B total parameters) without per-scale retuning. This controlled model family enables fitting Chinchilla-style compute-optimal scaling laws [36] over activated parameter count and visual training tokens. We further extend these laws by treating the image–video data mixture as an explicit scaling variable, capturing how the compute-optimal allocation depends on modality composition.

Across three independent estimators, the training-curve envelope, IsoFLOP profiles, and parametric loss-surface fitting, our scaling laws consistently prescribe nearly balanced growth of activated model size and training tokens for image pre-training(Fig. 1-S2), with , while video pre-training shifts modestly toward model capacity at higher compute budgets, with . Guided by the fitted scaling laws, we train a Chimera model with 11B total parameters and 2B activated parameters given our training budget. Under matched training compute (Fig. 1-A1), it reaches the training loss of a full-attention Wan 2.1 (2B) baseline [82] with fewer FLOPs. Chimera is also competitive with strong image generators such as FLUX.1-dev and Z-Image-Turbo [8] on the GenEval and DPG-Bench benchmarks. Notably, our final model is trained only around 600 H100 Days, substantially less than prior method: Z-Image-Turbo, for example, reports a total training budget of approximately 12.4K H100 Days, 20 times of ours.

Beyond guiding the final model allocation, Chimera’s architecture brings concrete advantages in long-sequence generation. Benefiting from its position-embedding-free design, Chimera generates -second videos despite being trained only on -second clips, requiring no length-specific fine-tuning and extending the generation horizon by while increasing frame-wise FID by only , compared with an increase of more than for prior arts [82]. Moreover, compared with the matched MHA/MLA full-attention counterpart, our KDA/MLA backbone supports more than longer sequences within a single  GB GPU (NVIDIA A100-SXM4-80GB) and runs faster at k tokens (Fig. 1-A2). ¹¹1We compare matched KDA/MLA and MHA/MLA backbones with approximately 2B activated parameters, using batch size one, BF16 activations, text tokens, and visual tokens per temporal slice. For the roll-out-cache memory test, we allocate, touch, and retain the cache tensors on device during one complete denoiser forward. FlashAttention does not write the full attention matrix to high-bandwidth memory: it streams blocks of , , and through on-chip SRAM, computes tile by tile, and accumulates the exact output using online softmax. This removes the quadratic attention workspace, but not the quadratic arithmetic. The 24 MHA layers therefore have sequence-dependent BF16 key–value cache memory of , where and are the number and width of the attention heads and . In contrast, the 24 KDA layers retain fixed-size FP32 recurrent states of shape , requiring memory independent of ; this is the state-memory term when denotes the per-head state width. Both variants retain the same eight BF16 MLA latent key–value caches. We report peak allocated CUDA memory on one NVIDIA A100-SXM4-80GB and stop at the first out-of-memory input. For speed, we exclude cache construction, use FlashAttention for MHA, and report the median of three CUDA-synchronized post-warm-up forwards. Together, these results support jointly considering architecture and scaling for visual generation.

Our contributions are threefold. First, we introduce Chimera, a hybrid single-stream visual diffusion backbone for token-extensive generation. It represents text, image, and video as a unified sequence. Our model combines KDA linear attention with periodic MLA-based global attention; uses modality-aware causal short convolutions to support a single raster-order scan without positional embeddings; and adopts MoE layers, identity hyper-connections, and sandwich normalization for capacity and training stability. Second, we develop, to our knowledge, the first comprehensive hyperparameter-transfer parameterization (HeteroP) for heterogeneous visual diffusion backbones. HeteroP allows optimization hyperparameters tuned on a small proxy model to transfer reliably across model widths and depths. Third, we fit Chinchilla-style compute-optimal scaling laws for image and video pre-training over activated model size and visual training tokens, and extend the analysis to the image–video data composition.

2 Chimera Overview

We aim to modernize diffusion models, particularly DiT [63], for the token-extensive visual generation regime. This objective involves two tightly coupled sub-goals: designing architectures that balance efficiency and capacity, and understanding how their advantages evolve with model size, the number of training tokens, and training-data composition. Accordingly, we study architecture design and scaling jointly: the architecture defines a controlled model family, while the scaling analysis determines how model size, training tokens, and training-data composition should be allocated under a fixed compute budget. Section 3 details the architecture, while Sec. 4 presents the scaling solution.

Model Architecture.

We summarize the key design choices of Chimera below, ordered from the overall framework to individual modules:

• •

  Single-stream Multimodal Backbone. Chimera concatenates multimodal tokens from text, images, and video into one sequence and processes them with a single shared backbone. Unlike cross-attention or dual-stream designs [82, 87], this single-stream interface enables direct token-level interaction across modalities without modality-specific branches. It also simplifies scaling: a shared stack over one token sequence inherits language-model practices such as hyperparameter transfer and compute-optimal scaling laws.

• •

  Hybrid Linear and Global Attention. Chimera uses KDA [47] in most layers for long-sequence computation and periodically interleaves MLA layers that restore explicit global token interactions through compressed key-value states. Together, they balance long-context efficiency and global modeling capacity. Unlike prior works that rely on complex scanning patterns for visual modeling [56, 39], Chimera applies linear attention through a single temporal-major raster scan for simple and uniform multimodal processing.

• •

  Minimal Modality-aware Inductive Bias and No Positional Embedding (NoPE). Attention-based models typically use positional embeddings to encode the token positions. For visual data, however, positional embeddings can hinder extrapolation beyond the training length, and factorizing them across spatial (image) and temporal axes (video) requires ad-hoc design choices. The state-tracking property of KDA offers a simple alternative: with fine-grained channel-wise state decay and recurrent delta updates, KDA behaves like local sliding-window attention with an adaptive window length and causal structure, which removes the need for positional embeddings, as observed in recent language models [47]. To adapt this design to multimodal visual data, we retain a single minimal inductive bias: modality-aware short convolutions that serve as a “Canon Layer’ [2] to capture local spatio-temporal dependencies among visual tokens. Order and locality thus come from the scan and the convolution, so the whole backbone processes multimodal tokens through one temporal-major raster scan without positional embeddings.

• •

  Sparse Capacity and Training Stability. For capacity, Chimera adopts sparse MoE activation [25], which increases total parameters while controlling per-token computation. For stability, it pairs identity hyper-connections [90], which stabilize the residual streams, with sandwich normalization [22], which regulates the variance of sub-layer outputs.

Pre-training and Chinchilla Scaling.

We study how Chimera scales during pre-training, the stage in which the model acquires its generative knowledge. We argue that the diffusion pre-training loss should be the reliable metric for measuring knowledge acquisition at this stage, while other vision-related evaluation metrics²²2FID, DPG [40], GenEval [28], etc are better treated as post-training metrics rather than pre-training objectives. This perspective allows us to treat visual pre-training analogously to language-model pre-training and fit compute-optimal scaling laws. We outline the key ingredients below.

• •

  Diffusion Loss as a Pre-training Metric. Under a fixed diffusion objective and noise parameterization, diffusion loss provides a sampler-independent metric for comparing models across scales. Unlike downstream generation metrics such as FID, it does not depend on sampling algorithms, classifier-free guidance scales, or post-training choices. Moreover, diffusion can be interpreted as approximate autoregressive modeling in the frequency domain [20]: denoising from high to low noise levels predicts visual information from coarse to fine scales, which parallels next-token prediction under progressively longer contexts. This motivates its use for a more faithful pre-training metric than downstream generation scores.

• •

  Hyperparameter Transfer and Fair Scaling Comparison. Hyperparameter transfer reduces the cost of hyperparameter search at the target scale and helps every model in a scaling study reach near-optimal performance. We build on P-style transfer [93, 18] across model scales, but Chimera poses an additional challenge: its heterogeneous modules do not share a single global scaling multiplier. We therefore propose HeteroP to replace this global-ratio approximation with a module-specific ratio derived from each parameter group’s functional fan-in, covering KDA, MLA, sparse MoE, identity hyper-connections, and diffusion conditioning. As a result, the data points used to fit scaling curves come from models trained with scale-appropriate and near-optimal hyperparameters, a factor often not explicitly controlled in prior scaling-law studies [36, 50, 86].

• •

  Fitting Chinchilla Scaling Laws. Using this controlled model family, we fit Chinchilla-style laws over activated model parameters and training tokens to predict their compute-optimal allocation under a fixed compute budget. We also use IsoFLOP analysis to estimate how the optimal image-video mixture changes with compute.

3 Chimera Architecture

3.1 Preliminaries and Training Objectives

Chimera follows the rectified-flow framework with -prediction and -loss [51, 55]. Each image or video is encoded by a frozen VAE from [82] with temporal, height, and width compression factors of ; a subsequent patchification layer converts the latents into the clean visual tokens . The text prompt is encoded separately by a frozen T5-style encoder [65], and we denote the resulting text-token sequence by . During training, we sample Gaussian noise and a timestep , and linearly interpolate between and to obtain the noised visual tokens . The velocity target is . We train the denoising model with

(1)

where denotes the trainable parameters and the loss is calculated only at visual-token positions.

3.2 Single-Stream Framework

Chimera adopts a single-stream framework in which text and visual tokens interact directly within one shared backbone. Its input sequence is

(2)

where and are separate trainable projections into the shared hidden space, and padding tokens are removed from before concatenation. The visual tokens are flattened frame by frame in temporal-major raster order, traversing each frame in row-major order, as formalized in Eq. 12. For the diffusion timestep , we construct a sinusoidal embedding and project it to the model dimension, yielding the timestep embedding , which conditions each block through the modulation pathway defined below.

Chimera consists of a stack of Transformer-style blocks, each containing an attention sub-layer followed by a feed-forward sub-layer (FFN). The attention operator is either KDA or MLA, following the periodic schedule (Sec. 3.3). The first and last blocks use dense SwiGLU FFNs [70], while all intermediate blocks use sparse MoE FFNs (Sec. 3.5). Before each KDA update, a modality-aware short convolution (mShortConv) injects local structure (Sec. 3.4). Both sub-layers use sandwich normalization [22] and are wrapped with stream identity hyper-connections (iHC; Sec. 3.6). With a slight abuse of notation, we continue to write for this multi-stream state, whose structure is made explicit in Sec. 3.6. Timestep conditioning enters through adaptive layer normalization (AdaLN) modulation and residual gates. Omitting the layer index, one block computes:

				(3)
				(4)
				(5)
				(6)
				(7)
				(8)
				(9)
				(10)

Here, , , denote the multi-stream states, whereas , , , , . The operators collapse the stream axis from to , whereas distributes each sub-layer update back to the streams. The timestep-conditioned vectors , , , , are broadcast across the token positions. The state becomes the input to the next block. Equation 3 maps to an attention shift , an attention scale , and residual gates and for the two sub-layers. Thus, the attention input receives timestep-conditioned AdaLN modulation [63], while both residual updates are timestep-gated (Eqs. 7 and 10). With parallel residual streams, both sub-layers share the same read–compute–write pattern over the streams. Each sub-layer starts with (Eqs. 4 and 8), which aggregates the streams into one sub-layer input. For attention, this input is pre-normalized and timestep-modulated, passed to the attention operator(Eq. 5 and 6). The resulting update is then gated by , and written back to all streams by (Eq. 7). The FFN sub-layer also repeats this pattern (Eqs. 8–10). Within each sub-layer, Sandwich Normalization places RMSNorm both before () and after () the operator to regulate the variance for training stability. We define the attention operator in Sec. 3.3, the feed-forward operator in Sec. 3.5, and the iHC read/write operators in Sec. 3.6.

After the final block, the residual streams are averaged into one representation (Sec. 3.6), and only visual-token positions enter the output head for rectified-flow velocity prediction. Text tokens condition these predictions through token-level interactions in the shared backbone, while the diffusion loss applies only to visual tokens.

3.3 Hybrid Attention Mechanism with NoPE

Chimera periodically interleaves Kimi Delta Attention (KDA) [47] and Multi-head Latent Attention (MLA) [52], at a KDA-to-MLA ratio. For a sequence of length , each KDA layer requires only sequence complexity because it carries a constant-size recurrent state rather than materializing a full attention map. Each MLA layer retains full bidirectional self-attention with compressed key-value representations. The operators are therefore complementary: KDA provides efficient long-sequence computation, while MLA periodically enables unrestricted bidirectional global interaction. We detail both formulations and the no-positional-embedding (NoPE) design below.

(11)

KDA with a Single Scan Pattern.

We traverse video latents frame by frame, using row-major order within each frame. For a latent grid of temporal length , height , and width , this temporal-major raster order is

(12)

where , , and index the temporal, height, and width dimensions, respectively. An image is treated as a single-frame latent grid. We omit the timestep subscript below for clarity.

After text tokens are prepended to the flattened visual tokens, every KDA head processes using the same scan order. The recurrent state update is causal in this order: at each visual position, the state summarizes the text prefix and previously scanned visual tokens. The update follows the KDA formulation of Kimi Linear [47], with fine-grained channel-wise state decay rather than a single head-wise scalar. For a single head, omitting its index, we have

				(13)
				(14)
				(15)

where , , and denote the query, key, and value vectors at scan position in , , and , respectively, while is the recurrent state after processing that position. The forget gate applies an independent decay to each key channel, while the head-wise scalar controls both the erase and write terms. The discrete scan index ranges over and is distinct from the diffusion timestep . The recurrent state is reset to at each packed-sample boundary. denotes the proposed modality-aware short convolution (Sec. 3.4), which supplies local structure and facilitates context-level information shortcut [2], while allowing the recurrent KDA update to retain a single scan order across modalities.

MLA for Global Context Modeling.

KDA compresses the scanned history into a constant-size recurrent state. This state is efficient to maintain, but its bounded size limits how much information can be recalled exactly from long visual sequences. We therefore interleave a small fraction of full-attention layers to restore direct content-based interactions across all tokens. This hybrid design is more than a cost-capacity compromise. Recent analysis shows that, under standard complexity assumptions, alternating linear-attention and full-attention layers is strictly more expressive than either type alone on formal sequence task: linear attention contributes state tracking and full attention contributes recall, so their composition solves problems that neither solves by itself [57]. To keep these global layers memory-friendly, we instantiate them with MLA, which retains full bidirectional attention while compressing the key-value representations:

				(16)
				(17)
				(18)
				(19)

Here, denotes the number of attention heads. is the compressed key–value latent. Both and are decoded from , whereas is projected directly from and shared across heads. This reduces the key–value cache cost while retaining full bidirectional attention. Unlike standard MLA [52], we leave unrotated (NoPE).

NoPE v.s. RoPE.

Chimera uses no explicit positional encoding (NoPE) in either KDA or MLA layers. To motivate this choice, we first examine what positional encoding contributes to attention. Absent explicit positional modulation, attention is a content-matching operation. That’s to say, its logits are inner products between learned query and key features, without direct dependence on token indices. Positional encodings add position-dependent inductive biases to this content pathway. For RoPE [74], the de facto choice in both language and visual generation models, these biases take three forms:

• •

  Position selection. Some operations must activate at a specific relative token offset. A canonical example is the previous-token head, which serves as a building block of induction heads [23, 61]. RoPE enables such token-index-aware/token-level-shortcut operations by rotating each query-key channel pair at a fixed frequency, allowing trained heads to exploit high-frequency pairs for precise offset selection. By aligning the phases of its highest-frequency pairs, an attention head can make its logit peak at a chosen small offset [3].

• •

  Recency decay. Nearby context should generally receive greater weight than distant context. Positional encoding schemes introduce this bias either explicitly, as with ALiBi’s distance-dependent penalty [64], or implicitly, as with RoPE. Under the conventional frequency spectrum and simplifying assumptions, RoPE’s rotations can induce a distance-decaying envelope on the attention logits. This long-term decay was a key motivation for RoPE [74].

• •

  Multidimensional encoding. Multimodal data has inherently multidimensional positions: text tokens have a one-dimensional index, whereas video tokens use temporal, height, and width coordinates. Axial RoPE assigns rotary query-key pairs to these axes and rotates each subset by its coordinate [85, 95, 82].

All three biases are implemented by rotating content-bearing query-key features, forcing positional and semantic information to share query-key capacity. Trained models mitigate this competition by separating the two across the frequency spectrum [3]. Low-frequency pairs remain nearly rotation-invariant over the relevant context range, so their dot products are governed primarily by token content and preserve distance-robust semantic matching. Short-range position selection, by contrast, are concentrated in the high-frequency pairs.

To characterize this frequency-wise division of labor, we first analyze a pretrained RoPE-based language model, Qwen3-4B [91], whose previous-token heads provide a clear diagnostic of short-range position selection. We use a -token window of natural text. Because RoPE rotates each channel pair independently, every attention logit decomposes into a sum of per-frequency contributions, allowing the role of each frequency to be measured directly. Specifically, we average the queries and keys over the window and, for each rotary channel pair , rotate the query component onto the axis while absorbing both vector norms into the pair amplitude . The resulting pairwise cosine terms sum exactly to the attention logit, agreeing with the directly computed value to within (Fig. 3(a)). We first examine the strongest previous-token head (i.e., layer 0, head 1), for which of queries attend most strongly to the immediately preceding token (Fig. 3(b)). In this head, the highest-frequency pair, which rotates by approximately one radian per token, produces a logit that peaks exactly at offset one: its learned phase places the cosine maximum at . By contrast, the largest-amplitude pairs all lie near zero frequency and contribute almost uniformly across offsets (Fig. 3(c)). These slowly rotating pairs are therefore modulated primarily by content rather than position. The largest such pair rotates by less than across the entire window, yet contributes little in this example because its averaged query and key vectors are nearly orthogonal. We then search across all heads for those that attend exactly tokens into the past, quantified by the fraction of queries whose attention peaks at that offset. Such heads appear only at short distances: the best head attains a fraction of at , but only at . These results indicate that RoPE-based position selection operates almost entirely over short ranges.

We then apply the same analysis to visual generation models, FLUX.2 [4] and Wan2.2 [82], and observe the same division of labor. That’s to say, slowly rotating channel pairs preserve distance-robust semantic matching, whereas rapidly rotating pairs enable short-range position selection (protocol and full results in Appendix C).

From this perspective, RoPE implements each of its three inductive bias through rotary query-key channels, while introducing several limitations. (1) First, position selection consumes substantial attention capacity. In the previous-token head above, a single fast-rotating pair fixes the target offset, while the large-amplitude slow pairs contribute only a flat, offset-independent background. As a result, an entire attention head is effectively dedicated to a fixed -token shift, yet reliable selection remains limited to very short ranges. (2) Second, recency decay is only implicit. The decaying envelope arises as an average property of the summed rotations rather than from an explicit content-adaptive decay mechanism, and can be bypassed by trained heads [3]. (3) Third, layout encoding is hand-designed. The channel partition must be specified manually, each additional modality or positional axis further divides the available channels and reduces the frequency resolution per axis, and the resulting allocation remains fixed at design time.

Chimera instead provides each inductive bias through a dedicated mechanism, none of which consumes attention-channel capacity. Token order is supplied by the recurrent scan itself: KDA processes the sequence in temporal-major raster order, making its state updates inherently order-aware. (1) Position selection is handled by the modality-aware short convolution (Sec.3.4). A depthwise kernel mixes tokens at explicit index offsets, implementing index-aware operations with only a small number of parameters rather than dedicating an entire attention head to them. Related convolutional mechanisms have also been used to inject positional information into vision Transformers [15]. Because the analyses above show that RoPE-based position selection is primarily short-range, a short kernel is sufficient to replace it. (2) Recency decay is intrinsic to KDA. The channel-wise forget gate (Eq. 14) decays the recurrent state at each scan step, linking each token to its predecessors over a learned, content-adaptive window rather than through a fixed spectral envelope. (3) Multidimensional encoding is determined by the structure of the convolution. Text tokens are mixed with a causal 1D kernel along the token index, whereas visual tokens are mixed with a 3D kernel over the temporal-height-width grid. Each modality’s native layout is therefore encoded directly by the operator, without partitioning the channels across positional axes. With these three positional biases assigned to dedicated modules, attention is free to perform its native function: content matching. MLA without RoPE corresponds to the limiting case in which every channel has zero rotary frequency, so its logits depend only on content [3]. The queries and keys in KDA likewise contain no positional phases. This design parallels recent hybrid language models that combine NoPE full attention with gated linear attention [47, 77]. Chimera applies it to multimodal data via modality-aware convolution.

Beyond this disentanglement, removing RoPE also avoids its dependence on training length. At relative offsets beyond those observed during training, nonzero rotary frequencies can produce phase configurations outside the training distribution. When a model trained on 5-second clips is used to generate 10- or 20-second videos, these unseen phase configurations can degrade [46, 3]. By contrast, our three mechanisms introduce no explicit positional phase tied to the training range: the convolution kernels have fixed support, and the same recurrent update is applied at each scan step. This, to some extent, removes one architectural obstacle to zero-shot extrapolation to longer sequences.

Overall, Chimera factors RoPE’s three positional functions into explicit, specialized modules: a short convolution for position selection, gated state decay for recency, and operator structure for layout encoding. Each inductive bias is provided in a direct and learnable form, without consuming attention capacity for simple positional operations, manually partitioning channels, or coupling the model to its training length.

3.4 Modality-aware Short Convolution

A short convolution injects local structure into the token features before each linear-attention update, following common practice in the linear-attention literature [47]. A plain one-dimensional convolution, however, mixes only tokens that are adjacent in the scan order and thus models locality only along a fixed-order axis. To recover the two- and three-dimensional locality of visual data, prior vision models pair such a convolution with multiple hand-designed scan directions [56, 39, 37]. We instead make the convolution modality-aware, so that it matches each modality’s native structure and captures local spatial-temporal correlations directly. This allows Chimera to retain a single scan pattern. Fig. 4 contrasts our modality-aware convolution this single scan pattern with the conventional pairing of a 1D convolution and multi-axis scanning.

Concretely, the modality-aware convolution uses modality-specific views of text and visual tokens while preserving their positions in the packed single-stream interface (Fig. 4). Text tokens are gathered from their packed positions and processed with a causal one-dimensional depthwise convolution over the text sequence. Visual tokens are reshaped into their spatial-temporal grid and processed with a depthwise 3D convolution, which models local correlations along the temporal, height, and width axes at once. An image is handled as the special case of a video with temporal length one. The convolutions are applied independently within each modality segment, so no kernel crosses a text-visual or packed-sample boundary. After convolution, the text and visual outputs are scattered back to their original positions in the packed sequence. Because the convolution itself captures multi-dimensional locality, KDA can keep a single temporal-major scan instead of the multi-directional scans discussed above.

Implemented naively, this gather–convolve–scatter procedure launches multiple kernels and materializes intermediate buffers per layer. We instead implement a fused Triton kernel that realizes the same text and visual convolutions directly over packed storage, avoiding explicit gather and scatter operations. In operator-level microbenchmarks against the unfused implementation on packed image ( and ) and video (-frame) batches, the fused kernel achieves a – forward speedup and a – forward–backward speedup, while reducing peak activation memory during the forward pass by up to .

Along the temporal dimension, the 3D convolution uses causal padding, so that each visual token mixes only with tokens from the current and past frames. We make the short convolution temporally causal to keep its local mixing consistent with KDA’s causal temporal-major scan, for the following reasons:

• •

  Consistency with State Tracking. The KDA operator updates a recurrent state while scanning the sequence, which aligns naturally with the temporal progression of video generation. A non-causal temporal convolution would introduce future-frame information into the local query, key, and value features before each KDA update, weakening this state-tracking interpretation. Causal padding keeps the local visual operator consistent with the causal dynamics of the KDA scan.

• •

  Compatibility with Sequential Generation. A temporally causal convolution also facilitates adapting the pretrained backbone to downstream streaming generation tasks. For example, if the model is later converted into an autoregressive world model, the KDA pathway, including its short convolution, is already causal in time from pretraining, rather than relying on a causal constraint introduced only during post-training [41, 38].

• •

  Performance. Empirically, causal temporal padding in the short convolution has little effect on generation quality compared with the non-causal alternative, so this design choice comes at negligible cost.

In summary, KDA follows a causal temporal-major scan, while the short convolution is causal along the temporal axis. This alignment preserves KDA’s state-tracking interpretation, while the convolution provides modality-specific local structure under a single scan pattern.

3.5 Mixture-of-Experts (MoE)

To enhance the capacity of Chimera without proportionally increasing activated computation per token, we replace the dense FFN with sparse MoE FFNs [25, 17] in all except the first and last blocks. This choice complements the hybrid KDA/MLA design: once the attention cost is reduced, FFN computation accounts for a larger share of per-token computation, so enlarging dense FFNs would erode the efficiency gains from hybrid attention. Sparse activation instead increases the total FFN parameters, and hence the model capacity, while keeping the activated computation per token matched to a dense baseline.

Concretely, an MoE FFN contains routed experts and a router . For each token input , the router produces expert scores, selects a small top- set , and combines only those expert outputs,

				(20)
				(21)

where is the number of experts activated per token, is a non-gradient balancing bias applied only during expert selection, and is the routing weight of expert . In intermediate blocks, instantiates the operator in Eq. 9. We use token-choice routing: each token selects its activated experts independently. Following the fine-grained expert design of DeepSeekMoE [17], each routed expert is a small SwiGLU FFN whose hidden dimension is of the dense FFN hidden width. In our main configuration, every MoE layer has routed experts and activates experts per token, corresponding to an activation ratio of . The total expert parameters are thus enlarged by roughly , while the activated experts together recover exactly the dense FFN hidden width, e.g., in our 2B configuration. We use no shared experts, as we empirically find that the routed experts alone provide sufficient capacity in this visual diffusion setting. The MoE configuration, including expert count and activation ratio, is fixed across experiments, keeping our scaling-study runs directly comparable.

The router is a lightweight two-layer MLP producing sigmoid scores. After top- selection, the selected scores are normalized and scaled by to form the weights . To avoid routing collapse and expert-load imbalance during training, we adopt the auxiliary-loss-free balancing strategy [84, 53]: the non-gradient bias enters only the top- selection, while the weights are always computed from the unbiased router scores. After each training step, the bias of overloaded experts is decreased and that of underloaded experts is increased by a small fixed step, according to the observed expert loads. This keeps the expert load balanced without adding any auxiliary loss to the diffusion objective.

To verify that this strategy indeed balances the load in our visual diffusion setting, we follow [84] and quantify the degree of load balance of an MoE layer by its maximal violation (MaxVio):

(22)

where is the number of tokens routed to expert in a training batch, and is the mean load over the experts, i.e., the per-expert load under perfect balance. MaxVio equals under a perfectly uniform load and reaches in our configuration when all tokens select the same experts. Fig. 5 tracks the batch-level MaxVio, averaged over all MoE layers, for two identical runs. With the balancing bias enabled, MaxVio drops below almost immediately and stabilizes around for the rest of training, i.e., even the most loaded expert receives only about its expected share of tokens. Disabling the bias instead leads to severe imbalance: MaxVio rises above within the first few hundred steps and keeps drifting toward the collapse bound, with the most loaded expert attracting over its expected load. These results confirm that the bias-based mechanism maintains well-balanced expert utilization throughout training without adding a load-balancing term to the diffusion objective.

3.6 Identity Hyper-Connections (iHC)

Deep networks rely on residual connections to preserve signal propagation across depth. Hyper-Connections (HC) [104] generalize this design by expanding the residual state into multiple streams and learning how each sub-layer reads from and writes back to them. mHC [90] further constrains the residual-stream transition to be doubly stochastic, preserving the approximate identity mapping required for stable large-scale optimization. This multi-stream interface is particularly well suited to Chimera. KDA, MLA, modality-aware short convolution, and MoE perform complementary computations and produce updates with different statistics, yet a standard residual block forces every sub-layer to read from and write to the same state. As these heterogeneous updates accumulate with depth, a single stream can restrict how operator-specific information is preserved and recombined. Multiple residual streams provide token-dependent read/write pathways that increase routing capacity and maintain stable feature statistics through an identity path.

We adopt Identity Hyper-Connection (iHC), a simplified variant of mHC that fixes the residual-stream transition to the identity while retaining token-dependent read and write mappings. At the input of the denoising backbone, iHC expands the standard residual state into residual streams by replication, where is the hidden dimension and is the per-token multi-stream state. Stacking these states across token positions yields the sequence-level state used in Eqs. 4–10. After the final block, the streams are averaged back into a single stream. Within each block, before each sub-layer, a read vector aggregates the streams into the sub-layer input,

(23)

which instantiates the operator in Eqs. 4 and 8. Let denote the gated sub-layer update, with for attention (Eq. 7) and for the FFN (Eq. 10). A write vector independently scales and adds the shared update to each residual stream,

(24)

which instantiates the operator. The token-dependent read and write vectors are generated by

(25)

Here, , , and is a learnable scalar. and denote the all-ones and all-zeros vectors in , respectively, and is applied elementwise. This gives and . We initialize and , so suppressing the input-dependent term yields and . Because is initialized to a small value and the streams begin as replicas of the input, each block starts close to a standard residual update. Input-adaptive multi-stream routing then emerges gradually during training. As described in Sec. 3.2, this read–compute–write pattern is applied separately around the attention and the FFN sub-layers, with residual streams in our main model.

The key simplification relative to mHC lies in the residual-stream transition. In the general HC/mHC formulation, the write step additionally mixes the streams themselves through a residual transition matrix , i.e., , where mHC learns a token-dependent and projects it onto the doubly-stochastic manifold with Sinkhorn iterations [90]. iHC instead fixes , yielding the write rule in Eq. 24. This removes the per-token residual mixing matrix and the Sinkhorn projection required by mHC. Beyond maintaining, reading, and writing the residual streams, the coefficient generator in Eq. 25 requires one RMSNorm, one projection, and elementwise gates per sub-layer. For fixed , this routing arithmetic scales linearly with , compared with the quadratic-in- projection cost of attention and FFNs. iHC therefore retains adaptive read and write mappings over multiple residual streams while preserving an exact identity residual transition.

4 Scaling Recipe

This section presents the scaling recipe for Chimera. Sec. 4.1 develops HeteroP, a module-wise hyperparameter-transfer parameterization that yields a consistently tuned model family across width and depth. Section 4.2 then uses this family to fit Chinchilla-style compute-optimal laws over model size and training tokens and extends the analysis to the image-video data ratio.

4.1 Hyperparameter Transfer

Studying the scaling behavior of a new architecture requires training a family of models of increasing scales. This immediately raises a practical question: which hyperparameters should be used at each scale? Re-tuning the learning rate, initialization, and weight decay independently at every scale is prohibitively expensive. Reusing the values tuned at a small scale is cheap but often suboptimal, because the optimal learning rate drifts with model width and depth under standard parameterizations [93, 18]. Hyperparameter transfer resolves this tension by reparameterizing the model and optimizer so that a set of base hyperparameters remains approximately stable across width and depth. One can tune these base values on a small proxy model and then transfer them to larger models

Existing hyperparameter-transfer methods address complementary scaling axes. The maximal-update parameterization (P) provides width-transfer rules and has been validated on Transformers and ResNets [93], while CompleteP extends transfer across depth for Transformers [18]. Recent diffusion-specific work further adapts P to full-attention diffusion Transformers [103]. These methods provide the principles on which we build, but do not specify a complete parameterization for a heterogeneous visual diffusion backbone that jointly combines KDA linear attention, MLA global attention, modality-aware short convolutions, sparse MoE layers, iHC, sandwich normalization, and diffusion timestep conditioning. We call our resulting method HeteroP (Heterogeneous Parameterization).

Why transfer is necessary for a fair scaling study.

Beyond reducing tuning cost, hyperparameter transfer makes a scaling study meaningful. A scaling curve compares the losses of models at different sizes to estimate how performance improves with compute. If each size is trained with scale-mismatched hyperparameters, differences in measured loss may reflect both model scale and suboptimal tuning, obscuring the true effect of scale. For example, a larger model trained with a suboptimal learning rate can even underperform a smaller one, making the resulting curve unreliable for design conclusions. Hyperparameter transfer reduces this confound effect, making differences in loss primarily reflect scale and architecture rather than tuning noise. Earlier diffusion scaling studies do not explicitly control this factor with a width- and depth-transfer parameterization [63, 50]. In Sec. 5.2, we verify that the optimal learning rate stays stable across width and depth under our recipe, but shifts under standard parameterization.

Why heterogeneous backbones require module-wise ratios.

A homogeneous Transformer is often widened by multiplying a common hidden dimension, making a single global width ratio a reasonable shorthand when instantiating transfer rules. In Chimera, “width” is instead a collection of distinct dimensions. A projection may take as input the model hidden state, the MLA KV-compression latent, a KDA head, an MoE expert hidden state, the router state, or the timestep-conditioning state. These dimensions may scale with the model width, grow at different rates, or remain fixed. Applying the global model-width ratio to every tensor would therefore mis-scale parameters whose updates are controlled by another fan-in. HeteroP resolves this mismatch by computing the target-to-proxy ratio separately for every parameter group from the dimension that controls its activation and update scale. The result is a fine-grained, dimension-accurate parameterization that preserves the intended transfer rule for structurally heterogeneous backbones.

HeteroP Parameterization.

Let denote a small proxy model on which the base hyperparameters are tuned, and let denote a target model to which they are transferred. The two models share the same architecture but may differ in width and depth, with and blocks, respectively. For each target parameter group , let denote its proxy counterpart. We write the transfer compactly as:

				(26)
				(27)
				(28)

Here, , , , and denote the base initialization variance, learning rate, weight decay, and AdamW . collects the initialization, optimizer, forward, and residual-scaling settings for , while denotes its functional role. is a module-specific width multiplier determined by the fan-in of and is independent of the number of blocks, whereas is the global depth multiplier determined by the block count. In tuning Chimera, unlike previous works that adopt a single global scale ratio, may be determined by the model hidden width, the MLA KV-compression rank, KDA head width, MoE expert width, router width, or timestep-conditioning width. The depth multiplier is shared because it is determined by the target-to-proxy block-count ratio. Our proxy has model width and depth (M activated parameters, Table 4), while the largest model has model width and depth . Hence for parameter groups whose fan-in is the model width, while other groups receive their own ratios, and . Operationally, HeteroP (i) tunes once on the proxy, (ii) assigns each target tensor a role and computes , and (iii) applies the role-dependent map together with . Eqs. 26–28 define these three ingredients, and Table 1 gives their module-wise instantiation.

媒体内容 · 前往原文查看

Module / Hyperparameter	SP	HeteroP
Input adapters (patch / text / timestep embedding)
Init. Var.
LR (AdamW)
Sandwich RMSNorm (pre-/post-norm gains)
Init. Var.
LR (AdamW)
Hidden weights (KDA/MLA projections, FFN & MoE experts, router)
Init. Var.
LR (AdamW)
Bias LR (AdamW)
WD (AdamW)
Attention residual (KDA, MLA; iHC)
FFN / MoE residual (iHC)
Final norm (pre-readout)
Init. Var.
LR (AdamW)
Diffusion readout head (velocity prediction)
Init. Var.
LR (AdamW)
Forward
AdamW (residual blocks)
AdamW (input adapters & readout)

Width and depth transfer rules.

The HeteroP rules in Table 1 combine module-wise hyperparameter transfer across both width and depth. Under width transfer, hidden matmul weights use initialization variance and learning rate scaled by , together with weight decay scaled by . These initialization and learning-rate corrections preserve width-consistent activation and update scales, while the paired scaling and keeps invariant. Input adapters, normalization gains, biases, and the readout keep their base initialization and learning rate, matching the input/output treatment of P. The readout is additionally multiplied by in the forward pass, and AdamW is scaled by to track the shrinking gradient scale.

HeteroP incorporates the depth correction of CompleteP [18]: each attention or FFN/MoE sub-layer output is scaled by before the corresponding iHC write, keeping the aggregate residual contribution on a depth-independent scale. Residual-block tensors receive the same additional correction to AdamW , while their learning rates require no additional depth correction, as listed in Table 1.

Module-wise assignments.

Mapping HeteroP onto Chimera requires assigning each learnable tensor to a rule. For each parameter group, we identify its functional role and, where applicable, the dimension that determines its fan-in. The resulting assignment is summarized below.

• •

  Input adapters and readout. The patch, text, and timestep embedding projections are input layers, and the velocity-prediction head is the output layer. Both keep the base learning rate and initialization, and the readout additionally uses the forward multiplier. The blockwise timestep-conditioning MLP in Eq. 3, however, follows the hidden rule; its input projection is parameterized by the width of .

• •

  Hybrid attention. All projections that read the hidden state, including the KDA query–key–value projection and the MLA query and key–value down-projections, scale with the model width. The MLA key–value up-projection scales with the KV-compression rank, while the KDA gating and the output projections of both operators scale with the corresponding head widths. The short-convolution kernels and the KDA low-rank gate up-projections have a fixed fan-in per output channel. They therefore use , retaining the base learning rate and weight decay while using the residual-block scaling. The complementary gate down-projections read the hidden state, so they follow the hidden rule.

• •

  FFN and MoE. The up- and gate-projections of dense SwiGLU FFNs and MoE experts scale with the model width, and the down-projections scale with their respective hidden widths. The router input projection likewise follows the model width, while the router hidden width is held fixed across scales. This keeps the routing logits on a consistent scale as the model grows. The routing sparsity is also fixed, as discussed below.

• •

  iHC and Sandwich Normalization. The iHC gains and biases, together with the RMSNorm gains, are control tensors and follow the norm treatment: base initialization, base learning rate, and the hidden . The lightweight projection that produces the iHC read/write vectors reads the concatenated residual streams, whose size grows with the model width, so it follows the hidden rule. This preserves residual stability across width and depth.

What is held fixed when scaling.

When constructing the model family, we vary the module-specific widths and block count represented by and , respectively, while holding the structural settings listed below fixed. The MoE routing configuration is constant across scales: every model uses routed experts with activated per token, an expert activation ratio of (Sec. 3.5). The MLA KV-compression ratio, defined as the compression rank divided by the model width, is likewise constant as width grows, so compression acts as a fixed architectural property rather than a hidden scaling variable.³³3Whether the optimal compression ratio is itself scale-dependent is an interesting question that we leave to future work. We use the proxy value. The same holds for the KDA-to-MLA layer ratio, the iHC residual-stream count, and the patchification factors. Fixing these settings prevents structural choices from being entangled with the scaling study and keeps all data points mutually comparable.

4.2 Chinchilla Laws for Chimera

HeteroP yields mutually comparable Chimera models with scale-appropriate hyperparameters. Using this family, we fit Chinchilla-style compute-optimal scaling laws [36] for visual diffusion.

Why a compute-optimal law is needed.

A central practical question in scaling is allocation. Given a fixed training budget of FLOPs, how large should the model be, and on how many tokens should it process? Training an overly large model for too few steps wastes capacity, whereas training an overly small model for too long spends compute on a saturated model. Hoffmann et al. [36] show that, for language models, the loss-minimizing split between and follows a predictable trade-off, and that ignoring it yields substantially undertrained models. For visual diffusion, this trade-off has not been characterized on a model family with hyperparameters controlled across scales through HeteroP (Sec. 4.1). Our goal is therefore to make the size-versus-data trade-off of Chimera explicit and predictive, so that the configuration of a large target run can be chosen a priori from the fitted law rather than by trial and error.

Formulation.

We adopt the parametric form of Hoffmann et al. [36], modeling the training loss as a function of the model size and the amount of training data :

(29)

where denotes the irreducible loss under the fixed data distribution and diffusion objective, i.e., the Bayes-optimal diffusion loss on the data distribution, and the two power-law terms capture the finite-size penalties from limited capacity and limited data. Three measurement choices adapt this form to our setting. First, the loss is the Chimera diffusion loss, the rectified-flow -prediction objective of Sec. 3.1, rather than a downstream generation score such as FID. Under the fixed objective and evaluation protocol used here, diffusion loss provides a sampler-independent metric for comparing pre-training across model scales. Second, because Chimera is sparsely activated, counts activated parameters, so the law reflects the compute actually spent per token rather than the total parameter store.⁴⁴4The load balance across experts does not affect computational cost, but it can affect model quality and hence the losses used for fitting. We therefore conduct this study under near-perfect load balance, as demonstrated in Fig. 5. Third, denotes the cumulative training-data volume, measured by the number of visual latent positions processed, and we relate it to compute through the standard accounting for the dominant matmul cost,⁵⁵5This approximation remains valid for the hybrid backbone. Most layers use linear attention with constant per-token cost, and the few MLA layers attend over compressed key–value states, so parameter matmuls dominate the overall compute. which lets us express the law and its optimum directly in terms of compute.

Fitting and usage.

All runs share the same base AdamW configuration and a warmup-then-constant learning-rate schedule without decay. Thus, the loss measured at an intermediate step is not affected by a learning-rate decay phase tailored to that training duration. A single run can therefore provide comparable loss measurements across multiple data budgets . We further ensure that all runs remain in the infinite-data regime: no run processes more than one epoch of the training corpus. Given a set of runs spanning model sizes and durations, we estimate the compute-optimal allocation with the three estimators of Hoffmann et al. [36]: the lower envelope of the loss-versus-compute curves, IsoFLOP profiles that compare losses across model sizes at fixed compute budgets, and a direct parametric fit of Eq. 29 that minimizes the Huber loss between predicted and observed log-losses over a grid of initializations. The fitted law yields the compute-optimal frontier and in closed form. We use this frontier to select the configuration of Chimera’s largest runs, and report the fits and their cross-estimator agreement in Sec. 5.3.

Extending the law to the image–video data ratio.

Unlike scaling laws for single-modality text data, visual generation trains on multimodal data that mixes images and videos. The two differ in information content and in the number of tokens each sample contributes, so the mixture shifts both the learning signal and how the compute budget is spent. The standard law over leaves this axis uncontrolled. We therefore extend the scaling law with an additional dimension, the image–video data ratio, and model how its optimal value changes as the training budget increases.

Specifically, we generalize the IsoFLOP methodology of Hoffmann et al. [36]. We train models of multiple sizes under multiple image–video ratios and record their diffusion losses at fixed compute budgets, tracking the image and video losses separately. At each budget, instead of fitting a one-dimensional quadratic profile over model size alone, we fit a quadratic surface over model size and data ratio for each loss. We then select the optimum of every fitted surface and fit the compute-optimal frontier to these per-budget optima, obtaining the optimal model size and image–video ratio as functions of compute. Sec. 5.4 presents the fitted surfaces and the resulting frontier.

5 Experiments

We evaluate Chimera along the two axes developed in this paper. We first validate the scaling methodology: Sec. 5.2 verifies HeteroP transfer across width and depth, Sec. 5.3 presents compute-optimal scaling laws for image and video generation, and Sec. 5.4 extends the law to the image-video data mixture. We then evaluate the models: Sec. 5.5 compares Chimera with strong baselines on generation quality, and Sec. 5.6 presents component ablations.

5.1 Implementation details.

All models instantiate the architecture of Sec. 3 and differ only in width, FFN width, and depth. Fig. 6 visualizes representative configurations spanning M to B activated parameters, of which the exact specifications are listed in Appendix B. As specified in Sec. 4.1, all structural ratios are held fixed across scales, including the MoE configuration, the KDA-to-MLA ratio, the residual-stream count, the MLA KV-compression ratio, and the patchification factors.

Frozen components.

Visual inputs are encoded by a frozen causal 3D VAE [82] with latent channels and compression along the temporal, height, and width axes. A clip of frames at resolution is encoded into latent frames of size , and a single image is the special case . Encoded latents are standardized with the per-channel mean and standard deviation of the VAE latent space, so the rectified-flow objective operates on approximately zero-mean, unit-variance targets. We then apply patchification. Text prompts are encoded by a frozen umT5-XXL encoder [16] with a maximum length of tokens. Padding positions are dropped and the remaining embeddings are packed into a variable-length sequence that is prepended to the visual tokens through a separate trainable projection (Sec. 3.2).

Training recipe.

All models are trained with the rectified-flow objective of Sec. 3.1, and the loss is the mean squared error on the velocity target, unweighted across timesteps. One timestep is drawn per sample and shared by all of its tokens, from a logit-normal distribution [24] that concentrates supervision on intermediate noise levels. Since a fixed timestep destroys more information in a longer token sequence, we then shift it toward higher noise by , with for a sample of visual tokens, taking the image bucket as the reference. Because is computed per sample, images and video clips of different sizes are shifted independently within the same batch.

Text conditioning is dropped with probability to enable classifier-free guidance at inference. We optimize with AdamW using a base learning rate , betas [62, 27, 9], , and a base weight decay . HeteroP instantiates the parameter-group learning rates, initialization variances, weight decays, and AdamW values at each scale according to Table 1. The base learning rate warms up linearly from over steps and then remains constant. The constant schedule keeps intermediate checkpoints comparable across training durations, which the scaling study in Sec. 5.3 relies on.

Gradients are clipped to a global norm of . As a stability guard, we skip the optimizer and EMA update at any step whose pre-clipping gradient norm is non-finite or exceeds . Expert load is balanced by the auxiliary-loss-free bias update of Sec. 3.5, with the selection bias adjusted by per step. No load-balancing term is therefore added to the training objective, and the reported loss is exactly the flow-matching loss above, which matters for the scaling study in Sec. 5.3 where this loss is the primary metric. We additionally maintain an exponential moving average of the trainable parameters with decay , updated every step.

Data.

We train on a large-scale curated visual corpus. Training uses multi-resolution pixel buckets: an early stage mixes , , and buckets with sampling ratios . Mixed-resolution batches yield highly variable sequence lengths, so a data sequence balancer [99] redistributes packed sequences across devices to balance the per-GPU load. Samples with aspect ratio above or below are discarded. During mixed image–video training, we retain the three image buckets and add approximately 5-second video clips at 180p and 360p, using loader sampling ratios for , , , 180p video, and 360p video, respectively. Each clip contains 81 frames sampled at 16 fps and is assigned to an area-preserving crop bucket according to its aspect ratio. Additionally, we conduct all trials in Secs. 5.2 and 5.3 using a deterministic data loader and under an infinite-training-data regime.

Infrastructure.

We train Chimera with Hybrid Sharded Data Parallel (HSDP) [101], sharding model states across the eight GPUs within each node while replicating them across nodes. HSDP is applied at block granularity to overlap parameter communication with computation. Parameters are gathered in bf16, gradients are reduced in fp32. We further apply activation checkpointing to every block. The dominant per-layer operators are implemented directly over the packed token representation using custom Triton kernels. These include a fused multimodal short convolution, chunked KDA with fused gating and normalization, and an MoE implementation based on expert-wise token sorting and grouped matrix multiplications.

Mixed image–video training can produce substantial load imbalance because different ranks may process samples with widely varying computational costs. We therefore introduce two lightweight workload balancers. For VAE encoding, ranks within each node exchange sample-shape metadata and estimate encoding cost from pixel count. Samples are assigned greedily in descending order of cost to the currently least-loaded rank, transferred in a single all-to-all, and encoded latents are returned to their original ranks and order through a second all-to-all. We additionally apply a DiT balancer before the first transformer block, following [99]. It estimates the cost of each packed sequence using a model that captures both the linear cost of KDA-dominated computation and the quadratic cost of periodic MLA layers, and redistributes complete sequences through a differentiable all-to-all so that outputs and gradients are routed back to their owners. Both assignments are recomputed at every step and applied only when the estimated reduction in the maximum per-rank workload exceeds a small threshold.

Evaluation protocol.

We evaluate Chimera at two levels. For the scaling study, the primary metric is the Chimera diffusion training loss of Sec. 3.1. Under a fixed objective and noise parameterization, this loss is sampler-independent and therefore comparable across model sizes, training durations, and data mixtures, which the analyses in Sec. 5.3 rely on. For the final models, we additionally report downstream generation quality. We assess image generation on the GenEval [28] and DPG-Bench [40] benchmarks, and video generation with FID and FVD under the length-extrapolation protocol of Sec. 5.5. To keep comparisons controlled, we hold the sampling configuration fixed across the variants under test within each such comparison, so that measured differences reflect the models rather than the sampling settings. We detail the setup specific to each experiment in the corresponding subsection.

5.2 HeteroP Hyperparameter Transfer

We first verify the central premise of the scaling study: under HeteroP, the base learning rate tuned on the proxy model remains near-optimal at larger scales.

Setup.

We sweep the base learning rate from to on two model families that vary one scaling axis at a time. Under our recipe, every model is trained for k steps at each candidate , with the parameter-group learning rates and other scaled optimizer values instantiated according to Table 1. For comparison, we repeat the same sweeps under standard parameterization (SP), where the swept learning rate is applied globally without HeteroP’s transfer rules. The width sweep spans M to B activated parameters. The depth sweep varies the number of layers from to , spanning M to M activated parameters. Across both sweeps, all other architectural and training settings are held fixed as specified in Sec. 5.1.

Results.

Fig. 7 plots the training loss after k steps against the base learning rate for both families. Two observations support the HeteroP transfer recipe. First, the optimal base learning rate is approximately stable across scales: every configuration attains its minimum near , drifting by at most one point on the sweep grid across a range of model sizes (Fig. 7(a)) and an range of depths (Fig. 7(b)), and the loss basin around the optimum remains wide and flat at all scales. Second, the curves are cleanly ordered: at every learning rate, wider and deeper models achieve strictly lower loss with no crossovers, indicating that the parameterization preserves the benefit of scale rather than trading it for stability. Together, these results support transferring the base learning rate tuned on the proxy model across both axes, and we therefore fix for every run in the scaling study of Sec. 5.3. In contrast, under SP, the optimal global learning rate shifts with scale and several high-rate runs become unstable (Figs. 7(c) and 7(d)), leading to suboptimal training and less reliable measurements for fitting the scaling law (discussed later in Sec. 5.6).

5.3 Compute-Optimal Scaling Laws

We estimate the compute-optimal allocation between parameters and training tokens with all three of their estimators: the training-curve envelope, IsoFLOP profiles, and a direct parametric fit of the loss surface (Eq. 29). In all cases, we start by training the model family in Table 4 with different numbers of tokens, and ensure all models are trained in the infinite-data regime. All fits use activated parameters for , since this is the quantity that determines the per-token compute of our sparse MoE backbone, and use the diffusion training loss defined in Sec. 3.1. The warmup-then-constant base-learning-rate schedule from Sec. 5.1 lets us treat intermediate checkpoints from a long run as comparable shorter-horizon runs.

Specifically, the envelope estimate follows the lower envelope of loss–FLOPs trajectories and records the checkpoint that attains the lowest loss at each compute budget. Although cumulative FLOPs grow with for a fixed model, the same FLOP budget corresponds to different visual-token counts across model sizes because the per-token cost varies with . Each envelope-winning checkpoint therefore jointly determines and . The latter is the optimal training volume associated with the selected model size, rather than an independent compute axis. We use the logged per-step FLOPs to map each checkpoint to . The IsoFLOP estimate instead slices the trajectories at fixed FLOPs, fits a local quadratic profile of loss against , and uses the profile minimum to infer and . We then fit power laws to these optima. This mirrors the first two empirical estimators in Hoffmann et al. [36], while adapting them to visual diffusion loss and activated-parameter accounting.

Scaling Laws on Image generation.

For image pre-training, the current fit uses usable runs covering activated model sizes and logged loss points. The envelope spans to FLOPs, while the IsoFLOP profiles use interior budgets from to FLOPs. Fig. 8 (8(a)-8(c)) shows that the training curves are cleanly ordered and that the lower envelope moves from smaller to larger models as the budget increases. The resulting frontier is close to balanced: the envelope fit gives and . The exponents sum to , consistent with the approximate compute relation . The compute is divided nearly evenly between activated model size and visual-token count.

Fig. 8 (8(d)-9(f)) gives an independent view of the same law. Each fixed-compute slice forms a smooth, approximately convex profile in model size, with a minimum that shifts rightward as compute increases. Fitting these minima yields and . The agreement between the envelope and IsoFLOP estimates indicates that additional image-training compute should be allocated nearly evenly between activated parameters and visual tokens, rather than spent mostly on either larger models or longer training alone.

Scaling Laws on Video generation.

We repeat the study on p videos using independent training-from-scratch runs on the same -model activated-parameter grid. Video curves cover to FLOPs on the envelope, and the IsoFLOP analysis uses budgets from to FLOPs, with the high-compute frontier anchored by the and FLOP slices.

The video frontier is again close to balanced, but the largest budgets now available tilt mildly toward larger activated models. The full envelope fit in Fig. 9 (9(a)-9(c)) gives and . The IsoFLOP profiles in Fig. 9 (9(d)-9(f)) give and when the high-compute slices are included.

Compared with image generation, video generation is consistently more model-heavy across both estimators: the envelope/IsoFLOP model-size exponents increase from for images to for videos, while the corresponding token exponents decrease from to . Thus, as the compute budget grows, compute-optimal video training allocates a modestly larger share of additional compute to model capacity rather than training tokens. One plausible explanation for this shift is that video generation places a broader representational burden on the backbone. In addition to the appearance and semantic structure required for images, a video model must capture frame correspondence, including object dynamics, temporal correspondences, and camera motion. This may reduce the marginal return from scaling relative to scaling , consistent with the observed shift of the video frontier.

Parametric loss fit.

The envelope and IsoFLOP estimators use only the per-budget optima. As a third, independent route, we fit the full loss surface of Eq. 29 to all measurements at once, following the third approach of Hoffmann et al. [36]: the five parameters are obtained by minimizing a Huber loss () on the residuals between predicted and observed log-losses, using L-BFGS-B from a grid of initializations (doubled for video with a second grid centered on the observed loss scale). For images, the fit uses the de-spiked training trajectories of the same runs, sampling geometrically spaced checkpoints per model ( points spanning to FLOPs); for video, it uses the final checkpoint of each of the runs. The fitted surfaces are

(30)

(31)

Both fits are accurate: the image surface reproduces the observed log-losses with and a median absolute relative error of ( under leave-one-model-out validation), and the video surface predicts the held-out terminal losses with a median error of .

Minimizing the fitted surfaces under the constraint yields , for images and , for video. These closed-form exponents agree closely with the two non-parametric estimators above: all three routes place the image frontier within and the video frontier within for the model-size exponent. A run-level bootstrap ( resamples over of the model settings) gives an interval of for the image exponent; the corresponding video interval is wider (), since the terminal checkpoints span only decades in , so we read the video fit as confirming the direction of the allocation rather than sharpening its exponent. Notably, the fitted irreducible losses are nearly identical across the two modalities ( versus ). The near equality of is consistent with using the same models, including the denoiser and VAE, for both modalities, as architecture changes tend primarily to shift the offset of a scaling curve [31]. Overall, the three estimators consistently indicate that image pre-training compute should be split almost evenly between activated parameters and visual tokens, while video pre-training currently favors allocating a modestly larger share to model size.

5.4 Optimal Image-Video Data Ratio

We extend the compute-optimal analysis along the mixture axis introduced in Sec. 4.2 and estimate how the optimal image–video data ratio evolves with the training budget.

Setup.

Unlike the single-modality studies above, these runs train on image and video data jointly. We train models spanning roughly M to B activated parameters under several image:video mixture ratios between and , following the recipe of Sec. 5.1, and record the image and video diffusion losses separately. Note that this ratio searching range is decided by the actual amount of image and video data, considering two facts: i) Video data is usually less than image data; ii) The selected ratio should be ideally maintain the infinite-data regime (less video data reuse). In detail, we fix the total batch size during training and balance the number of image and video samples. We follow the previous and p settings for image and video data, respectively. The analysis covers eight compute budgets that span to FLOPs. Slicing all training trajectories in a given budget yields – measurements per budget, each corresponding to one model-size and ratio configuration. Following the extended IsoFLOP methodology of Sec. 4.2, we fit a quadratic surface of each loss over model size and mixture ratio at every budget, and take the surface optimum as the compute-optimal configuration for that budget.

IsoFLOP loss surfaces.

Fig. 10 shows the fitted image-loss surfaces. The quadratic surfaces reproduce the measurements closely at all budgets, with root-mean-square residuals of at most , about of the measured loss. The surfaces are smooth and approximately convex, and their optima (stars) lie in the interior of the ratio range. We notice two consistent patterns. First, the optimal model size shifts toward larger models as the budget grows, in line with the compute-optimal frontier of Sec. 5.3. Second, near the optimum the loss varies far more strongly along the model-size axis than along the ratio axis. Choosing the model size correctly therefore matters most, while moderately off-optimal mixtures cost little.

Compute-optimal data ratio.

Fig. 11 tracks the optimal image-to-video ratio at each compute budget for both image and video losses, together with power-law fits. The two losses lead to clearly different optima. For image loss, the optimum is consistently image-heavy across all examined budgets, lying between and . It shifts only gradually toward video as compute increases, moving from roughly at FLOPs to roughly at FLOPs. The fitted exponent is , meaning that the optimal ratio decreases by about per decade of compute. For video loss, the fitted optimum remains at , the video-heavy boundary of our search range, for all budgets, with a fitted exponent of . Because this optimum lies on the boundary, we interpret it as a one-sided result: within the mixtures explored here, video loss improves monotonically as the video share increases.

These results suggest a simple principle for data mixing: the compute-optimal mixture is image-heavy, but shifts slowly toward video as the training budget grows. This behavior is intuitive given the different roles and costs of image and video data. Images provide appearance and semantic supervision at a much lower token cost per sample, making image-heavy mixtures more compute-efficient in the low-compute regime. As compute increases, the optimal mixture allocates a modestly larger share to video, suggesting that additional budget can be used to learn temporally rich and long-tail dynamics after the model has acquired sufficient appearance priors from images. The small negative slope further indicates that this transition is gradual rather than abrupt: video becomes increasingly valuable with scale, but does not eliminate the token-efficiency advantage of image data.

We also convert the sample ratio into a total-token ratio, including text tokens. Under this view, the same optima correspond to a much more video-heavy allocation in terms of training tokens, reflecting the substantially higher token cost of each video sample. Together with the size–data frontier in Sec. 5.3, these fits specify the compute-optimal model size, token budget, and data mixture for our largest runs.

5.5 Generation Quality and Comparisons

Loss comparisons under matched compute.

We construct 2B instantiations of the Wan2.1 [82] and Z-Image [8] architectures and train them alongside two 2B Chimera variants on the same data, with the same diffusion objective, and to a shared compute budget of FLOPs. The first Chimera variant, Chimera-dense, uses dense FFNs and standard parameterization, providing a matched comparison without sparse capacity or HeteroP. The complete variant adds MoE, iHC, and HeteroP.

Fig. 12 shows a clear convergence advantage for Chimera. At the common training loss of , Wan requires FLOPs and Z-Image requires FLOPs. Chimera-dense reaches the same loss with FLOPs, corresponding to higher compute efficiency than Wan despite using neither MoE nor HeteroP. The complete Chimera configuration reaches the target with only FLOPs, a compute-efficiency gain over Wan. Its curve remains below all dense baselines throughout the shared compute range. These results separate two sources of improvement: the dense Chimera architecture already converges more efficiently than matched full-attention baselines, while sparse capacity, iHC, and HeteroP compound this advantage.

Qualitative results.

Fig. 13 presents text-to-image samples generated by Chimera at a resolution of , using prompts ranging from short phrases to detailed descriptions. The samples cover diverse visual styles, including photorealistic macro and portrait photography, studio-style food photography, watercolor and ink illustration, and anime-inspired artwork. Across these examples, Chimera captures fine-grained structures such as petal and stamen geometry, skin texture, and reflections on liquid and glass surfaces, while maintaining coherent lighting, depth of field, object boundaries, and artistic style. The last row further examines typography-aware generation, with each example combining a mythical chimera with the word “Chimera.” The word is rendered using materials specified by the prompts, including glowing bubbles, drizzled sauce, and marker strokes, while remaining legible and visually consistent with the geometry and illumination of each scene. Overall, these examples suggest that Chimera can produce high-resolution images with strong local detail, global coherence, and text–image alignment despite relying predominantly on linear attention. Despite being trained exclusively on 1K images, Chimera can directly generate coherent 2K and 4K images in a zero-shot manner, without resolution-specific finetuning.

Fig. 15 presents qualitative results for text-to-video generation, showing five uniformly sampled frames from each 81-frame (5-second) video. Across diverse scenarios involving object emergence, natural and articulated motion, and camera movement, Chimera produces temporally consistent content while preserving object identity, scene composition, and lighting. The examples containing the words Relax” and Chimera” further show that rendered text remains legible and geometrically consistent across frames. These results suggest that Chimera maintains strong semantic and visual consistency over time.

Zero-shot length extrapolation.

Wan2.1
HunyuanVideo-1.5
Chimera

Wan2.1
HunyuanVideo-1.5
Chimera

To test whether KDA and our modality-aware causal convolution provide a natural mechanism for length extrapolation, we evaluate a Chimera checkpoint trained with 5-second video clips at inference lengths of 5, 10, 15, 20, 25, and 30 seconds, without length-specific finetuning or positional interpolation. We compare against Wan2.1-T2V-1.3B [82] and HunyuanVideo-1.5 [80]. For every model and duration, we generate 512 videos and evaluate them against 512 reference videos sampled from the validation set. To keep the evaluation window fixed while measuring quality progressively farther beyond the training horizon, we compute both metrics only on the final 5 seconds of each video. This yields generated frames for FID and 512 generated tail clips for FVD. We use the same prompts, seeds, resolution, frame rate, and duration grid across all models, although the shared and 16-fps setting lies outside the baselines’ native configurations.

Fig. 16 shows that Chimera preserves quality substantially better as the generation horizon increases. From 5 to 30 seconds, its FID increases only from to , a relative degradation of , compared with for Wan2.1 and for HunyuanVideo-1.5. Its FVD increases from to , corresponding to degradation, whereas Wan2.1 and HunyuanVideo-1.5 degrade by and , respectively. At 30 seconds, Chimera also attains the lowest measured absolute FID and FVD among the three models. These results are consistent with the architectural motivation of our NoPE design: KDA repeatedly applies the same content-adaptive recurrent update beyond the training horizon, while the temporally causal convolution supplies local structure through fixed-support kernels that introduce no length-dependent positional phases. The resulting backbone thus exhibits natural zero-shot temporal extrapolation.

The qualitative comparisons in Fig. 17 reveal the same trend: Wan2.1 becomes blurry and nearly static during long-horizon inference, while HunyuanVideo-1.5 produces sharper frames but likewise exhibits little motion. In contrast, Chimera remains sharp and continues to evolve throughout the 30-second sequence, avoiding both failure modes.

Benchmark evaluation.

媒体内容 · 前往原文查看

	GenEval							DPG
Model	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall	Overall
Generation Only
PixArt-[12]	0.98	0.50	0.44	0.80	0.08	0.07	0.48	71.11
SD v2.1 [67]	0.98	0.51	0.44	0.85	0.07	0.17	0.50	68.09
DALL-E 2 [66]	0.94	0.66	0.49	0.77	0.10	0.19	0.52	–
SDXL	0.98	0.74	0.39	0.85	0.15	0.23	0.55	74.65
DALL-E 3	0.96	0.87	0.47	0.83	0.43	0.45	0.67	83.50
SD3-Medium [24]	0.99	0.94	0.72	0.89	0.33	0.60	0.74	84.08
FLUX.1-dev	0.98	0.93	0.75	0.93	0.68	0.65	0.82	84.00
Seedream 3.0	0.99	0.96	0.91	0.93	0.47	0.80	0.84	88.27
Z-Image-Turbo [8]	1.00	0.95	0.77	0.89	0.65	0.68	0.82	84.86
Chimera	0.98	0.88	0.85	0.88	0.62	0.68	0.82	85.12

To quantitatively assess the image-generation capability of Chimera, we evaluate it on two widely used image benchmarks, GenEval and DPG-Bench. Together, these benchmarks measure compositional accuracy across attributes such as object count, color, and spatial relations, as well as overall text–image alignment. Table 2 reports the results.

Chimera achieves an overall GenEval score of and a DPG-Bench score of . It matches FLUX.1-dev and Z-Image-Turbo on GenEval while outperforming both on DPG-Bench, and it surpasses every other evaluated baseline except Seedream 3.0 on both aggregate metrics. Although our limited training budget prevents Chimera from matching the strongest large-scale image-generation model, its consistent advantage over the remaining baselines demonstrates strong compositional understanding and prompt alignment for a unified model trained with much less compute.

5.6 Ablation Studies

Hybrid KDA–MLA attention improves memory efficiency and latency.

To isolate the system-level benefit of hybrid attention, we compare our KDA/MLA backbone with an otherwise matched MHA/MLA counterpart, both with approximately 2B activated parameters. Both use the same schedule, but every KDA layer in the counterpart is replaced by conventional multi-head self-attention, while the periodic MLA layers and all other components remain unchanged. We use batch size one and BF16 activations on an NVIDIA A100-SXM4-80GB, with text tokens and visual tokens per temporal slice, and vary the total sequence length. The memory experiment measures peak allocated CUDA memory: we allocate, touch, and retain all cache tensors on device during one complete denoiser forward. The 24 MHA layers store sequence-dependent BF16 key–value caches of size , whereas the 24 KDA layers retain fixed-size FP32 recurrent states of shape , with memory independent of . Both backbones retain the same eight BF16 MLA latent key–value caches. MHA is implemented via FlashAttention. For latency, we time only the denoiser forward, exclude cache construction, synchronize CUDA, and report the median of three post-warm-up runs.

Fig. 18 shows that the MHA/MLA backbone reaches the GPU memory limit and first runs out of memory at approximately k tokens, whereas the KDA/MLA backbone completes the k-token test on the same GPU, supporting more than the sequence length under the roll-out-cache setting. Although FlashAttention removes the quadratic attention workspace, MHA still incurs quadratic attention arithmetic and its latency therefore grows more rapidly with sequence length. At k tokens, KDA/MLA is faster than MHA/MLA. Thus, replacing most MHA layers with KDA substantially reduces persistent cache growth and long-sequence latency, while the shared periodic MLA layers preserve direct global token interactions.

MoE, iHC, and HeteroP improve pre-training compute efficiency.

We ablate the three components cumulatively from a 2B Chimera-dense baseline. We first replace its dense FFNs with sparsely activated MoE layers while matching the activated parameter count. We then add iHC to provide adaptive read and write mappings over multiple residual streams. Finally, we apply the HeteroP (Sec. 4.1), which transfers the near-optimal base learning rate and the associated module-wise optimizer settings.

Fig. 19 shows that every addition improves the loss–compute trajectory. At a common training loss of , Chimera-dense requires FLOPs. Introducing MoE reduces this requirement to FLOPs, an approximately compute-efficiency gain. Adding iHC further reduces it to FLOPs, yielding a gain over the dense baseline. The complete configuration with HeteroP reaches the same loss with FLOPs, corresponding to a gain. The progression indicates that sparse capacity and multi-stream residual routing provide complementary gains, while HeteroP contributes the largest additional improvement. Consistent with Fig. 7, the HeteroP parameterization allows the larger base learning rate to remain stable and near-optimal, accelerating convergence without sacrificing training stability.

Learning-rate transfer fails without HeteroP.

We first ablate the transfer recipe at the level of individual runs. The bottom row of Fig. 7 repeats the width and depth sweeps of Sec. 5.2 under standard parameterization (SP), using the same numerical grid for the unscaled global learning rate but without our transfer rules. In contrast to the aligned minima in the top row, the optimal learning rate under SP is no longer consistent across scales. The fitted optima in Figs. 7(c) and 7(d) spread over a range along the width family, from for the smallest model to about for the larger ones, and shift toward smaller values as depth grows. No single global learning rate is simultaneously near-optimal for the whole family, so a global rate tuned on the proxy is necessarily mis-tuned at most other scales.

The sweeps further reveal a stability failure at high learning rates. Under HeteroP recipe, every curve keeps a wide and flat basin around and degrades gracefully up to . Under SP, the loss instead rises abruptly once the global learning rate exceeds each model’s optimum by only a small factor, and several runs blow up outright. The smallest width model leaves the plotted loss range already near , mid-sized widths explode between and , and the deepest models spike sharply at . These explosions also break the clean loss ordering by scale that HeteroP maintains at every swept base learning rate. SP therefore fails in two compounding ways, drifting optima and high-learning-rate divergence, which makes proxy-based hyperparameter tuning unreliable.

Scaling-law bias without HeteroP.

The mis-tuning above is not only a per-run nuisance: it biases the scaling law fitted on such runs. To quantify this, we repeat the image study of Sec. 5.3 on a matched model family trained under SP, where every model reuses the proxy-tuned global learning rate and therefore carries a scale-dependent mis-tuning penalty. The effect is directly visible in Fig. 20(a): the loss trajectories are noisier than their HeteroP counterparts in Fig. 8, and several large models exhibit loss spikes late in training.

The distortion propagates into the fitted law. The envelope estimate on the SP runs gives and (Fig. 20 (20(a)-20(c))), and the IsoFLOP estimate gives and (Fig. 20 (20(d)-20(f))). Both estimators inflate the model-size exponent by about – relative to the HeteroP fits ( and ) and deflate the token exponent by a similar margin. Because the mis-tuning penalty varies systematically with scale, the per-budget optima are shifted rather than merely noisier, and the near-balanced allocation of Sec. 5.3 is replaced by a spurious preference for parameters over data. Taken at face value, the SP law prescribes a compute-optimal model that grows about larger per decade of compute relative to the HeteroP law, reaching roughly oversized, and correspondingly undertrained, when extrapolated three decades beyond the fitted range. HeteroP is therefore not an optional refinement but a prerequisite for reliable compute-optimal scaling laws.

5.7 Discussions

We discuss several techniques, including Muon optimizer, MoE, and details of time embedding.

Muon optimizer. Following language model research, we experiment with using the Muon optimizer [42] for hidden layers. Although we investigated sufficient efforts in tuning hyper-parameters and splitting the correct parameter groups, we report that the loss under Muon optimizer is systematically worse than using AdamW. The only case where Muon outperforms is the beginning of training under large learning rate. This conclusion holds across different model architectures, scales, and whether HeteroP and MoE are used. We hypothesize that Muon underperforms Adam in diffusion visual generation because it lacks Adam’s implicit low-rank bias. Diffusion training combines denoising objectives across diverse timesteps and noise levels, so the useful optimization signal may concentrate in a small set of stable directions, while many other directions are noisy or inconsistent. To test this, we compare the spectra of parameter updates and trained weights from Adam and Muon models. We find that Adam concentrates most of its update and weight energy in a much smaller number of singular directions, whereas Muon produces substantially more full-rank and spectrally uniform updates. This supports the hypothesis that Adam implicitly filters for persistent low-dimensional denoising directions, while Muon spreads update energy more evenly across the matrix. Consequently, Muon can improve early training under large learning rates, but later makes slower progress because its update energy is diluted away from the stable low-rank subspace important for diffusion optimization.

MoE. We observe that MoE scaling provides much weaker compute efficiency in diffusion-based visual generation than what is commonly observed in language modeling. In our experiments, increasing sparsity yields only around a improvement in compute efficiency, substantially below the gain predicted by the common heuristic. This suggests that simply increasing the number of inactive experts does not translate into proportional effective capacity for diffusion models. We hypothesize that this gap arises because visual denoising offers less exploitable expert specialization: visual tokens are continuous, spatially redundant, and highly correlated, while diffusion timesteps share a large common denoising component that must be modeled consistently. Consistent with this hypothesis, we find that expert routing exhibits weak specialization and often becomes more uniform across tokens or timesteps. These results suggest that MoE sparsity in diffusion models is constrained by the intrinsic shared structure of visual denoising, rather than only by routing or implementation details.

Time embedding. We also observe that the timestep embedding network is a model-size-sensitive component in diffusion training. In our default design, sinusoidal timestep features are mapped to the model dimension through an MLP. However, using an overly wide hidden dimension for this MLP can destabilize training; for example, a 4096-dimensional hidden layer is unstable for a 1024-width backbone. This suggests that the timestep MLP should not be treated as a fixed auxiliary module independent of model scale. We hypothesize that the reason is that timestep embeddings act as a global conditioning signal, often controlling layer-wise modulation such as scale, shift, or gating. If the timestep MLP is too wide relative to the backbone, it can produce an overly strong or sharp timestep-dependent modulation, amplifying gradient conflicts across noise levels and disrupting the shared denoising representation. We therefore scale the hidden dimension of the timestep MLP with the backbone width, which improves training stability.

6 Related Works

Diffusion Models for Visual Generation.

Diffusion models have been repeatedly adapted to the structure, scale, and controllability requirements of visual data for improving viusal generation quality. Early work adopt convolutional U-Net backbones [33, 73], improvements in noise schedules [60], and guidance [19] to improve generation fidelity. Later systems adopted hierarchical [68, 66] and latent-space designs [67]. Combined with classifier-free guidance [32] and strong language encoders [59], these models enabled high-quality controllable generation. Following work has revisited both the denoising backbone and the latent representation. DiT replace convolutional U-Nets with Transformer architectures, enabling stronger scaling [63, 12, 24]. At the same time, the VAE latent space used by latent diffusion has been identified as a bottleneck for visual quality, motivating representation autoencoders [102, 10] and renewed interest in pixel-space or hybrid pixel-latent generation [49, 96].

Beyond image generation, Diffusion models for video generation have evolved through several stages. Early work primarily adapted image diffusion models to videos by adding temporal modeling modules, reusing strong image priors while learning motion [35, 71, 34, 6, 5]. More recently, the focus has shifted from generating short clips to long-horizon video. One line of work reformulates video diffusion as autoregressive or causal rollout [11, 72, 41]. Another line of work, which is more relevant to Chimera, is adopting efficient Transformer variants, by leveraging spatial-temporal attention patterns [89, 21, 88], adopting linear attention [83, 13], and sparse attention [98, 100, 76]. In this paper, we develop a novel hybrid attention architecture with minimal inductive bias, enabling native joint training of image and video generation. We showcase efficient, stable training and a scaling law that attains the Chinchilla-optimal allocation.

Efficient Attention Mechanisms.

The efficient attention mechanisms broadly categorized by what they reduce: the representation being attended to, the attention operator itself, or the set of token interactions. The first line compresses the key-value representation while preserving the standard softmax attention formulation [69, 1, 52]. The second line removes the softmax operation, providing linear-time or constant-state inference while preserving parallelizable training [45, 75, 29, 94]. The third line sparsifies attention by computing only a subset of token interactions, by designing attention patterns [14, 105] or making it learnable [97, 54, 103]. In this work, we adapt the architecture of Kimi Linear [78] to visual generation while injecting minimal inductive bias of spatial-temporal modeling.

Model Scaling.

Model scaling research studies how model quality changes as we allocate more parameters, data, and compute. Early scaling laws showed that language-model loss follows predictable power-law trends with model size, dataset size, and training compute [44]. Compute-optimal scaling later systematically studies the optimal allocation of data and model size given a fixed compute budget [36]. These scaling principles have also been extended beyond language modeling to autoregressive image, video, and multimodal generation [30, 50]. To reduce the cost of hyperparameter search, a complementary line studies parameterization and hyperparameter transfer [92, 18, 58]. We propose HeteroP that builds on these principles but targets a different structural issue: heterogeneous backbones require module-specific width ratios rather than one global model-width ratio. Besides, model scaling is not limited to dense parameter growth. Sparse activation and mixture-of-experts architectures scale total capacity while keeping activated compute relatively low [48, 25, 26]. In this work, we incorporate these techniques for a systemically study of scaling laws of visual Diffusion model in the contemporary era.

7 Future Exploration and Conclusion

Conclusion. We introduced Chimera, a hybrid, heterogeneous, single-stream diffusion backbone co-designed with a principled scaling recipe for token-extensive visual generation. Architecturally, Chimera combines KDA for efficient long-context state tracking, periodic MLA for global interaction, and modality-aware short convolutions for local spatio-temporal structure, enabling a unified text–image–video sequence without positional embeddings. Sparse MoE, iHC, and sandwich normalization further improve capacity and training stability. To scale this heterogeneous architecture, HeteroP transfers proxy-tuned hyperparameters and produces a consistently tuned model family from which we derive compute-optimal laws over activated model size, visual training tokens, and image–video composition. We hope that the architectural principles embodied by Chimera provide useful guidance for designing future long-context visual generators. More broadly, we hope HeteroP and the compute-optimal framework offer a reusable way to turn inexpensive proxy experiments into reliable model, data, and compute-allocation decisions for multimodal generators.

Future Exploration. There are some extensions for Chimera’s future explorations. First, our study deliberately isolates width and depth transfer. Although HeteroP parameterizes the heterogeneous modules according to their functional fan-in, the scaling family still holds several structural axes fixed, including the iHC residual-stream count, the MoE expert and top- routing configuration (and hence its sparsity), and the MLA KV-compression ratio as detailed in Sec. 4.1. A more complete and flexible parameterization should transfer from an inexpensive dense proxy to sparse target models and make residual-stream count, expert count, routing sparsity, and related structural choices scale-dependent rather than preset. Second, while Chimera already processes text, image, and video tokens in one stream, the present study evaluates only text-to-image and text-to-video generation. Incorporating the successful design choices in Chimera to single-stream hybrid-linear-global architectures (e.g., Qwen3.5 [79], K3 [77]) is a promising path toward a single model for both long-context multimodal understanding and generation. Third, Chimera still relies on a diffusion-specific AdaLN modulation. Our preliminary experiments suggest that this can be absorbed into iHC by conditioning its stream read and write mappings directly on the diffusion timestep. Removing the separate AdaLN modulation would further align hybrid language-model and hybrid diffusion architectures toward a shared backbone.

Acknowledgments. We thank Hongwu Peng for insightful discussions of the Muon optimizer and P; Bi Sai for constructive feedback; and Tong Sun, Rajiv Jain, Kalyan Sunkavalli, and John Yang for supporting this project.

References

• Ainslie et al. [2023] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023.
• Allen-Zhu [2026] Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers. Advances in Neural Information Processing Systems, 38:42349–42369, 2026.
• Barbero et al. [2024] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205, 2024.
• Black Forest Labs [2025] Black Forest Labs. FLUX.2: Frontier Visual Intelligence. https://bfl.ai/blog/flux-2, 2025.
• Blattmann et al. [2023a] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a.
• Blattmann et al. [2023b] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563–22575, 2023b.
• Brooks et al. [2024] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024. https://openai.com/research/video-generation-models-as-world-simulators.
• Cai et al. [2025] Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, et al. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699, 2025.
• Cattaneo and Shigida [2026] Matias D Cattaneo and Boris Shigida. The effect of mini-batch noise on the implicit bias of adam. arXiv preprint arXiv:2602.01642, 2026.
• Chen et al. [2025a] Bowei Chen, Sai Bi, Hao Tan, He Zhang, Tianyuan Zhang, Zhengqi Li, Yuanjun Xiong, Jianming Zhang, and Kai Zhang. Aligning visual foundation encoders to tokenizers for diffusion models. In The Fourteenth International Conference on Learning Representations, 2025a.
• Chen et al. [2024a] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024a.
• Chen et al. [2024b] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International conference on learning representations, volume 2024, pages 57611–57640, 2024b.
• Chen et al. [2025b] Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, et al. Sana-video: Efficient video generation with block linear diffusion transformer. arXiv preprint arXiv:2509.24695, 2025b.
• Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
• Chu et al. [2023] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In International Conference on Learning Representations, 2023.
• Chung et al. [2023] Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151, 2023.
• Dai et al. [2024] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024.
• Dey et al. [2026] Nolan Dey, Bin Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: Completep enables compute-efficient deep transformers. Advances in Neural Information Processing Systems, 38:137707–137739, 2026.
• Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
• Dieleman [2024] Sander Dieleman. Diffusion is spectral autoregression. https://sander.ai/2024/09/02/spectral-autoregression.html, September 2024. Blog post.
• Ding et al. [2025] Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile. arXiv preprint arXiv:2502.06155, 2025.
• Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. In Advances in Neural Information Processing Systems, volume 34, pages 19822–19835, 2021.
• Elhage et al. [2021] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
• Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
• Fedus et al. [2022] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
• Fei et al. [2024] Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Scaling diffusion transformers to 16 billion parameters, 2024. URL https://arxiv. org/abs/2407.116, 33, 2024.
• Fernández-Hernández et al. [2026] Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I Mestre, Manuel F Dolz, and Enrique S Quintana-Ortí. Why adam works better with beta1=beta2: The missing gradient scale invariance principle. arXiv preprint arXiv:2601.21739, 2026.
• Ghosh et al. [2023] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023.
• Gu and Dao [2023] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
• Henighan et al. [2020] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
• Hestness et al. [2017] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
• Ho and Salimans [2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
• Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
• Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
• Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022b.
• Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022.
• Hong et al. [2025a] Yicong Hong, Long Mai, Yuan Yao, and Feng Liu. Pushing the boundaries of state space models for image and video generation, 2025a. https://arxiv.org/abs/2502.00972.
• Hong et al. [2025b] Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, et al. Relic: Interactive video world model with long-horizon memory. arXiv preprint arXiv:2512.04040, 2025b.
• Hu et al. [2024a] Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model. In European conference on computer vision, pages 148–166. Springer, 2024a.
• Hu et al. [2024b] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024b.
• Huang et al. [2026] Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. Advances in Neural Information Processing Systems, 38:167283–167308, 2026.
• Jordan et al. [2024] Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon, 6(3):4, 2024.
• Ju et al. [2025] Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, Daniil Pakhomov, Zhe Lin, Soo Ye Kim, and Qiang Xu. Editverse: Unifying image and video editing and generation with in-context learning. arXiv preprint arXiv:2509.20360, 2025.
• Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
• Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning, 2020.
• Kazemnejad et al. [2023] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892–24928, 2023.
• Kimi Team [2025] Kimi Team. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.
• Lepikhin et al. [2020] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.
• Li and He [2026] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026.
• Liang et al. [2024] Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprint arXiv:2410.08184, 2024.
• Lipman et al. [2022] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
• Liu et al. [2024a] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a.
• Liu et al. [2024b] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024b.
• Liu et al. [2025] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.
• Liu et al. [2022] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.
• Liu et al. [2024c] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37:103031–103063, 2024c.
• Merrill et al. [2026] William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, et al. Olmo hybrid: From theory to practice and back. arXiv preprint arXiv:2604.03444, 2026.
• Mlodozeniec et al. [2025] Bruno Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and Marco Cuturi. Completed hyperparameter transfer across modules, width, depth, batch and duration. arXiv preprint arXiv:2512.22382, 2025.
• Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
• Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021.
• Olsson et al. [2022] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
• Orvieto and Gower [2026] Antonio Orvieto and Robert Gower. In search of adam’s secret sauce. Advances in Neural Information Processing Systems, 38:63404–63442, 2026.
• Peebles and Xie [2023] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
• Press et al. [2022] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
• Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
• Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
• Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
• Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.
• Shazeer [2019] Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
• Shazeer [2020] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
• Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
• Song et al. [2025] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764, 2025.
• Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
• Su et al. [2024] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
• Sun et al. [2023] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
• Tan et al. [2025] Xin Tan, Yuetao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training. arXiv preprint arXiv:2502.07590, 2025.
• Team [2026a] Kimi Team. Kimi k3: Open frontier intelligence, 2026a. https://arxiv.org/abs/2607.24653.
• Team et al. [2025] Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.
• Team [2026b] Qwen Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026b.
• Tencent Hunyuan Foundation Model Team [2025] Tencent Hunyuan Foundation Model Team. Hunyuanvideo 1.5 technical report, 2025. https://arxiv.org/abs/2511.18870.
• Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
• Wan et al. [2025] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
• Wang et al. [2025] Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 2578–2588, 2025.
• Wang et al. [2024a] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664, 2024a.
• Wang et al. [2024b] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b.
• Weng [2026] Lilian Weng. Scaling laws, carefully. lilianweng.github.io, June 2026. https://lilianweng.github.io/posts/2026-06-24-scaling-laws/.
• Wu et al. [2025a] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025a.
• Wu et al. [2025b] Jianzong Wu, Liang Hou, Haotian Yang, Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, and Yunhai Tong. Vmoba: Mixture-of-block attention for video diffusion models. arXiv preprint arXiv:2506.23858, 2025b.
• Xi et al. [2025] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776, 2025.
• Xie et al. [2025] Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025.
• Yang et al. [2025a] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
• Yang et al. [2021] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084–17097, 2021.
• Yang et al. [2022] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.
• Yang et al. [2024] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522, 2024.
• Yang et al. [2025b] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations, volume 2025, pages 83048–83077, 2025b.
• Yu et al. [2026] Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu, and Jiebo Luo. Pixeldit: Pixel diffusion transformers for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14273–14282, 2026.
• Yuan et al. [2025] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025.
• Zhan et al. [2025] Chenlu Zhan, Wen Li, Chuyu Shen, Jun Zhang, Suhui Wu, and Hao Zhang. Bidirectional sparse attention for faster video diffusion training. arXiv preprint arXiv:2509.01085, 2025.
• Zhang et al. [2025] Kai Zhang, Peng Wang, Sai Bi, Jianming Zhang, and Yuanjun Xiong. Knapformer: An online load balancer for efficient diffusion transformers training. arXiv preprint arXiv:2508.06001, 2025.
• Zhang et al. [2026] Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. Advances in Neural Information Processing Systems, 38:152509–152534, 2026.
• Zhao et al. [2023] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023.
• Zheng et al. [2025] Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025.
• Zheng et al. [2026] Chenyu Zheng, Xinyu Zhang, Rongzhen Wang, Wei Huang, Zhi Tian, Weilin Huang, Jun Zhu, and Chongxuan Li. Scaling diffusion transformers efficiently via mup. Advances in Neural Information Processing Systems, 38:19372–19413, 2026.
• Zhu et al. [2025] Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. In International Conference on Learning Representations, volume 2025, pages 97183–97219, 2025.
• Zhu et al. [2026] Yuchen Zhu, Jing Shi, Chongjian Ge, Hao Tan, Yiran Xu, Wanrong Zhu, Jason Kuen, Koustava Goswami, Rajiv Jain, Yongxin Chen, et al. Flare: Diffusion for hybrid language model. arXiv preprint arXiv:2606.01774, 2026.

Appendix A Notation

Table 3 summarizes the main notation used throughout the paper. Symbols are grouped by component, and the scope of section-local or reused symbols is stated explicitly in the table.

媒体内容 · 前往原文查看

Symbol	Meaning
Diffusion objective (Sec. 3.1)
	clean visual tokens (patchified VAE latents)
	Gaussian noise
,	diffusion timestep; noised visual tokens at
	rectified-flow velocity target
	text-token sequence from the frozen text encoder
	trainable parameters of the denoiser
	diffusion training objective evaluated at visual-token positions
Sequence and layout (Secs. 3.2, 3.3)
	packed input sequence; reused after iHC expansion for the sequence-level multi-stream state
,	modality-specific input projections
	timestep embedding
, ,	temporal, height, width sizes of the latent grid
	flattened index of the temporal-major raster order
	sequence length
Block computation (Secs. 3.2, 3.6)
	shared hidden dimension
,	AdaLN shift and scale of the attention input
,	timestep-conditioned residual gates
	number of parallel residual streams ()
,	per-token multi-stream state; its -th stream (stream-local index)
,	token-dependent iHC read and write vectors
	learnable scale in the iHC coefficient generator
	gated attention or FFN update written to the streams
	residual-stream transition matrix (identity in iHC)
KDA (Sec. 3.3)
	scan index (distinct from diffusion timestep )
, ,	query, key, and value of token
, ,	per-head query, key, and value dimensions
	recurrent state () after token
	channel-wise forget gate (state decay)
	head-wise erase/write strength
	modality-aware short conv. (Sec. 3.4)

Symbol	Meaning
MLA (Sec. 3.3)
	number of attention heads
	compressed key–value latent
	direct-key branch projected from the MLA input and shared across heads
,	latent keys and values decoded from
	head- key
MoE (Sec. 3.5)
	token input to the MoE FFN
,	section-local expert count (); activated experts per token ()
	the -th routed expert
,	router; routing weight of expert
	non-gradient load-balancing bias
	top- expert set selected for token
,	per-expert load; mean expert load; maximal load violation (Eq. 22)
RoPE analysis (Sec. 3.3, App. C)
	rotation frequency of rotary pair (rad/token)
	token offset, query position minus key position
,	components of , in rotary channel pair
	amplitude of pair (window-averaged , )
	per-pair energy over token positions (distinct from expert )
local / far mass	attention fraction at Chebyshev distance /
Scaling (Secs. 4, 5.3)
,	proxy and target models
,	proxy and target block counts
,	corresponding proxy and target parameter groups
,	target-to-proxy fan-in ratio; target-to-proxy block-count ratio
	tuple of proxy-tuned base hyperparameters
,	transferred settings for ; its functional role
	role-dependent transfer map
	activated parameter count in the scaling law
	cumulative number of visual latent positions processed
	training compute, approximated by
	fitted diffusion-loss surface
; ;	irreducible loss; positive coefficients; scaling exponents

Appendix B Representative Model Configurations

Table 4 reports the exact configurations visualized in Fig. 6. The proxy is used for hyperparameter tuning, the scaling family supports the compute-optimal analyses, and the final model is selected for full training.

媒体内容 · 前往原文查看

Model	Act. Params	Total Params	Width	FFN Hidden	Depth
Proxy	22M	59M	512	2048	4
Scaling family	55M	253M	384	1536	16
	200M	893M	896	2432	16
	291M	1.35B	1024	4096	16
	393M	1.88B	1152	5120	16
	482M	2.46B	896	4096	32
	748M	3.58B	1280	4096	32
	873M	4.41B	1280	5120	32
	999M	5.25B	1280	6144	32
	1.55B	7.99B	1664	6656	32
	2.01B	9.93B	2048	6400	32
	2.26B	10.33B	2432	5248	32
	2.50B	11.92B	2432	6400	32
	2.65B	10.79B	2944	4736	32
	2.80B	14.26B	2304	9216	32
	2.93B	14.25B	2560	7296	32
	2.99B	15.09B	2432	9088	32
	4.02B	19.31B	3072	8448	32
Final Training Model	2.22B	11.28B	2048	8192	32

Appendix C Visual RoPE Analysis: Protocol and Results

This appendix details the FLUX.2 and Wan2.2 analysis summarized in Sec. 3.3. All statistics are computed from the models’ own sampling trajectories, so the probed activations match the regime in which the models operate. Fig. 21 visualizes the per-pair decompositions, the frequency-band ablations, and the division of labor across all heads.

Models and Sampling.

We probe the public weights of Wan2.2-T2V-A14B and FLUX.2-dev. Wan2.2 has layers with self-attention heads of dimension ; axial RoPE splits each head into temporal, height, and width channel pairs with per-axis frequencies , spanning down to rad/token. Its two denoising experts (applied above and below ) are probed separately. FLUX.2 has double-stream and single-stream blocks with heads of dimension ; its axial RoPE assigns pairs to each of four axes (time, height, width, text index) with base , spanning down to rad/token. Image tokens carry coordinates only on the height and width axes, so the time- and text-axis channels, half of every head, are unrotated between image tokens; text tokens attend in the same sequence and use the text axis. We sample Wan2.2 at with frames (a token grid, visual tokens) using UniPC with steps and classifier-free guidance , and FLUX.2 at (a grid) using its -step flow schedule with embedded guidance . Each model is probed on six shared prompts covering people, animals, nature, urban scenes, close-ups, and still-life motion, at three denoising timesteps ( for Wan2.2, spanning both experts; noise levels for FLUX.2), giving settings per model.

Capture and Head Statistics.

At each captured step we record, for every self-attention layer, the per-head queries and keys after query-key normalization and before rotation (for Wan2.2, on the conditional branch of classifier-free guidance). For a fixed random subset of visual query tokens, shared across all layers and settings, we form the full attention rows and measure per head: the local attention mass on the immediate spatio-temporal neighborhood, i.e., the fraction of a query’s attention falling on tokens within Chebyshev distance on the token grid, self excluded ( neighbors for video, for images); the far-field mass at Chebyshev distance ; the mean attended distance; and the rate at which the strongest non-self key is a neighbor. For FLUX.2, attention rows span the concatenated text and image tokens, and we additionally record the mass on text keys.

Frequency Bands and Head Selection.

Within each positional axis we sort the channel pairs by rotation speed and split them into fastest, middle, and slowest thirds; for FLUX.2 the unrotated time- and text-axis channels form an additional static group. Band-restricted attention is obtained by recomputing the logits from a channel subset, keeping the scale, and re-applying the softmax. For each model we select the eight heads with the highest local attention mass averaged over all settings; the ablation numbers below average over these heads and settings.

Local Heads and Position Selection.

Both models contain sharp local heads, the visual analogue of previous-token heads. In Wan2.2, layer , head — at the same position in both denoising experts, whose top-eight local-head lists share seven of eight entries — attends to the immediately preceding token of the raster scan: its local attention mass is , and of queries place their argmax exactly there. This rate equals the fraction of queries that do not start a row, i.e., selection is exact up to grid boundaries. In FLUX.2, head of layer , a late single-stream block, mirrors this with the immediately following token ( of queries, again saturating the boundary limit; local attention mass ). Each remains the top-ranked local head of its layer in all settings, indicating that locality is a stable head property rather than an artifact of a particular input. The per-pair decomposition traces this selection to the fast end of the spectrum, exactly as in Qwen3-4B: we average each head’s pre-rotation queries and keys over all visual tokens and express every channel pair by its amplitude , phase, and frequency; the summed width-axis contributions peak at offset one (Figs. 21(a) and 21(b)), pairs on other axes contribute offset-independent constants, and the pairwise sum matches the directly rotated dot product to within (Wan2.2) and (FLUX.2) in float64. The band ablations confirm the attribution. In Wan2.2, retaining only the fastest third of each axis’s pairs preserves the selected heads’ local attention mass ( versus in the low-noise expert; versus in the high-noise expert), while retaining only the slowest third collapses it to (Fig. 21(d)). In FLUX.2, the unrotated channels select which tokens to attend to by content while the fast pairs place the peak at the target offset: attention recomputed from the two groups together matches the full heads almost exactly (cosine similarity ; local mass versus ), whereas either group alone fails ( from the fast pairs alone, without them; Fig. 21(e)).

Far-Field Heads and Semantic Matching.

The slowly rotating channels serve the opposite role. Using full per-token dumps at one setting per model, we measure each head’s per-pair energy and summarize its spectrum by the energy-weighted geometric mean of the rotating pairs’ frequencies. Heads whose attention lies far outside the local neighborhood (far-field mass ; mean attended distance of tokens in Wan2.2 and in FLUX.2) concentrate their energy at near-zero rotation speeds: their median centroid is rad/token in Wan2.2 and in FLUX.2, more than slower than that of local heads (local mass ; medians and , respectively), so their dot products turn by only a few degrees over the attended range (Fig. 21(c)). FLUX.2 makes the division explicit: its far-field heads park half of their energy (median , versus for local heads) in the unrotated channels, and the slow channels alone (the slowest third plus the unrotated group) reproduce their attention almost exactly (cosine similarity ).