1]Nanjing University 2]Alibaba Group 3]Zhejiang University \contribution[§]Corresponding author \contribution[†]Project lead \contribution[♯]Work done during internship at Alibaba \checkdata[E-mail]{limaohua, zhangsq}@lamda.nju.edu.cn \checkdata[Code]https://github.com/Met4physics/DiT-Interpretability

Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Maohua Li

Qirui Li

Yanke Zhou

Yiduo Li

Zhaosheng Chi

Chao Xu

Cuifeng Shen

Yixuan Xu

Hanlin Tang

Kan Liu

Tao Lan

Lin Qu

Shao-Qun Zhang

Abstract

Text-to-image diffusion transformers (DiTs) jointly process text and image tokens, yet their internal computation during denoising remains poorly understood. We introduce a causal interpretability framework for modern large-scale DiTs that combines attention decomposition with targeted interventions across token spans, heads, and layers. Using it to separate prompt-content tokens from structural template tokens, we find that the structural tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly, with prompt semantics first injected into the image latents and then read back into the template tokens rather than transferred directly from the prompt tokens. Inspired by the above findings, we design a training-free pruning rule for DiTs. Heads that attend most strongly to prompt tokens are dispensable, and pruning them removes of attention FLOPs with only a -point drop on GenEval. We further reveal how generative computation in DiTs is organized across heads and depth, separating semantic routing from visual synthesis and progressing from identity formation to propagation and refinement. Our work not only reveals that the tokens encoding semantics at input need not be those that maintain it during generation, but also provides a causal view of internal mechanisms in DiTs.

1 Introduction

Modern text-to-image generation has increasingly moved from U-Net backbones with cross-attention conditioning to diffusion transformers in which text and image tokens interact through joint attention [37, 11]. At the same time, the text side has also changed. Recent systems no longer rely only on CLIP [40] or T5 [41] encoders, but may condition generation on hidden states from large language models (LLMs). In such systems, the user prompt is not encoded in isolation. It is first serialized into a chat template containing system, user, assistant, and delimiter tokens, and the resulting hidden states are passed to the diffusion transformer. These structural tokens are usually treated as formatting residue, necessary for the encoder interface but not themselves part of the user’s semantic request.

This view leaves a gap in our understanding of modern multimodal diffusion transformers. In a joint-attention DiT, all conditioning tokens compete as keys for the image stream. A token that carries little explicit prompt content can still become a privileged computational site if the model learns to route attention or information through it. Similar phenomena are known in language and vision transformers, where attention sinks and register tokens absorb disproportionate attention or provide content-free slots for internal computation [57, 9]. The open question is whether chat-template structural tokens remain inert formatting residue, behave as transparent conditioning tokens, or acquire a deeper role in the generative computation.

More broadly, this question reflects a larger gap in our understanding of visual generation diffusion transformers. In language models, a growing mechanistic literature has mapped layer-wise inference trajectories, localized factual computations, and decomposed attention heads into functional circuits [53, 34, 6, 39, 25]. For diffusion transformers, such a mechanistic picture is only beginning to emerge. Recent work analyzes cross-modal attention blocks, concept-level representations, and attention interactions [44, 15, 32]. We still know relatively little about which layers commit semantic content, which heads causally affect generation, how text–image attention mediates conditioning, or how these roles change across the denoising trajectory in modern large-scale DiTs.

In this work, we propose a causal interpretability framework for probing the internal mechanisms of text-to-image diffusion transformers. The framework combines token-level attention decomposition, span-level conditioning interventions, cross-trajectory head transplantation, and layer-wise causal masking to ask where conditioning information is read, routed, and stored during denoising. Applying this framework to chat-templated text-to-image DiTs, we split the retained conditioning sequence into a semantic span from the user prompt and a trailing structural span from the chat template. Our analysis reveals a counterintuitive result: these structural tokens form dominant image-to-text attention sinks and, despite carrying little prompt-specific semantics at the encoder output, causally participate in carrying object identity inside the DiT. They acquire this content only implicitly. Rather than reading the prompt tokens as one might expect, they draw the object’s identity, at one remove, from the evolving image latents into which that content has already been injected. In other words, the tokens that encode the prompt’s semantics at the input are not the ones that maintain it during generation. Because the prompt-reading heads prove causally inert, we turn the analysis into a training-free head-pruning rule that cuts of the attention FLOPs at a loss of only points of GenEval accuracy. Taken together, we characterize a generative mechanism of the DiT spanning both heads and depth: separate heads hold the identity as registers and render the image, while early layers commit object identity, middle layers carry it forward, and late layers refine it.

Overall, the main contributions of this paper are summarized as follows:

• •

  We introduce a causal interpretability framework for text-to-image DiTs that combines token-level attention decomposition with span interventions, head transplantation, and layer-wise masking, enabling us to trace where conditioning information is read, routed, and stored during denoising.

• •

  We identify chat-template tokens as dominant text-side attention sinks in the image-to-text pathway, robust across models, timesteps, prompt complexity, and language. Yet they are not merely sinks: even though their VLM-encoded representations carry little semantics, these tokens causally function as semantic registers.

• •

  We derive a training-free head-pruning rule from this analysis. Because the prompt-reading heads are causally inert, ranking heads by their image-to-semantic attention and pruning the top ones accelerates generation in a prompt-independent way while preserving object correctness.

• •

  We reveal how the DiT’s generative mechanism is organized across heads and depth, with register-mediated identity routing separated from visual synthesis at the head level, and identity formation, propagation, and refinement distributed sequentially through the network.

2 Related Work

This section reviews seminal studies on attention sinks and registers. An extended discussion is provided in Appendix 8.

2.1 Attention Sinks in Generative Models

Attention sinks are positions that absorb disproportionate attention mass despite carrying little apparent semantic content. They were first characterized in autoregressive LLMs, where initial tokens act as stable attention anchors and preserving their key–value states enables streaming inference [57]. Subsequent work links this behavior to massive activations, depth-wise mixing control, and training dynamics, showing that sinks emerge during pre-training rather than appearing at initialization [49, 5, 14]. Similar content-agnostic concentration has also been reported in multimodal models, where sink tokens can be either largely removable or carriers of global decoding signals depending on the model [22, 60].

Attention sinks have also been reported in diffusion language models [43, 8]. In visual generation, however, evidence is still limited. Concurrent and independent work studies sinks in SD3/SDXL with causal interventions [56]. Our focus is different: we analyze modern large-scale diffusion transformers with LLM-based text encoders and identify chat-template structural tokens as the dominant attention sinks that function as implicit semantic registers.

2.2 Registers as Attention and Computation Buffers

A complementary line of work asks how to control where sinks reside by giving the model dedicated content-free slots. In vision transformers, register tokens absorb high-norm artifacts and internal computation, yielding cleaner feature or attention maps; related test-time variants redirect register-like activations without retraining [9, 21]. In language models, dedicated sink tokens similarly concentrate otherwise diffuse sink behavior and stabilize streaming inference [57]. These works frame registers primarily as attention or computation buffers, slots that make sink behavior explicit and steerable.

Recent work has begun to ask whether this register view transfers to diffusion transformers. Added register tokens can improve pixel-space DiTs by acting as norm sinks and global-information carriers [48], while some text-to-image DiTs develop passive sink registers in the image stream of models such as FLUX-Schnell [23] and PixArt- [7, 19]. In contrast, we show that chat-template structural tokens are pre-existing text-side tokens that not only play an attention-register role, but also implicitly participate in semantic conditioning inside the generation process, thereby functioning as semantic registers—a role absent from the traditional register view.

3 Preliminaries

3.1 Text-to-Image Diffusion Transformers

Modern text-to-image (T2I) generators synthesize an image by iteratively denoising a latent under a flow-matching objective. Let be the sequence of image latent tokens with timestep , and let denote the text conditioning. A transformer with blocks predicts the flow , which a sampler integrates from pure noise to a clean latent that is decoded to pixels by a VAE. We primarily study the publicly available Qwen-Image family [55], a dual-stream multimodal diffusion transformer (MMDiT [11]) in which text and image tokens are processed by separate per-modality parameters within each block, yet interact through a single joint attention stream. Our analysis applies both to the multi-step base model and to its few-step distilled variant. Results on other models are reported in Appendix 10.

3.2 Chat-Templated Text Conditioning

In contrast to CLIP [40]/T5-based [41] models (e.g., FLUX [23], SD3 [11], SDXL [38]), Qwen-Image obtains from a large vision–language model (Qwen2.5-VL [3]) applied to a chat-formatted prompt. The user prompt is wrapped in a chat template,

and is read from the encoder’s last-layer hidden states after discarding the fixed template prefix. The complete template is given in Appendix 7. The retained sequence therefore splits into two disjoint, contiguous parts,

where the semantic span holds the prompt-content tokens and the structural span holds the trailing chat-template tokens that delimit the dialogue format and carry no prompt-specific content. These structural tokens arise solely from chat-template conditioning and have no counterpart in CLIP/T5 text encoders; they are the central object of our study.

3.3 Joint Text–Image Attention

Within each MMDiT block, text and image tokens are processed by a single self-attention over the concatenation , so the image stream attends to the conditioning tokens. As shown in Fig. 2, partitioning the rows (queries) and columns (keys) of the joint attention matrix into their text and image segments yields four blocks: text queries attending to text and image keys (T2T and T2I) and image queries attending to text and image keys (I2T and I2I), the same decomposition used in recent analyses of MMDiT attention [44, 15, 32]. Among these, the I2T block is the cross-modal pathway through which the text conditioning acts on the image latents [44, 32]. We therefore focus our analysis on this block. To quantify how much attention each token—and in particular each structural token in —receives, we work with the full (post-softmax) joint attention matrix at block and head , , whose entry is the attention weight from query position to key position over the joint sequence (so each row sums to one). For a query group and a key span , we define the attention mass on from as

with the subscript naming the key span that absorbs the mass and the argument the query group that emits it. We focus on the I2T block, i.e. image queries (with ) attending to the conditioning keys , and abbreviate and , so that and are the mass absorbed by the structural and semantic spans, aggregated over heads, blocks, or timesteps where stated. Since and partition the text keys and the softmax is taken over the joint sequence, is the total mass of the I2T block, i.e. the fraction of I2T attention, and is in general below one, the remainder falling on the I2I block.

4 Text Template Tokens as Implicit Semantic Registers

In this section, we introduce our causal interpretability framework and apply it to show that the chat-template tokens act as the model’s implicit semantic registers. Combining token-level attention decomposition with span-level conditioning interventions, cross-trajectory head transplantation, and layer-wise causal masking, it traces where conditioning information is read, routed, and stored during denoising. Building on this analysis, we derive a training-free head-pruning rule and further characterize distinct generative mechanisms across heads and depths. The following subsections develop these findings in turn.

More analyses, generalization to other models, and additional qualitative examples are provided in Appendix 10.

4.1 Template Tokens Are Dominant Attention Sinks

We begin with a purely descriptive question: across the denoising trajectory, where does the image stream actually direct its attention over the conditioning tokens? Using the I2T mass of Sec. 3.3, we aggregate over the blocks, heads, and sampled timesteps, writing for the mean mass absorbed by a key span .

A minimal example.

We start from the simplest possible prompt, “An apple”, whose retained conditioning sequence consists of only content tokens followed by the template tokens. On both Qwen-Image and Qwen-Image-2512, image queries place an order of magnitude more attention on the content-free template span than on the prompt content. Fig. 3 visualizes this for the top sink heads (those with the largest ). The attention surface forms a sharp ridge over the structural span while attention among image tokens stays low, with the top head reaching . On Qwen-Image (resp. Qwen-Image-2512), () versus ()—an – span-level gap, or () per token—and the structural span outweighs the semantic span at of all sites for both checkpoints. The mass collapses onto a few delimiter tokens: <|im_end|> alone absorbs () of all image-query attention at the highest noise level—more than the content noun “ apple”—and is the dominant sink at every sampled timestep. The structural span thus behaves as a classical attention sink [57]: a few positions absorbing a disproportionate share of attention while carrying no prompt-specific content.

媒体内容 · 前往原文查看

Benchmark	Model
GenEval [13]	Qwen-Image	7.9	0.193	0.046	0.146	0.027	0.761	6.4	0.892
	Qwen-Image-2512		0.187	0.042	0.137	0.022	0.776	6.9	0.899
DPG-Bench [18]	Qwen-Image	82.1	0.076	0.169	0.046	0.140	0.292	7.2	0.123
	Qwen-Image-2512		0.081	0.172	0.047	0.143	0.291	7.5	0.122
Qwen-Image-Bench [27]	Qwen-Image	60.5	0.092	0.149	0.058	0.122	0.361	6.2	0.221
	Qwen-Image-2512		0.095	0.149	0.059	0.122	0.360	6.5	0.223

At scale across prompts.

To show this is not an artifact of a single toy prompt, we measure the same quantities at scale over three benchmarks spanning two axes along which the sink could plausibly weaken: prompt complexity and language. Along the first axis the prompts grow in length and compositional density, from the minimal “An apple” (), to the moderately complex, object-focused prompts of GenEval [13] ( prompts, mean content tokens), to the highly complex, long and dense multi-object prompts of DPG-Bench [18] ( prompts, mean content tokens). The second axis is covered by Qwen-Image-Bench [27] ( Chinese prompts, mean content tokens). We evaluate each on both checkpoints (Table 1). On GenEval the picture is unchanged. On Qwen-Image (resp. Qwen-Image-2512), the five template tokens absorb () of all image-query attention versus () for the content tokens, a per-token ratio of (). Equivalently, () of all I2T attention lands on the template span. The dominance holds site by site: the structural span outweighs the entire semantic span at () of sites, and a structural token is the single most-attended text token at () of sites, with <|im_end|> alone the top sink at (). The span-level ratio () is smaller than for “An apple” (–) only because GenEval prompts are longer (a mean of content tokens versus ), so the semantic mass is spread over more tokens; the per-token gap, which controls for length, is stable. Pushing further along the complexity axis, the long, dense prompts of DPG-Bench average content tokens, more than GenEval, so the span-level mass inverts as expected (; drops to ), but the per-token ratio rises to (), confirming that the structural sink strengthens rather than weakens with prompt length. For language, the Chinese prompts of Qwen-Image-Bench [27] likewise invert the span-level mass (), yet the per-token ratio holds at () and <|im_end|> remains the single largest sink. Taken together, these results indicate that the structural sink is robust to both prompt complexity and language.

4.2 Template Tokens Carry Little Semantics

Since the trailing template tokens attend to the preceding prompt tokens in the encoder, do they carry the prompt’s semantics? We probe this with a cross-prompt swap. For a pair of prompts we build, at fixed seed and sampler, a template-token swap that keeps ’s content tokens but substitutes ’s template tokens. If template tokens carried semantics, the template-token swap would drift toward . Fig. 4a shows it does not: the template-token swap preserves ’s semantics, with only minor changes to prompt-irrelevant details, as confirmed by the DINOv3 [45] [CLS] cosine similarity between the template-token swap and .

To test whether holds any prompt-specific content at all, we average over GenEval prompts into a single prompt-agnostic template , then substitute it for the structural span of each of disjoint held-out prompts, forming . Although this changes the template tokens’ representation by a relative of and perturbs the model’s first denoising-step prediction by (), the object is essentially unchanged. On Qwen-Image (resp. Qwen-Image-2512), the median own-vs-average image similarity is () and the mean is (), so a single content-free template shared across all prompts suffices to render any object. The exceptions are instructive: among the evaluation prompts below , most keep the object, count, and attributes, differing only in prompt-irrelevant content, while a few alter the object, count, or attributes themselves. As shown in Fig. 4b, many of these deviations are drastic. The shared average template can replace the object entirely, change its count, or drop key attributes. With classifier-free guidance [16], however, these deviations are largely suppressed and the intended object is recovered. Together, these experiments show that structural template tokens carry semantics, but only weakly.

A natural question is why the genuine failures collapse to a generic human portrait rather than to noise or to some other object. Generating from an empty prompt reproduces the same outcome, so the portrait is the model’s unconditional default, presumably reflecting the predominance of people in photo–caption training data. Based on this observation, we recompute the averaged template from the same prompts with the “a photo of” prefix stripped and regenerate. The output is unchanged (Fig. 4c). This shows that the semantic content carried by the template tokens is almost negligible. This collapse is therefore better understood as a thresholding effect: when the semantics are too weak to assert the target at low guidance, generation relaxes to this default.

4.3 Semantics Reside in Template Registers Implicitly

Attention sinks are usually taken to be content-free, so one would expect the generated object to be determined by the semantic span rather than by . Indeed, in the previous section we found that the VLM-encoded structural-token latents carry little semantic information. However, we find that inside the DiT it is in fact the template tokens that act as the semantic registers. We also trace the process by which semantics enter these tokens, and find that the flow is implicit: semantics pass first from the semantic tokens into the image tokens, and only then from the image tokens into the template tokens . Throughout, we use the public -step distilled LoRA checkpoint from Wuli-art [52] on Qwen-Image-2512, but run only a single denoising step, so that the timestep is fixed and its effect can be ignored, keeping the experimental design and analysis simple.

Template tokens are semantic registers.

We probe the computation with a cross-trajectory transplant. We run two denoising trajectories, one for “An apple” and one for “A banana”, and progressively copy per-head attention projections from the run into the run. At round we replace, in , the of of the attention heads with their counterparts. Under the conventional “semantic-head” view, the object should live in the heads that most read the semantic tokens, so transplanting those first ought to flip the identity soonest. Fig. 5a tells the opposite story: how much a head reads is a poor indicator of its causal role. Transplanting the top semantic readers first fails to transfer the target identity and instead pulls generation toward the unconditional default. The identity turns over only once the heads that barely attend to are swapped in. In this reverse order, just of the heads suffice to flip the apple into a banana. Reading the prompt and carrying the object are thus decoupled, our first indication that the semantics reside in the structural registers rather than in the span that is read.

Semantics enter the registers implicitly.

We next ask how the registers acquire their object content. In the joint attention we mask, at the register queries , their attention to a single key span—either the semantic tokens or the image latents—before the softmax, applied cumulatively over the first blocks. The two spans behave oppositely (Fig. 5b). Cutting ’s attention to leaves the object essentially unchanged. The register does not obtain its content by reading the prompt tokens directly. Cutting ’s attention to the image latents instead collapses the object within the first two blocks. The register therefore assembles its content by attending to the image stream, and does so early in the network—not from the prompt span, and not from its own near-content-free embedding. We further note that masking the first block alone has little effect. Since the image latents there are still pure Gaussian noise and carry no object content, this indicates that the prompt-to-image injection of semantics () completes within the first block.

4.4 Semantic Registers Guide Training-Free Head Pruning

Our analysis translates directly into a training-free acceleration rule. Because heads with high image-to-semantic attention are causally inert, with transplanting them first failing to transfer object identity, we rank the heads by descending and, for a budget , prune the computations of the top- heads. Pruning a head removes its full per-head cost, namely the and output projections together with the attention. In practice we prune only over the last of the denoising steps, since the early steps are more sensitive and commit the object’s identity. Every component of the joint-attention block scales linearly with the number of heads, so pruning of the heads over the late fraction of the schedule removes a fraction of its FLOPs, with . The rule is prompt-independent.

媒体内容 · 前往原文查看

	GenEval	LPIPS	HPSv3	FLOP
	76.1	–	9.56
	75.8	0.21	9.47
	75.5	0.29	9.29
	74.7	0.39	8.76
	74.0	0.40	8.44

Table 2 reports GenEval accuracy on Qwen-Image-2512 across silencing budgets, alongside LPIPS [61] and the HPSv3 [33] preference score. Object correctness is well preserved. Pruning a quarter of all heads () removes of the joint-attention FLOPs while GenEval accuracy drops only points, from to . Perceptual quality declines more quickly, so a lighter gives the best trade-off. What matters is which heads, not how many: at , GenEval falls from under our ranking to under the structural-sink ranking , whose register heads are more load-bearing, and to under random selection (HPSv3 , , ).

4.5 Distinct Mechanisms at Different Heads and Different Depths

In visual generation DiTs, what role the different heads and the layers at different depths play during generation has long remained unclear. Building on our head-wise analysis, we examine both and uncover a division of labor.

Different heads.

The heads themselves fall into two functionally distinct groups. The first are the semantic heads identified above, the register heads that encode which object to generate and hold that identity across the denoising trajectory. The second are rendering heads, whose attention concentrates almost entirely in the I2I block. They hold no object identity but instead knit the image latents into a spatially coherent picture. Ablating even a small fraction of them collapses the entire output into a mosaic of incoherent patches (Fig. 16). Note that the division is not absolute—a few heads blend these roles to some degree—but it remains clear overall.

Different depths.

To examine how identity processing changes with depth, we localize the heads that most drive the identity transfer (top of the order) across the transformer blocks (Fig. 15). Their distribution is bimodal— heads in the early blocks (L1–10), in the late blocks (L51–60), and only thinly spread across the forty middle blocks (L11–50)—and restricting the swap to each group exposes a matching division of labor. The early heads alone flip the object cleanly in both directions, so identity is committed early, consistent with the injection completing in the first block. The middle heads alone leave it almost unchanged, carrying the identity forward rather than rewriting it. And the late heads alone flip it only with residual distortion, refining rather than deciding it. This three-stage organization—early commit, middle carry, late refine—mirrors that reported in LLMs [39, 25].

5 Discussion and Conclusion

Despite rapid progress in diffusion transformers, how they actually read, route, and transform information during generation remains poorly understood. Our findings revise the conventional picture of text conditioning in diffusion transformers. Prompt semantics do not simply remain attached to the tokens that encode them, hinting that the model’s internal representation is more complex than our current understanding. Our proposed interpretability framework provides a systematic way to trace the causal flow of information through the model. We believe that a deeper understanding of these internal mechanisms can inform the design of more capable and efficient generative models.

References

• AI [2026] Krea AI. Krea 2. https://www.krea.ai/blog/krea-2-technical-report, 2026.
• Bai et al. [2025a] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a.
• Bai et al. [2025b] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
• Bao et al. [2023] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023.
• Barbero et al. [2025] Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732, 2025.
• Belrose et al. [2023] Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.
• Chen et al. [2024] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Springer, 2024.
• Dai et al. [2026] Xin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, and Maosong Sun. Revealing the attention floating mechanism in masked diffusion models. arXiv preprint arXiv:2601.07894, 2026.
• Darcet et al. [2024] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International conference on learning representations, volume 2024, pages 2632–2652, 2024.
• Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
• Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
• Gao et al. [2023] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389, 2023.
• Ghosh et al. [2023] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023.
• Gu et al. [2025] Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations, volume 2025, pages 97114–97144, 2025.
• Helbling et al. [2025] Alec Helbling, Tuna Han Salih Meral, Ben Hoover, Pinar Yanardag, and Duen Horng Chau. Conceptattention: Diffusion transformers learn highly interpretable features. arXiv preprint arXiv:2502.04320, 2025.
• Ho and Salimans [2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
• Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
• Hu et al. [2024] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024.
• Jamal et al. [2025] Amna Jamal, Mika Tan, Clarissa Aurelia Nahid Saputra, Quan Huynh, Kevin Zhu, and Antonio Mari. Diffusion transformers use sink registers. In Second Workshop on XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge, 2025. URL https://openreview.net/forum?id=Fc7s3UrQr1.
• Jiang et al. [2024] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481–52515, 2024.
• Jiang et al. [2026] Nicholas Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers. Advances in neural information processing systems, 38:56557–56595, 2026.
• Kang et al. [2025] Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321, 2025.
• Labs [2024] Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024.
• Labs [2025] Black Forest Labs. FLUX.2. https://bfl.ai/blog/flux-2, 2025.
• Lad et al. [2026] Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of llms: Stages of inference? Advances in Neural Information Processing Systems, 38:130050–130083, 2026.
• Lai et al. [2025] Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766, 2025.
• Li et al. [2026] Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, et al. Qwen-image-bench: From generation to creation in text-to-image evaluation. arXiv preprint arXiv:2605.28091, 2026.
• Li et al. [2024] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970, 2024.
• Lipman et al. [2022] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
• Liu et al. [2025] Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.17761, 2025.
• Liu et al. [2023] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36:52342–52364, 2023.
• Lv et al. [2025] Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, and Kwan-Yee K Wong. Rethinking cross-modal interaction in multimodal diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5934–5943, 2025.
• Ma et al. [2025] Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025.
• Meng et al. [2022] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022.
• ModelScope [2024] ModelScope. DiffSynth-Studio. GitHub repository, 2024. https://github.com/modelscope/DiffSynth-Studio.
• Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021.
• Peebles and Xie [2023] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023.
• Podell et al. [2024] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, volume 2024, pages 1862–1874, 2024.
• Queipo-de Llano et al. [2025] Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in llms are two sides of the same coin. arXiv preprint arXiv:2510.06477, 2025.
• Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
• Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
• Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
• Rulli et al. [2025] Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. Attention sinks in diffusion language models. arXiv preprint arXiv:2510.15731, 2025.
• Shin et al. [2025] Joonghyuk Shin, Alchan Hwang, Yujin Kim, Daneul Kim, and Jaesik Park. Exploring multimodal diffusion transformers for enhanced prompt-based image editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19492–19502, 2025.
• Siméoni et al. [2025] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.
• Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
• Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
• Starodubcev et al. [2026] Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers. arXiv preprint arXiv:2605.16147, 2026.
• Sun et al. [2024] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024.
• Tang et al. [2025] Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads. In International Conference on Learning Representations, volume 2025, pages 16632–16646, 2025.
• Tang et al. [2024] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024.
• Team [2026] Wuli Team. Qwen-Image-2512-Turbo-LoRA-2-Steps. Hugging Face model repository, 2026. https://huggingface.co/Wuli-art/Qwen-Image-2512-Turbo-LoRA-2-Steps.
• Wang et al. [2022] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
• Wang et al. [2026] Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, and Julian McAuley. Fasa: Frequency-aware sparse attention. arXiv preprint arXiv:2602.03152, 2026.
• Wu et al. [2025] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025.
• Wu and Summa [2026] Fangzheng Wu and Brian Summa. Attention sinks in diffusion transformers: A causal analysis. arXiv preprint arXiv:2605.09313, 2026.
• Xiao et al. [2024] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pages 21875–21895, 2024.
• Xiao et al. [2025] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. In International Conference on Learning Representations, volume 2025, pages 37228–37253, 2025.
• Xu et al. [2026] Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, Yanke Zhou, Yunhe Li, Cuifeng Shen, Hanlin Tang, Kan Liu, Tao Lan, et al. Rethinking cross-layer information routing in diffusion transformers. arXiv preprint arXiv:2605.20708, 2026.
• Yoo et al. [2026] Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the nature of attention sink that shapes decoding strategy in mllms. arXiv preprint arXiv:2603.14337, 2026.
• Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
• Zhang et al. [2023] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710, 2023.
• Zhou et al. [2026] Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Lan Tao, Lin Qu, Yuan Yao, and Xiaoxing Ma. Full attention strikes back: Transferring full attention into sparse within hundred training steps. arXiv preprint arXiv:2605.16928, 2026.

6 Limitations and Future Work

Our study is primarily analytical. The training-free head-pruning rule is only a naive first-cut use of our analysis, and more sophisticated designs—for instance, sink-aware sparse attention that keeps the register positions as keys while pruning the rest—could yield substantially larger gains. Equally open is why these content-free structural tokens come to serve as registers in the first place, which we leave to future investigation.

7 Qwen-Image Chat Templates and Retained Spans

We consider three Qwen-Image family checkpoints: the original Qwen-Image, the updated Qwen-Image-2512 text-to-image model, and the instruction-based editing model Qwen-Image-Edit-2511. The first two use the same text-to-image chat template. Fig. 6 gives an overview of this chat-templated conditioning pipeline. Line numbers in the bubbles mark literal newline positions.

Qwen-Image and Qwen-Image-2512.

After preprocessing before entering the DiT.

Qwen-Image-Edit-2511.

This editing checkpoint uses a different template. The system message asks the encoder to describe the input image and apply the user’s edit instruction, while the user turn contains visual placeholder tokens before the text prompt. For multiple input images, the pipeline prepends one visual placeholder per image by looping over the image list and formatting it as

Picture i:

. In the bubble below, the ellipsis abbreviates repeated

Picture i:

blocks and is not a literal template token.

After preprocessing before entering the DiT.

8 More Discussion on Related Work

8.1 Evolution of Diffusion Models

Diffusion generative modeling evolved from denoising and score-based formulations that learn to reverse a gradual noising process into practical, high-fidelity image generators. DDPMs and score-based SDEs established the core denoising/score view, while DDIM sampling, learned variances, and guidance improved sampling efficiency and controllability [17, 47, 46, 36, 10, 16]. A second shift moved generation into compressed latent spaces and attached text conditioning through pretrained encoders and cross-attention, enabling scalable text-to-image systems [42, 38]. Recent architectures increasingly replace U-Net denoisers with transformer backbones, spanning ViT-style tokenization, scalable DiT blocks, masked-token training, weak-to-strong text-to-image scaling, and broader DiT architectural refinements [4, 37, 12, 7, 59]. This DiT line is often paired with flow-matching or rectified-flow objectives and has expanded to multimodal joint-attention streams [29, 11, 55]. This transition is important for attention-sink analysis because conditioning tokens are no longer only side information injected through cross-attention. In joint-attention DiTs, prompt tokens, structural template tokens, and image latents compete within the same attention computation.

8.2 Attention Sinks in Sparse Attention

Beyond mechanistic analyses, attention sinks have also motivated sink-aware sparse attention. In long-context LLMs, sink tokens and related high-mass positions are often preserved while low-mass tokens are evicted, yielding stable streaming or compressed KV caches [57, 62, 31, 28]. Recent sparse-attention systems extend this idea from token retention to head specialization: local or streaming heads rely on recent tokens and attention sinks, while retrieval heads keep access to long-range context [58, 50, 63]. Complementary dynamic schemes instead choose sparse patterns or query-relevant token/page subsets through context-aware budgets or lightweight token indexers [20, 26, 51, 54]. This line of work treats sinks not merely as artifacts to be explained, but as useful attention anchors that can be preserved, isolated, or combined with retrieval to reduce inference cost.

9 Experimental Setup

Unless otherwise specified, all experiments use Qwen-Image-2512 as the default model with a fixed random seed of , and are implemented on top of the DiffSynth-Studio [35] framework. All runs are carried out on NVIDIA H20 GPUs with a 192-core CPU host.

10 Extended Experiments

In this section, we provide additional results for the experiments presented in the main text, along with further experiments not included there.

10.1 Extending the Analysis to FLUX.2

FLUX.2 [24] is a 32B rectified-flow DiT whose text conditioning is produced by a Mistral-3 (24B) vision–language model. Its prompt is wrapped in a Mistral chat template with system and user roles (with no assistant turn) and encoded by the VLM. The complete template is given below. Unlike Qwen-Image, no fixed prefix is discarded, so the entire serialized sequence is retained as the conditioning .

The I2T sink of Sec.

carries over as well. Aggregating the I2T mass over all GenEval prompts, the retained template tokens of absorb of all image-query attention against for the mean content tokens, so of all I2T attention lands on the template span, at a per-token ratio of . The dominance is near-total site by site, with the structural span outweighing the entire semantic span at of sites. The one difference from Qwen-Image is where the mass sits within . Rather than collapsing onto a single delimiter, it spreads across the system block, with tokens such as “ that”,

[/SYSTEM_PROMPT]

<s>

as the leading sinks and “ that” alone the most-attended token at of sites. Still, every dominant sink is structural and no content token appears among them, so the chat-template sink is intrinsic to FLUX.2 despite its different encoder, template, and architecture. On FLUX.2, the progressive head-swap experiment likewise supports our conclusion (Fig.

10.2 Extending the Analysis to Krea-2-Turbo

Krea-2-Turbo [1] is a 12B single-stream flow-matching DiT whose text conditioning is produced by a Qwen3-VL [2] (4B) vision–language model, a design close to Qwen-Image. The template is identical to that of Qwen-Image (Appendix 7).

The structural sink recurs here. Aggregating the I2T mass over blocks, heads, and the model’s sampling steps and averaging over the GenEval prompts (mean ), the structural span absorbs of the image-query attention versus for the prompt content. Equivalently, of all I2T attention lands on the template span, a per-token ratio of . Site by site, the structural span outweighs the entire semantic span at of sites, and a structural token is the single most-attended text token at of them. As on Qwen-Image the mass collapses onto one delimiter, though here it is the trailing newline after assistant (dominant at of sites) rather than <|im_end|>. The dominant sinks are again all structural, so the chat-template sink carries over to this single-stream model despite its different encoder and architecture.

Fig. 9 reports the progressive head-swap experiment on Krea-2-Turbo, whose outcome is consistent with our conclusion.

10.3 Extending the Analysis to Distilled and Edit Models

Few-step distillation.

If the structural span truly acts as a register, its attention signature should be a stable part of the model’s computation rather than an incidental feature of the long multi-step trajectory. Few-step distillation is a natural stress test. It collapses the -step sampler into a -step one and substantially alters the denoising dynamics. On the same -step distilled LoRA, now run with its native -step schedule, we recompute the I2T statistics of Sec. 4.1 over GenEval. The sink survives essentially intact. Structural tokens still absorb of all I2T attention (per-token ratio ), outweigh the entire semantic span at of sites, and <|im_end|> remains the single dominant sink, with each statistic within a point of the undistilled checkpoint. This indicates the register is intrinsic to the transformer’s computation rather than an artifact of any particular sampling schedule.

Instruction-based editing.

A complementary stress test changes the conditioning regime rather than the sampler, adding entirely new conditioning blocks. The editing model Qwen-Image-Edit-2511 is conditioned on a reference image that enters twice: in the text stream as a block of vision tokens produced by the VLM encoder, and in the image stream as VAE latents appended after the generated latents . The full sequence processed by joint attention thus becomes, in token order, . The structural tokens now compete not only with the instruction but also with two representations of the reference image—the vision tokens and the latents —that carry the actual visual content of the edit. We recompute the I2C (Conditions) statistics over GEdit-Bench [30]. Per token, remains the single strongest attractor of the whole conditioning sequence. Each structural token draws the attention of a vision token in , that of an instruction token, and that of a reference-image latent in , exceeding the per-token mass of all three spans in of edits, with <|im_end|> again the dominant structural sink. Thus, even when a new and highly informative conditioning modality is added, the structural span retains its register role per token, reinforcing that the sink is intrinsic to the transformer’s computation rather than tied to any particular task.

10.4 Extra Cases for Cross-Prompt Swaps

Fig. 10 reports additional prompt pairs for the cross-prompt swap of Fig. 4a. For each pair we build, at a fixed seed and sampler, the template-token swap , which keeps ’s content tokens but substitutes ’s template tokens. Across every pair the template-token swap preserves the semantics of prompt and differs at most in prompt-irrelevant content not specified by .

10.5 Extra Cases for Structural-Token Averaging

Fig. 11 collects additional examples for the structural-token averaging experiment of Fig. 4b, where each prompt’s structural span is replaced by a single prompt-agnostic average and grafted onto the prompt’s own semantics . For the vast majority of prompts the object is unchanged, so we focus here on the rare tail of low-similarity cases in which the shared template induces large semantic changes.

10.6 Seed Sweep for Structural-Token Averaging

Fig. 12 sweeps the random seed for the structural-token averaging experiment of Fig. 4b, regenerating across seeds to separate the semantic effect of the shared average from ordinary sampling variation. The degree to which perturbs the semantics varies markedly across prompts.

10.7 Extra Cases for Progressive Head Swap

Fig. 13 reports additional prompt pairs for the progressive head-swap experiment of Fig. 5a, where heads are swapped in order of their semantic-span attention . Across every pair the outcome is the same: swapping the top semantic readers first fails to transfer the identity, while the reverse order flips the object after only a small fraction of the heads, confirming that reading and carrying the object are decoupled.

10.8 Per-Span Progressive Swap

The progressive head-swap of Fig. 5a transplants per-head projections ranked by attention. To attribute the effect to a specific conditioning span rather than to individual heads, we instead swap the full of one entire span at a time from the “A banana” run into the “An apple” run, progressively over the transformer blocks. As in the head-swap experiment, we run on the -step distilled LoRA at a single denoising step.

As shown in Fig. 7, swapping the structural span changes the image drastically after only a fraction of the blocks, whereas swapping the semantic span leaves it almost unchanged until nearly all blocks are swapped. In neither case is the target banana reproduced. With the image tokens left untouched, a text-side swap alone, even at full depth, only pushes the image toward the unconditional default, unlike the head swap of Fig. 5a, which also transplants the image-token projections. We can therefore conclude that part of the object’s semantics is carried by the semantic span and, in particular, the image tokens, rather than by the structural register alone.

10.9 Qualitative Results of Head Pruning

Fig. 17 shows generations at increasing pruning budgets , illustrating how object identity is preserved while perceptual quality degrades gradually as more high- heads are silenced.

10.10 Temporal and Depth Structure of Attention Sinks

Aggregated over all GenEval prompts, each structural token receives far more I2T attention than a semantic token throughout denoising (Fig. 14). On Qwen-Image (resp. Qwen-Image-2512), the per-token structural mass is largest at high noise, reaching about at on both checkpoints, and decays to () near , while each semantic token decays from () to about . The structural-token advantage thus persists and even grows over time, from () at high noise to () near the end of denoising. Where the sink sits in depth may itself be informative about its origin. The per-token structural mass is near zero in the first block, crosses half its later-block plateau only at block (), peaks around block , and stays substantial through later blocks; this onset depth varies little across prompts. A mere input artifact such as large-norm template embeddings would presumably appear already at the first block and stay roughly flat with depth. The near-zero early blocks therefore seem hard to attribute to such an input-driven effect, while the prompt-independent onset would be an unlikely coincidence if it were content-driven. Together, these observations suggest a learned, depth-localized mechanism assembled by specific early-to-middle blocks rather than inherited from the conditioning embeddings, mirroring how attention sinks emerge in language models [14].