GEAR: 引导式端到端自回归图像合成
阅读原文· arxiv.orgGEAR联合训练向量量化(VQ)分词器与自回归(AR)生成器,通过表示对齐实现端到端学习。为解决VQ索引不可微导致梯度无法回传问题,采用双读出机制:硬one-hot分支用于next-token预测训练AR模型,可微分软分支传递对齐损失指导分词器更新,使AR引导分词器生成更易预测的索引分布。相比LlamaGen-REPA,在ImageNet gFID收敛速度最高提升10倍,并泛化至VQVAE、LFQ、IBQ等量化器及文生图任务。
Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.