# 具有内省能力的扩散语言模型

- 来源：Hacker News 热门（buzzing.cc 中文翻译）
- 作者：zagwdt
- 发布时间：2026-04-14 21:13
- AIHOT 链接：https://aihot.virxact.com/items/cmnyook9e004osl0fg8pgdjgv
- 原文链接：https://introspective-diffusion.github.io/

## AI 摘要

Introspective Diffusion Language Models（内省扩散语言模型）正式发布，该架构在传统扩散模型基础上引入内省机制，支持生成过程中的自我评估与优化。项目技术细节与代码已开源至introspective-diffusion.github.io。该研究成果在Hacker News技术社区获得100点关注度，于2026年4月14日公开。

## 正文

Introspective DiffusionLanguage Models

Abstract

Diffusion language models (DLMs) offer a compelling promise: parallel token generation could break the sequential bottleneck of autoregressive (AR) decoding. Yet in practice, DLMs consistently lag behind AR models in quality.

We argue that this gap stems from a fundamental failure of introspective consistency: AR models agree with what they generate, whereas DLMs often do not. We introduce the Introspective Diffusion Language Model (I-DLM), which uses introspective strided decoding (ISD) to verify previously generated tokens while advancing new ones in the same forward pass.

Empirically, I-DLM-8B is the first DLM to match the quality of its same-scale AR counterpart, outperforming LLaDA-2.1-mini (16B) by +26 on AIME-24 and +15 on LiveCodeBench-v6 with half the parameters, while delivering 2.9-4.1x throughput at high concurrency. With gated LoRA, ISD enables bit-for-bit lossless acceleration.

Why Introspective Consistency?

We identify three fundamental bottlenecks in current DLMs:

The I-DLM Method

Introspective-Consistency Training

Convert pretrained AR models via causal attention, logit shift, and an all-masked objective.

Introspective Strided Decoding

Generate N tokens per forward pass while verifying prior tokens via the p/q acceptance criterion.

AR-Compatible Serving

Strict causal attention enables direct integration into SGLang with no custom infrastructure.

Results

I-DLM is the first DLM to match same-scale AR quality while surpassing all prior DLMs across 15 benchmarks.

End-to-End Quality

Blue = best non-AR 1 means parallel decoding actually saves total compute vs. AR. This is why I-DLM's throughput scales with concurrency while SDAR and LLaDA plateau in the throughput figure above.

Per-Position Acceptance Breakdown

Acceptance compounds geometrically: position k has probability $p^{k-1}$. Position 1 is always accepted (logit shift).

Documentation & Resources

Everything you need to train, serve, and deploy I-DLM. Click any card to expand.

Setup, dependencies, and environment

Run I-DLM inference in 5 minutes

Introspective-consistency training recipe

Strided decoding algorithm and config

Production deployment with SGLang

Gated LoRA for bit-for-bit output

Available models and weights

Reproduce our evaluations

Installation

git clone https://github.com/Introspective-Diffusion/I-DLM.git cd I-DLM/inference bash install.sh

See inference/README.md for detailed environment setup.

Quick Start

1. Launch server:

python -m sglang.launch_server \ --model-path yifanyu/I-DLM-8B \ --trust-remote-code --tp-size 1 --dtype bfloat16 \ --mem-fraction-static 0.85 --max-running-requests 32 \ --attention-backend flashinfer --dllm-algorithm IDLMBlockN \ --dllm-algorithm-config inference/configs/idlm_blockN4_config.yaml \ --port 30000

2. Generate:

curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "Prove that sqrt(2) is irrational."}], "max_tokens": 4096, "temperature": 1.0 }'

Training

Convert a pretrained AR model into I-DLM via introspective-consistency training:

Input: Concatenate fully-masked sequence with clean sequence [x_t | x_0]

[x_t | x_0]

Attention: Strict causal masking across all positions

Loss: Auto-balanced CE on both masked and clean positions

Data: 4.5B tokens, 8 H100 GPUs, 2 epochs with stride curriculum (N=2 then N=3)

See training/README.md for scripts and configs.

Inference & ISD

Introspective Strided Decoding (ISD) generates and verifies in a single forward pass:

MASK positions: Propose new tokens (distribution q)

Clean positions: Verify prior tokens (anchor distribution p)

Acceptance: min(1, p(x)/q(x)) guarantees AR-distribution output

min(1, p(x)/q(x))

Stride N=4: TPF=2.96, ~3x wall-clock speedup in memory-bound regime

See inference/README.md for algorithm configs.

Serving (SGLang)

I-DLM uses strict causal attention, enabling direct integration into SGLang with no custom infrastructure:

Paged KV cache and continuous batching

CUDA graph capture (+42-76% throughput)

Stationary-batch decode-loop scheduling (+11-21%)

Argmax proposals (+11-15%)

Paged-only attention kernel (+10-14%)

Full system achieves 2.1-2.5x throughput over naive baseline.

Lossless R-ISD

Residual ISD (R-ISD) adds a gated LoRA adapter for bit-for-bit lossless acceleration:

LoRA active only at MASK positions; verify positions use base-only weights

Output is identical to the base AR model by construction

LoRA rank=128, overhead factor ~1.12x

Model: yifanyu/I-DLM-8B-lora-r128

Model Zoo

ModelBaseDescription I-DLM-8BQwen3-8BMain model, matches AR quality I-DLM-32BQwen3-32BLarge scale, outperforms LLaDA-2.1-flash (100B) I-DLM-8B-LoRAQwen3-8BGated LoRA (rank=128) for lossless R-ISD

All models use trust_remote_code=True (custom SDARForCausalLM architecture).

trust_remote_code=True

Benchmarks

We evaluate on 15 benchmarks across 4 categories with thinking mode enabled:

Knowledge: ARC-C, MMLU, MMLU-Pro, GPQA-D, GPQA

Math: GSM8K, MATH-500, MathBench, AIME-24, AIME-25

Code: HumanEval, MBPP, LiveCodeBench-v6

Instruction: IFEval

See inference/eval/ for reproduction scripts.

Citation

@article{yu2026introspective, title={Introspective Diffusion Language Models}, author={Yu, Yifan and Jian, Yuqing and Wang, Junxiong and Zhou, Zhongzhu and Zhuang, Donglin and Fang, Xinyu and Yanamandra, Sri and Wu, Xiaoxia and Wu, Qingyang and Song, Shuaiwen Leon and Dao, Tri and Athiwaratkun, Ben and Zou, James and Lai, Fan and Xu, Chenfeng}, journal={arXiv preprint arXiv:2604.11035}, year={2026} }