摘要：transformers vLLM 后端现在对于许多大语言模型架构而言，速度已与定制 vLLM 实现相当（甚至更快）。模型作者可以自动利用其 transformers 实现，免费获得超快的 vLLM 推理性能。
# Upgrade the vllm pip package
uv pip install --upgrade vllm --torch-backend auto
transformers 库已成为机器学习的参考建模库。它通过一致的 API 支持 450 多种架构，其设计的主要目标是让模型实现能够自包含且易于理解。阅读 transformers 代码有助于贡献者轻松了解架构的工作原理，然后将其移植到其他框架，例如 vLLM、SGLang、MLX、llama.cpp 等。
我们已完全接纳了在生态系统中的这一角色，并投入大量精力使其更加便捷。朝此方向迈出的一大步是去年将 transformers 作为建模后端集成到 vLLM 中。这使得模型作者能够在 vLLM 内部运行 transformers 模型（包括大语言模型和视觉语言模型），而无需进行任何移植工作。Transformers 提供建模代码，vLLM 则提供高度优化的推理技术，例如连续批处理和定制注意力核。
这项集成现在变得更好了 🚀！
成果展示
我们将 vLLM 的 transformers 建模后端与 vLLM 手写的原生实现，在三种截然不同的 Qwen3 模型上进行了正面比较：
- 单 GPU 上的 4B 密集模型
- 采用张量并行的 32B 密集模型
- 在同一个 8×H100 节点上，采用数据并行和专家并行的 235B 参数 FP8 混合专家模型
 |
|---|
| 结果：transformers 建模后端在每一个模型上的吞吐量都达到或超越了原生实现。 |
通过 transformers 建模后端运行任何* Hugging Face 模型，只需一个标志——`--model-impl transformers`。它可以与常用的并行选项组合使用，因此你的服务设置无需任何更改：
# Qwen3-4B dense, single GPU
vllm serve Qwen/Qwen3-4B --model-impl transformers
# Qwen3-32B dense, tensor-parallel across 2 GPUs
vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2
# Qwen3-235B-A22B-FP8 MoE, data-parallel + expert-parallel across 8 GPUs
vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel
# add --max-model-len 8192 if your node is memory constrained
*目前尚不支持使用线性注意力机制的模型，但很快会支持！代码位于 Hub 仓库中的定制模型不太可能正常工作，因为它们可能未按兼容性要求编写。
我们的测量方法
每个模型都在三种条件下进行比较，这些条件除代码路径外完全相同：
- 原生 —— `--model-impl vllm`，即 vLLM 手写模型（作为对标基准）
- 之后 —— `--model-impl transformers`，包含该 PR
- 之前 —— `--model-impl transformers`，不包含该 PR
完整且可复现的运行脚本以 gist 形式提供：benchmark.sh
那么，有哪些新变化？
vLLM 的 transformers 建模后端此前一直将注意力机制视为推理的瓶颈。通过在运行时接入 vLLM 的注意力实现，我们可以让 transformers 模型在 vLLM 引擎内高效运行。然而，部署场景涉及多个维度，只有通过定制化移植才能充分挖掘极致推理性能。跨 GPU 并行化、编译优化、融合内核以及更多技术手段，共同助力充分利用硬件资源，实现超高速推理。
 |
|---|
| 过去，一个新模型需要分别为 transformers 和 vLLM 各集成一次，且 vLLM 版本需附带定制优化 |
当模型作者追求极致性能时，他们仍需编写自定义的 vLLM 实现。
 |
|---|
| 如今，一个新模型一旦集成到 transformers 中，即可立即在 vLLM 中使用，并达到原生 vLLM 实现的运行速度 |
vLLM 的 transformers 建模后端最新版本，能够在运行时动态应用推理专用的层融合操作，对于兼容的架构而言，其速度可媲美自定义代码实现。
它是如何工作的？
vLLM 的 transformers 建模后端现在使用 torch.fx 对模型计算图进行静态分析。该过程会搜索可优化的已知模式。识别出模式后，再利用 ast（抽象语法树）对源代码进行操作，并就地重写部分运算。
我们能实现什么效果？
- 融合操作被多对一映射到（超）优化的 vLLM 内核上，例如用于混合专家（MoE）模型中专家并行化（EP）的那些内核。
- 其他主要的融合操作包括 vLLM 的 MergedColumnParallelLinear 和 QKVParallelLinear。这些模块使我们能够推断出张量并行（TP）的并行方案。如果解码器模块列表易于识别，流水线并行（PP）方案也可以被推断出来。
- 经过操控的模型仍然完全可（torch）编译，能够通过 `torch.compile` 和 CUDA Graphs 处理，与专门的 vLLM 模型实现完全相同。
- 与 vLLM 模型实现不同，Transformers 模型实现可用于训练。因此，你可以使用同一套模型代码进行训练、评估和强化学习 rollout。
如上所示，对于兼容的模型，这能够实现原生 vLLM 推理速度，而无需编写一行代码来优化模型的推理性能。
我们正在撰写一篇详细的博文，深入探讨这些优化的推理方法，并详细解释我们如何操控模型以适应这些方法。
资源
- Transformers 模型定义
- vLLM 中的 Transformers 建模后端
- 大规模服务
- Torch FX
- 抽象语法树
TL;DR: The transformers vLLM backend is now as fast (or faster) than custom vLLM implementations for many LLM architectures. Model authors can automatically leverage their transformers implementations to get ultra fast vLLM inference, for free.
# Upgrade the vllm pip package
uv pip install --upgrade vllm --torch-backend auto
The transformers library has become the reference modeling library for Machine Learning. It supports 450+ architectures through consistent APIs, and is designed with the main goal that model implementations are self contained and easy to understand. Going through transformers code makes it easy for contributors to learn how an architecture works, and then port it to other frameworks such as vLLM, SGLang, MLX, llama.cpp, and many others.
We have fully embraced this role in the ecosystem and are investing a lot of effort to make it easier. A big step in this direction was the integration last year of transformers as a modeling backend in vLLM. This has been allowing model authors to run transformers models (LLMs and VLMs alike) inside vLLM, without having to port anything. Transformers provides the modeling code, and vLLM provides extremely optimized inference techniques such as continuous batching and custom attention kernels.
This integration gets better now 🚀!
Showcase
We put the transformers modeling backend for vLLM head to head with vLLM's hand written native implementations across three very different Qwen3 models:
- 4B dense model on a single GPU
- 32B dense model on tensor parallelism
- 235B-parameter FP8 Mixture-of-Experts on data + expert parallelism on the same 8×H100 node
|
|---|
| The result: the transformers modeling backend now meets or beats native throughput on every one of them. |
Running any* Hugging Face model through the transformers modeling backend is a single flag — --model-impl transformers. It composes with the usual parallelism options, so nothing about your serving setup changes:
# Qwen3-4B dense, single GPU
vllm serve Qwen/Qwen3-4B --model-impl transformers
# Qwen3-32B dense, tensor-parallel across 2 GPUs
vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2
# Qwen3-235B-A22B-FP8 MoE, data-parallel + expert-parallel across 8 GPUs
vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel
# add --max-model-len 8192 if your node is memory constrained
*Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly.
How we measured
Each model is compared under three conditions that are identical in every way except the code path:
- native —
--model-impl vllm, vLLM's hand-written model (the bar to match) - after —
--model-impl transformerswith the PR - before —
--model-impl transformerswithout the PR
The full, reproducible runner is available as a gist: benchmark.sh
So, what's new?
The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLM’s attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference.
|
|---|
| A new model used to be integrated once for transformers, and once for vLLM with custom optimizations |
When model authors wanted the absolute best performance, they were still writing custom vLLM implementations.
|
|---|
| A new model once integrated to transformers, can now be immediately used in vLLM with native vLLM implementation speed |
The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures.
How does it work?
The transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place.
What can we achieve with this?
- Fused operations that are many-to-one mapped to (ultra) optimized vLLM kernels, such as the ones used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models.
- The main other fused operations are vLLM's
MergedColumnParallelLinearandQKVParallelLinear. These blocks allow us to infer parallel plans for TP (tensor-parallel). PP (pipeline-parallel) plans can also be inferred if the decoder block list is easily identifiable. - The manipulated models are still fully (torch) compilable, being passed through
torch.compileand CUDA Graphs, just the same as a dedicated vLLM model implementation. - Unlike vLLM model implementations, Transformers model implementations can be used in training. So you can use the same model code for training/evals/RL rollouts.
As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference.
We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them.