Sentence Transformers 是一个用于使用和训练嵌入向量与重排序模型的 Python 库，适用于检索增强生成、语义搜索等应用场景。在 v6.0 更新中，它新增了第四种模型类型：MultiVectorEncoder，用于 ColBERT 风格的后期交互检索。任何 PyLate 检查点和任何 Stanford-NLP ColBERT 检查点都可以直接加载到其中，同时，用于视觉文档检索的 colpali-engine 模型也可以通过你已熟悉的、用于稠密、稀疏和重排序模型的同一 API 来使用。普通的嵌入模型会将整段文本压缩成一个向量，而多向量模型则为每个 token 保留一个向量，并使用 MaxSim 算子对查询和文档进行评分。这保留了单个向量必须平均掉的 token 级匹配信息，这通常意味着更强的检索能力，但代价是索引更大。它也是视觉文档检索的最先进技术，在这种场景下，文本查询直接与页面图像进行匹配，中间无需 OCR 步骤。
在这篇博客文章中，我们将向你展示如何使用这些模型：加载各种检查点格式、编码和评分、将它们接入搜索栈、在页面图像上运行，以及保持索引成本可控。以下所有内容都只需通过简单的 pip install -U sentence-transformers 即可运行。
目录
什么是多向量模型？
- MaxSim 算子
- 你得到什么，以及代价是什么
安装
加载模型
- 检查检查点配置了什么
编码查询和文档
使用 MaxSim 进行评分
- 分数量级与 MeanMaxSim
语义搜索
检索与重排序
索引
视觉文档检索
音频检索
视频检索
可解释性
Token 池化
加速推理
评估模型
从 PyLate 或 colpali-engine 迁移而来
支持的模型
致谢
其他资源
什么是多向量模型？
稠密嵌入模型读取一段文本，并返回一个固定大小的向量。模型注意到的一切都必须压缩进那 384、768 或 1024 个数字里，而相似度就是两个这样的摘要向量之间的一个点积。这种方法效果非常好，但这种压缩在特定方面是有损的：一个稀有实体、一个精确的标识符，或长段落中一个关键从句，都必须争夺同一个向量里的空间。一个同时包含多个条件的查询也会撞上同样的墙。对于“带木腿和圆润靠垫的绿色沙发”，单个向量必须把全部四个条件融合成一个点，于是腿型不对的绿色沙发最终会和你要的那款在向量空间中挨得很近。
多向量模型（也叫晚期交互或 ColBERT 风格模型，得名于 ColBERT 论文）跳过了这种压缩。它运行同样的 Transformer 架构，但不再把 token 嵌入向量池化成一个向量，而是把每个 token 的嵌入向量投影到一个小维度（经典做法是 128 维），并保留全部向量。一个 9 个 token 的文档就变成一个 9x128 的矩阵，而不是一个 1x128 的向量。
查询与文档之间的交互被推迟到打分阶段才进行，这正是“晚期交互”这个名字的由来。交叉编码器是早期交互：两段文本一起通过模型，这样很准确，但没有任何可以预计算的内容，因为每个文档都必须针对每个新查询重新编码。双编码器——也就是上面说的稠密嵌入模型——几乎不交互（两个现成摘要之间做一个点积），而这恰恰让你可以一次性编码整个语料库并快速查询。晚期交互介于两者之间：文档仍然独立编码，可以离线建索引，但打分时会拿每个查询 token 与每个文档 token 进行比较，给两者之间留出了大得多的交互空间。
MaxSim 算子
打分使用 MaxSim：对于每个查询 token，取它与任意文档 token 的最高相似度，然后将这些最大值在查询范围内求和。
$$ \text{MaxSim} \left(\right. Q , D \left.\right) = \underset{Q_{i} \in Q}{\sum} \underset{D_{j} \in D}{max } Q_{i} \cdot D_{j} $$
MaxSim(Q,D)=Q i∈Q∑D j∈D maxQ i⋅D j
由于 token 嵌入向量经过了 L2 归一化，这些点积中的每一个都是 [-1, 1] 范围内的余弦相似度，因此整个求和结果落在 [-num_query_tokens, num_query_tokens] 区间内。
你可以把这个算子理解为一种软对齐：每个查询 token 都指向最能解释它的那一个文档 token，而得分则衡量文档对查询的整体支持程度。
这种对齐不一定是词面上的，因为 token 嵌入向量是上下文相关的。用 lightonai/mLateOn 将“企鹅生活在哪里？”与“企鹅栖息在南极洲。”进行编码，查询 token “live” 会在 “inhabit” 上找到最佳匹配，相似度达到 0.94，而这两个词没有任何共同字符！这正是词法检索做不到的事情，BM25 及其同类算法需要词条本身出现，因此同义词和改写表述会从它们眼皮底下溜走。当然，稠密嵌入模型也能弥合这一差距。晚期交互（late interaction）带来的额外优势在于，它在做到这一点的同时，并没有放弃另一个方向：当精确匹配至关重要时（比如产品代码、姓氏、函数名），MaxSim 仍然能让那个 token 独立存在，而单向量模型则不得不把它与其他所有信息平均在一起。它也不是一对一的，因为多个查询 token 通常会落在同一个文档 token 上。
你得到什么，以及代价是什么
你得到的是检索质量的提升，尤其是在以下场景中：查询的相关性取决于文档中某一个特定片段时；在像上面沙发那样的多条件查询中，每个条件都能找到自己的证据时；以及在领域外数据上，稠密模型的压缩是针对不同分布调优的。这种压缩是从训练查询中学习到的，因此模型学会了保留训练查询所需的信息而丢弃其余部分，而丢弃的部分可能恰恰是你的生产环境查询所关心的内容。这种效应会随着文档长度的增加而增强，因为更多的文本必须被压缩进同样大小的固定向量中。
代价是索引大小。每个 token 一个向量，而不是每个文档一个向量，这意味着向量数量要多得多，而较小的维度只能部分抵消这一影响。用 lightonai/LateOn 对 4,874 条 Natural Questions 段落进行编码，产生了 608,414 个 token 向量，平均每个段落 124.8 个：
| 表示方式 | 向量数量 | 维度 | float32 大小 |
|---|---|---|---|
| 稠密向量，all-MiniLM-L6-v2 | 4,874 | 384 | 7.5 MB |
| 稠密向量，gte-modernbert-base | 4,874 | 768 | 15.0 MB |
| 多向量，LateOn | 608,414 | 128 | 311.5 MB |
这大约是 MiniLM 索引存储量的 42 倍，即每个段落 62 KiB。不过，索引通常会被压缩，例如同样的 608,414 个向量在 fast-plaid 索引下仅占 92 MB，因为 PLAID 存储的是每个向量的质心 ID 加量化残差，而非向量本身。作为参照，像 Qwen3-Embedding-8B 这样的 4096 维稠密模型，处理同样的 4,874 个段落大约需要 80 MB，因此压缩后的多向量索引与人们已经在运行的稠密索引处于同一量级。Token Pooling 能在这一切之前削减向量数量，而 Retrieve and Rerank 则完全无需构建索引。
PyLate 在这篇文章中反复出现，简单说明一下：Sentence Transformers 支持稠密和稀疏模型，但不支持后期交互，因此 LightOn 在其基础上构建了 PyLate 来填补这一空白，增加了这些模型所需的训练、推理和检索组件。下面你将加载的大部分内容都是用 PyLate 训练的，而且 LightOn 还围绕它构建了一个生态系统，包括 fast-plaid——即索引部分提到的后期交互索引。在 v6.0 版本中，这些能力已直接集成到 Sentence Transformers 中。
在了解了这些权衡之后，让我们来运行一个模型。
安装
多向量模型通过常规安装即可使用：
pip install -U sentence-transformers
对于 ColPali 风格的视觉文档检索，你还需要图像依赖项（所有附加项请参阅安装文档，多模态支持总体情况请参阅多模态嵌入与重排序模型文档）：
pip install -U "sentence-transformers[image]"
Sentence Transformers v6.0 需要 transformers v5.x、torch 2.2+ 和 huggingface-hub v1.x。如果你将其中任何一个版本固定得更低，请先规划升级。完整的破坏性变更列表请参阅迁移指南。
加载模型
加载多向量模型与加载任何其他 Sentence Transformers 模型完全一样：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/LateOn")
要找到可用的模型，请在 Hub 上查找 multi-vector 和 sentence-transformers 标签。任何带有这些标签的模型都可以用上面那行代码加载，无论它最初是 PyLate 检查点、Stanford-NLP ColBERT 检查点，还是用于视觉文档检索的 ColPali 系列模型。我们正在整个生态系统中推进，把该标签添加到所有可用的模型上，所以这个列表会不断增长。
在底层，MultiVectorEncoder 会读取这些检查点历年来发布时所用的各种格式，因此即使某些检查点尚未添加标签，PyLate 和 Stanford-NLP 检查点也能直接加载：
from sentence_transformers import MultiVectorEncoder
# Native Sentence Transformers checkpoints. PyLate builds on the same schema,
# so any PyLate checkpoint loads identically
model = MultiVectorEncoder("lightonai/LateOn")
model = MultiVectorEncoder("mixedbread-ai/mxbai-edge-colbert-v0-17m")
model = MultiVectorEncoder("LiquidAI/LFM2.5-ColBERT-350M", trust_remote_code=True)
# Any Stanford-NLP ColBERT checkpoint, detected via the `HF_ColBERT` architecture
# marker. The inline projection weight and the recipe come from `artifact.metadata`
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
model = MultiVectorEncoder("answerdotai/answerai-colbert-small-v1")
# A bare transformer: a fresh random projection is appended, so training is required
model = MultiVectorEncoder("answerdotai/ModernBERT-base")
视觉文档检索模型是个例外。ColPali 系列检查点以 colpali-engine 自己的格式发布，这种格式不携带 Sentence Transformers 可用的任何信息，因此每个模型都需要在其仓库中添加一个小型配置后才能加载。这部分工作大部分已经完成，正在等待合并。请参阅 Supported Models 了解当前状态以及现在如何加载它们。
检查检查点配置了什么
多向量模型带有若干因检查点而异的配方参数：查询和文档的标记前缀、长度上限、查询是否用 [MASK] token 填充，以及评分文档时跳过哪些 token。所有这些都存放在模块配置中，因此 print(model) 可以精确显示你加载的内容。下面是原始的 ColBERTv2 检查点，它将每个查询填充到恰好 32 个 token，并将文档截断到 180：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
print(model)
"""
MultiVectorEncoder(
(0): Transformer({..., 'document_length': 180,
'query_expansion': {'strategy': 'fixed', 'attend': False, 'token': None, 'length': 32}})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, ...})
(2): MultiVectorMask({'skiplist_words': ['!', '"', '#', ...], 'skiplist_tasks': ['document'], ...})
(3): Normalize({...})
)
"""
print(model.prompts)
# {'query': '[unused0] ', 'document': '[unused1] '}
这就是经典的 ColBERT 流程：一个生成上下文相关 token 嵌入向量的 Transformer、一个将每个 token 投影到 128 维的 token 级 Dense 层、一个在评分时决定哪些 token 计入的 MultiVectorMask，以及一个 token 级 Normalize 层。其他检查点会填入不同的值。lightonai/GTE-ModernColBERT-v1 使用相同的四个模块，带有 [Q] 和 [D] 提示词，不进行查询扩展，上限分别为 48 和 300。
你通常不需要改动这些设置，因为每个发布的检查点都会配置好自身的参数。只有在从裸骨干网络构建模型时才需要关注这一点，相关内容在 Creating Custom Models 中有介绍。
不过，有一个数值值得用你自己的数据来检验一下。`document_length` 会进行截断，因此超过该长度的任何内容都不会进入索引。例如，一段 662 个 token 的文本，经过 LateOn 的 300 上限截断后，会返回 273 个向量，而该文本的其余部分则直接丢失。这些检查点中的大多数都是在短文本上训练的，因此如果你的分块长度超过上限，你可以通过 `encode_document(..., processing_kwargs={"text": {"max_length": 512}})` 在单次调用中提升该上限，但请记住，你是在让模型运行超出其训练长度，并且索引大小也会大致按比例增长。多向量模型通常能很好地容忍这种情况。在 MLDR（一个长文档检索基准）上，上述这对模型的多语言版本清晰地展示了这一差距：mLateOn 得分为 77.92，而 mDenseOn 得分为 51.59。
编码查询与文档
多向量模型是不对称的：查询和文档经过不同的前缀、不同的长度上限以及不同的评分掩码。与许多密集模型（其中两者可以互换）不同，要获得正确的嵌入向量，必须使用 `encode_query()` 和 `encode_document()`：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/mLateOn")
queries = ["What is the capital of France?"]
documents = [
"Paris is the capital of France.",
"Berlin is the capital and largest city of Germany, by both area and population.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape)
# (10, 128)
print(document_embeddings[0].shape, document_embeddings[1].shape)
# (10, 128) (19, 128)
注意你得到的结果：一个 2D 张量列表，每个输入对应一个，每个张量的形状为 (num_tokens, embedding_dim)。与密集嵌入不同，你不能将这些张量堆叠成一个矩形张量，因为每个输入都有自己独立的 token 数量。第二个文档比第一个长，因此它返回的矩阵会更高。
每次调用都会应用模型自身的处理方式。`encode_query` 会添加查询标记，如果检查点要求，会将查询扩展到固定长度，并将其限制在查询长度内。`encode_document` 会添加文档标记，将其限制在文档长度内，并从评分掩码中丢弃任何跳过的 token（对于大多数检查点，即标点符号）。
通常的 `encode()` 参数仍然全部适用，因此 `batch_size`、`show_progress_bar`、`convert_to_tensor`、`device` 和多进程池的工作方式都符合你的预期：
document_embeddings = model.encode_document(
documents,
batch_size=64,
convert_to_tensor=True,
show_progress_bar=True,
)
使用 MaxSim 进行评分
`model.similarity()` 计算完整的全对 MaxSim 矩阵：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/LateOn")
query_embeddings = model.encode_query(["Which planet is known as the Red Planet?"])
document_embeddings = model.encode_document([
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
])
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[10.7942, 11.1104, 10.9743, 11.0811]])
火星获胜，实至名归。注意亚军之间的差距有多小：土星也包含字面短语“红色星球”，而木星是一颗有红斑的行星，因此一个基于 token 级别的算子在这三者身上都有大量可抓住的特征。排序才是关键。
分数往往如此接近，正如 GLInt 通过测量整个候选池的分数分布所展示的那样。MaxSim 对每个查询 token 取最大值，因此一个文档通常会给每个查询 token 一个不错的最高匹配，分数从一个基础值起步。上下文相关的 token 嵌入向量也是各向异性的，它们聚集在一个狭窄的锥形区域内而非分散开来，所以即使是任意的 token 对也倾向于获得高分。
还有 `model.similarity_pairwise()`，适用于当你已经拥有匹配好的配对、只想要配对分数而非完整相似度矩阵的情况：
scores = model.similarity_pairwise(query_embeddings, document_embeddings[:1])
print(scores)
# tensor([10.7942])
分数量级与 MeanMaxSim
MaxSim 对查询 token 求和，因此其量级随查询 token 数量而变化，这意味着你无法比较使用不同查询处理方式的不同模型之间的分数。LateOn 将上述“红色星球”查询编码为 12 个 token。将同样的查询和同样的文档通过 ColBERTv2 运行——该模型会将每个查询填充并截断到恰好 32 个 token——分数就会落在一个完全不同的区间：
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
# ... same encode_query / encode_document / similarity calls ...
print(scores)
# tensor([[12.7970, 27.1945, 23.8495, 24.5656]])
在同一个模型内部，排序就是你所需要的全部；但如果你想要一个有界范围内的分数，可以将模型的相似度函数切换为 MeanMaxSim，它会除以查询 token 数量。回到 LateOn 上：
model = MultiVectorEncoder("lightonai/LateOn", similarity_fn_name="meanmaxsim")
# or on an already-loaded model: model.similarity_fn_name = "meanmaxsim"
print(model.similarity(query_embeddings, document_embeddings))
# tensor([[0.8995, 0.9259, 0.9145, 0.9234]])
现在每个分数都是 [-1, 1] 范围内的平均余弦相似度，尽管在实践中你只会看到 [0, 1]。
语义搜索
如果你的语料库很小，对整个语料库进行穷举式 MaxSim 是最简单且有效的做法。先对整个语料库进行一次编码，然后对每个查询与所有内容进行打分：
import time
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
# Several questions share an answer passage, so drop repeats but keep the order
corpus = list(dict.fromkeys(dataset["answer"])) # 5,000 rows -> 4,874 passages
model = MultiVectorEncoder("lightonai/LateOn")
corpus_embeddings = model.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)
query = "when did richmond last play in a preliminary final"
start = time.perf_counter()
query_embeddings = model.encode_query([query], convert_to_tensor=True)
scores = model.similarity(query_embeddings, corpus_embeddings)[0] # 98ms
top_scores, top_indices = scores.topk(3)
print(f"Search took {(time.perf_counter() - start) * 1000:.1f}ms")
for score, index in zip(top_scores.tolist(), top_indices.tolist()):
print(f"{score:.4f} {corpus[index][:100]}")
"""
Search took 122.7ms
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contest
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fou
"""
那 4,874 个段落在一张 RTX 3090 上用 20 秒完成编码，每次搜索端到端大约耗时 120ms，其中大部分时间花在针对全部 608,414 个 token 向量进行 MaxSim 打分上。这是精确的，但它随语料库总 token 数线性扩展，并且需要将所有 token 向量保留在内存中，所以当你拥有几千篇文档而非几百万篇时再使用它。此脚本的可运行版本是 `semantic_search.py`。
超过这个规模，你就需要一个真正的后期交互索引，而 Sentence Transformers 并不自带这种能力。它也不需要自带：这类索引存储的是 encode_document 产生的任何输出，所以你在这里完成编码，然后把 token 嵌入向量交给专门为它们构建的工具。索引部分为其中四种方案提供了可运行的代码片段，下面紧接着的一节则介绍如何完全跳过索引。
检索与重排
你也可以在不维护后期交互索引的情况下获得后期交互质量，方法是使用多向量模型作为重排器。一个快速的双编码器先把大规模语料库缩小到少量候选结果，然后多向量模型只对这些候选结果重新打分：
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder, SentenceTransformer
from sentence_transformers.util import semantic_search
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:50000]")
corpus = list(dict.fromkeys(dataset["answer"]))
retriever = SentenceTransformer("jinaai/jina-embeddings-v5-text-nano-retrieval")
reranker = MultiVectorEncoder("perplexity-ai/pplx-embed-v1-late-0.6b", trust_remote_code=True)
# First stage: index the corpus once with a fast bi-encoder
corpus_embeddings = retriever.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)
# Retrieve the top 50
query = "when did richmond last play in a preliminary final"
hits = semantic_search(retriever.encode_query([query], convert_to_tensor=True), corpus_embeddings, top_k=50)[0]
candidates = [corpus[hit["corpus_id"]] for hit in hits]
# Second stage: rescore just those candidates with MaxSim
query_embeddings = reranker.encode_query([query])
document_embeddings = reranker.encode_document(candidates)
scores = reranker.similarity(query_embeddings, document_embeddings)[0]
for index in scores.argsort(descending=True)[:3].tolist():
print(f"{scores[index].item():.4f} {candidates[index][:100]}")
只有这 50 个候选结果会被编码为多向量，所以你的索引仍然是普通的稠密索引，token 向量只是临时存在的。这与交叉编码器在检索-重排架构中扮演的角色相同，但多向量模型对每个候选结果的处理成本要低得多。你可以一次性批量编码文档，然后用矩阵乘法进行打分，而不是对每个查询-文档对都做一次前向传播。可运行的脚本是 retrieve_rerank.py，它会打印两个阶段的耗时。
索引
有几个向量数据库原生支持多向量的索引和打分：Qdrant 自 v1.10 起、Weaviate 自 v1.29 起、Vespa 已经支持多年、LanceDB 自 v0.15.0 起，以及 VectorChord——它为 Postgres 增加了 MaxSim 算子，这是普通 pgvector 所不具备的。Milvus 在 v2.6.4 版本中加入了这个行列，采用的是结构体数组的形式，而不是它所谓的“多向量搜索”这一不相关的功能。如果你完全不想运行服务器，LightOn 的 fast-plaid 只需 pip install 即可安装，它直接实现了 PLAID 算法，而 PyLate 则把它封装在更完整的检索栈中。
还有几个方案只能实现部分功能。OpenSearch 和 Elasticsearch 可以用 MaxSim 对候选结果进行重新打分，但不能基于它进行检索，而且 Elasticsearch 的这个功能还处于技术预览阶段，并且仅限企业版。turbopuffer 的后期交互索引目前处于私有测试阶段。
以下片段索引的是文本，但其中没有任何内容是文本特有的。encode_document 返回的是同一组 token 向量矩阵，无论文档是段落、页面图像、音频片段还是视频，因此来自视觉文档检索的 ColPali 风格模型可以原封不动地用于所有这些场景。只是每个文档的向量更多，这正是 Token Pooling 在这些场景中更值得尽早采用的原因。
fast-plaid、Qdrant、Weaviate 和 Vespa 都能直接接收 encode_document 返回的内容，因此代码在客户端库之前是完全相同的。以下是针对每个方案的可用代码片段，均基于语义搜索示例中的 4,874 个段落和 608,414 个 token 向量运行。每个片段都附带了在一台机器（RTX 3090、i7-13700K）上产生的摄取和查询耗时，除代码所示内容外未做任何调优，以便让读者了解工作的大致形态。这四个方案回答查询的速度都快于该节中 model.similarity 的 98 毫秒，其中三个在 CPU 上完成，因为 fast-plaid 是这里唯一使用 GPU 的方案。
这四个方案返回了相同的三个段落，顺序也与本文前面穷举式 PyTorch MaxSim 的结果一致，而且三个数据库将其得分复现到了小数点后四位！这是因为它们的代码片段对每个文档都进行评分，在此规模下成本可承受，并且消除了近似计算这一变量。fast-plaid 在设计上就是近似算法，因此其得分略有不同。每个方案下方的注释说明了切换到近似索引后会发生哪些变化，而排名正是从那里开始出现偏差的。
fast-plaid fast-plaid 是 LightOn 对 PLAID 的 Rust 实现，而 PLAID 正是 ColBERT 最初所基于的索引。它无需启动服务器，并且可以直接读取 encode_document 返回的张量，无需任何转换。
# pip install sentence-transformers datasets fast-plaid
from datasets import load_dataset
from fast_plaid import search
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32, convert_to_tensor=True)
query_embedding = model.encode_query(query, convert_to_tensor=True)
fast_plaid = search.FastPlaid(index="natural-questions", device="cuda")
# 4,874 documents (608,414 token vectors) indexed in 5s
fast_plaid.create(documents_embeddings=document_embeddings)
results = fast_plaid.search(queries_embeddings=query_embedding.unsqueeze(0), top_k=3) # 11ms
for index, score in results[0]:
print(f"{score:.4f} {corpus[index][:90]}")
"""
11.8828 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7676 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6758 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
index 参数是一个目录，而不仅仅是一个标签，因此索引在构建时就会写入磁盘。将新的 FastPlaid 指向同一路径即可重新打开该索引进行搜索或添加更多文档，而无需每次都从嵌入向量重新构建。在此语料库上，它占用 92 MB，而原始 float32 向量占用 311.5 MB。
这是四个方案中唯一一个近似计算，也是本节中分数与详尽 MaxSim 不完全一致的地方。PLAID 使用质心进行剪枝并存储量化残差，因此这三个分数与之前计算的 11.9192 / 11.7591 / 11.6710 相比，会在两个方向上产生百分之几的偏差。这里的排名不受影响，而这正是 PLAID 所做的权衡：它专为远大于此的语料库而设计，在那种规模下，扫描全部内容是不可行的。
Qdrant Qdrant 需要服务器：`docker run -p 6333:6333 qdrant/qdrant`。客户端也有本地模式（`QdrantClient(":memory:")`），无需服务器即可运行，但它是纯 Python 的重新实现，因此适合用来试用功能，而不是用来进行计时测试。
# pip install sentence-transformers datasets qdrant-client
from datasets import load_dataset
from qdrant_client import QdrantClient, models
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
client = QdrantClient("http://localhost:6333")
client.create_collection(
collection_name="natural-questions",
vectors_config=models.VectorParams(
size=model.get_embedding_dimension(),
distance=models.Distance.COSINE,
multivector_config=models.MultiVectorConfig(
comparator=models.MultiVectorComparator.MAX_SIM
),
# MaxSim never walks the HNSW graph, so skip building one
hnsw_config=models.HnswConfigDiff(m=0),
),
)
# 4,874 documents (608,414 token vectors) ingested in 26.3s
client.upload_points(
collection_name="natural-questions",
points=[
models.PointStruct(id=idx, vector=embedding, payload={"text": text})
for idx, (embedding, text) in enumerate(zip(document_embeddings, corpus))
],
batch_size=64,
)
results = client.query_points(
collection_name="natural-questions",
query=query_embedding,
limit=3,
with_payload=True,
).points # 18ms
for result in results:
print(f"{result.score:.4f} {result.payload['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
MAX_SIM 是 Qdrant 提供的唯一比较器，而 `hnsw_config=HnswConfigDiff(m=0)` 是他们针对后期交互字段的建议，因为这些向量用于重新评分而非图遍历。请注意，Qdrant 官方建议将后期交互保留用于对几百个候选结果进行重排，而不是扫描整个集合，这也就是“检索后重排”（Retrieve and Rerank）模式。在 4,874 篇文档的规模下，全量扫描耗时 18 毫秒且结果精确，但这个结果不能外推到更大规模。
Weaviate Weaviate 也需要服务器：`docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:1.34.0`。多向量支持需要 1.29 或更高版本，且嵌入式模式在 Windows 上不可用。
# pip install sentence-transformers datasets weaviate-client
import weaviate
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.query import MetadataQuery
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
client = weaviate.connect_to_local()
collection = client.collections.create(
"Documents",
# self_provided turns on MaxSim late interaction
vector_config=[Configure.MultiVectors.self_provided(name="colbert")],
properties=[Property(name="text", data_type=DataType.TEXT)],
)
# 4,874 documents (608,414 token vectors) ingested in 41s
with collection.batch.fixed_size(batch_size=64) as batch:
for text, embedding in zip(corpus, document_embeddings):
batch.add_object(properties={"text": text}, vector={"colbert": embedding.tolist()})
results = collection.query.near_vector(
near_vector=query_embedding.tolist(),
target_vector="colbert",
limit=3,
return_metadata=MetadataQuery(distance=True),
) # 17ms
for result in results.objects:
# Weaviate reports the MaxSim score as a negated distance
print(f"{-result.metadata.distance:.4f} {result.properties['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
client.close()
这里使用默认配置就足够了：Weaviate 的动态 ef 在 top-3 查询中解析为 100，而该排名从大约 32 起就已经是精确的了。这个余量是嵌入向量本身的特性，而非 Weaviate 的特性，因此建议在你自己的模型上确认这一点，而不是假设默认配置始终成立。
Weaviate 还支持 MUVERA 编码，在我们的测试中，这使得数据摄入速度提升了 3 倍，查询速度提升了 1.8 倍。但在当前规模下，它损失的精度远超速度提升带来的价值：正确的第三段文本甚至没有出现在其前 50 名结果中。
Vespa Vespa 同样在容器中运行，但 pyvespa 会自动为你启动它，因此无需单独执行 docker run。
# pip install sentence-transformers datasets pyvespa
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from vespa.deployment import VespaDocker
from vespa.package import (
ApplicationPackage, Document, Field, FirstPhaseRanking, Function, RankProfile, Schema,
)
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
# "dt" is a mapped dimension over the variable token count, "x" the dense 128-dim vector
package = ApplicationPackage(
name="colbert",
schema=[
Schema(
name="doc",
document=Document(fields=[
Field(name="text", type="string", indexing=["summary"]),
Field(name="colbert", type="tensor<float>(dt{}, x[128])", indexing=["attribute"]),
]),
rank_profiles=[
RankProfile(
name="colbert",
inputs=[("query(qt)", "tensor<float>(qt{}, x[128])")],
functions=[Function(
name="max_sim", # per query token take the best document token, then sum
expression="sum(reduce(sum(query(qt) * attribute(colbert), x), max, dt), qt)",
)],
first_phase=FirstPhaseRanking(expression="max_sim"),
)
],
)
],
)
app = VespaDocker(port=8080).deploy(application_package=package) # ~40s to boot
# Vespa reads a mixed tensor as {token index: vector}, for documents and queries alike
def to_tensor(embedding):
return {str(token): vector for token, vector in enumerate(embedding.tolist())}
# 4,874 documents (608,414 token vectors) ingested in ~80s
app.feed_iterable(
({"id": str(idx), "fields": {"text": text, "colbert": to_tensor(embedding)}}
for idx, (text, embedding) in enumerate(zip(corpus, document_embeddings))),
schema="doc",
)
response = app.query(body={
"yql": "select text from doc where true",
"ranking.profile": "colbert",
"hits": 3,
"input.query(qt)": to_tensor(query_embedding),
}) # ~75ms warm, ~115ms on the first call
for hit in response.hits:
print(f"{hit['relevance']:.4f} {hit['fields']['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
Vespa 是这四个方案中要求最前置结构的，因为你声明的是一个排序流水线，而不仅仅是一个索引。作为回报，你可以把 MaxSim 写成张量表达式，并精确看到它计算的内容。这个版本把 MaxSim 放在第一阶段，对全部 4,874 篇文档打分，因此输出与穷举式 MaxSim 完全一致。这刻意不是 Vespa 在大规模场景下的推荐做法：他们的 ColBERT 示例应用存储 int8 二值化向量，并把 MaxSim 移到第二阶段，对更廉价的第一阶段结果进行重排。
切换到这种分阶段设置需要谨慎：默认情况下，第二阶段只对最优的 100 个候选重新打分，而在这里，这个窗口导致三个正确段落中有两个完全未被评分。提高重排数量以覆盖你的候选集可以解决这个问题，不过在这个规模下，分阶段版本仍然比直接扫描全部内容更慢。
视觉文档检索
晚期交互是视觉文档检索的最先进技术：将文本查询与页面图像匹配，图表、表格和布局完整保留，且无需 OCR 步骤。这正是 ColPali 系列模型所做的，这些检查点通过相同的 API 加载和运行，修订版本固定了添加该模型 Sentence Transformers 配置的开放拉取请求（受支持模型中有完整列表）。图像文档以 URL、本地路径或 PIL 图像的形式传入：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("vidore/colqwen2.5-v0.2")
queries = [
"What is the variable represented on the y-axis of the graph?",
"Total outlay is maximum in which year?",
]
images = [
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (25, 128) (755, 128)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[13.8672, 12.3115, 12.1670, 11.0293],
# [ 7.2012, 14.7207, 6.9414, 6.9746]])
每个查询检索到各自的页面（对角线），第二个查询的区分度比第一个清晰得多，因为四个页面中只有一个涉及随时间变化的支出。
代码没有变化。底层上，处理器处理视觉提示词和图像块，MaxSim 对查询文本 token 与文档图像块进行评分。一个页面包含许多独立区域，这正是晚期交互在这里如此自然契合的原因，因为单个向量必须把图表、表格和三个段落平均成一个摘要。不过，这种保真度会消耗索引空间。上面的形状是一个页面的 755 个 token 向量对比查询的 25 个 token，而之前 Natural Questions 的一个段落平均约 125 个，因此在这里，token 池化比文本场景更值得提前采用。
这些是视觉语言模型（VLM），所以要为它们所需的内存做好规划。Supported Models 表格中的参数规模从 252M 到 8.8B 不等，其中小规模的模型在 CPU 上依然实用，而数十亿参数的大模型则不行。
页面图像是常见情况，但并非唯一的非文本模态。Sentence Transformers 支持文本、图像、音频和视频，而某个检查点支持其中哪些模态取决于其处理器，这一点可通过 model.modalities 查看。单个文档也可以组合多种模态，方法是传入一个字典（如 {"text": ..., "image": ...}）来代替单一值。Multimodal Embedding & Reranker Models 更全面地介绍了 Sentence Transformers 中的多模态模型，Usage 文档则明确列出了每种模态接受的具体输入格式。
音频检索
vidore/colqwen-omni-v0.1 基于 Qwen2.5-Omni 构建，支持全部四种模态。用它检索一段录制的对话与检索页面一样，只需两次调用：
# pip install -U "sentence-transformers[audio,video]"
import torch
from datasets import Audio, load_dataset
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"vidore/colqwen-omni-v0.1",
model_kwargs={"dtype": torch.bfloat16},
)
print(model.modalities)
# ['text', 'image', 'audio', 'video', 'message']
# 20 recorded conversations, averaging 28 seconds each
dataset = load_dataset("eustlb/dailytalk-conversations-grouped", split="train[:20]")
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
audio = [row["array"] for row in dataset["audio"]] # raw mono waveforms, float32 at 16 kHz
query_embeddings = model.encode_query(["medicine for car nausea"])
document_embeddings = model.encode_document(audio, batch_size=2)
scores = model.similarity(query_embeddings, document_embeddings)[0]
top_scores, top_indices = scores.topk(3)
for score, index in zip(top_scores.tolist(), top_indices.tolist()):
print(f"{score:.4f} {' / '.join(dataset[index]['texts'][:2])}")
"""
50.8902 Excuse me? Do you have anything for a carsickness? / Yes, but you look fine.
46.1028 Excuse me, could you tell me where you have got that music book? / Certainly. Let me see. Oh, it's on that shelf.
46.0514 Jeff, I'm going to the supermarket. Do you want to come with me? / I think the supermarket is closed now.
"""
ColQwen-Omni 纯粹在图像-文本对上训练，因此它的音频检索是零样本的：它从未听过训练样本，整个流程中也没有任何转写步骤。查询词说的是“恶心”（nausea），而录音里说的是“晕车”（carsickness），它依然能从二十段对话中大幅领先地选出药房那段对话。
视频检索
视频的工作方式相同，但需要对帧进行采样，否则会耗尽你的显存。其发布博客文章对此直言不讳，称视频“非常消耗内存，因此最适合短视频片段”：
import torch
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"vidore/colqwen-omni-v0.1",
model_kwargs={"dtype": torch.bfloat16},
)
# Sparse, low-resolution frames: 0.5 fps rather than the full frame rate
model[0].processing_kwargs.update(
{"video": {"max_pixels": 32 * 28 * 28, "do_sample_frames": True, "fps": 0.5}}
)
query_embeddings = model.encode_query(["How to cook Mapo Tofu?"])
document_embeddings = model.encode_document([
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/mapo_tofu.mp4",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/zhajiang_noodle.mp4",
], batch_size=1)
print(model.similarity(query_embeddings, document_embeddings))
# tensor([[53.3100, 51.0561]])
在 1 fps 和全分辨率下，同样的两段视频会产生 8,426 和 5,137 个 token 向量，峰值显存占用达 20.8 GB；而这里（采样后）只有 4,240 和 2,446 个向量，显存占用 12.5 GB，该模型本身占用 9.0 GB。两种方式的排序结果完全一致。长音频也需要同样的处理，发布博客文章建议使用 30 秒的片段，每段大约对应 800 个 token。
可解释性
由于 MaxSim 是每个查询 token 最大值的总和，因此排序可以被精确分解：文档分数的每一个点都归属于一个查询 token 和一个文档 token。这让你能够精确回答“为什么这条排在这里？”，而不是凭肉眼判断。
对于图像文档，`sentence_transformers.multi_vector_encoder.interpretability` 会将这种分解以标准 ColPali 热力图的形式叠加到页面上，既可以按整个查询聚合，也可以为每个查询 token 生成一张热力图。针对上面的支出页面询问“水资源和电力方面支出了多少？”时，这就是 water token 的分布位置：
`heatmap.py` 是可运行版本，其中包含将文档嵌入向量与 patch 网格对齐的掩码步骤。
文本文档没有可叠加的 patch 网格，但同样的分解方法同样适用。`text_similarity_map.py` 对语料库进行排序，然后逐 token 归因排名第一结果的得分，这里使用的是之前 Natural Questions 语料库上的 32M 参数模型 mxbai-edge-colbert-v0-32m：
Query: when did richmond last play in a preliminary final
Top 3 of 4874 documents by exhaustive MaxSim (191.0ms):
12.3489 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 19
12.1771 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contested betwee
12.0591 2018 UEFA Champions League Final The 2018 UEFA Champions League Final was the final match of the 201
query token best document token sim share
when since 0.9154 7.4%
did had 0.9675 7.8%
rich rich 0.9764 7.9%
mond mond 0.9856 8.0%
last to 0.9249 7.5%
play game 0.9384 7.6%
in the 0.9732 7.9%
a a 0.9587 7.8%
preliminary preliminary 0.9394 7.6%
final final 0.9654 7.8%
--------------------------------------------------------
3 special tokens 2.8038 22.7%
MaxSim score 12.3489 100.0%
rich、mond、preliminary 和 final 匹配到了它们自身，而 settled 匹配到了 since，play 匹配到了 game。特殊 token 也值得注意：其中三个贡献了 22.7% 的得分，却不携带查询的任何内容。在这张表下方，脚本会打印出段落本身，并用高亮标记出胜出的 token。
Token 池化
如果你担心索引占用空间，最有效的调节手段是存储更少的 token 向量。`HierarchicalTokenPooling` 实现了 Clavié、Chaffin 和 Adams 提出的 token 池化技术：它使用 Ward 链接和余弦距离对每个文档的 token 向量进行聚类，并用每个聚类的均值替换该聚类，从而大约保留 1 / pool_factor 的 token。在单个文档内部，大量 token 向量最终会彼此接近，因此你丢弃的大部分是冗余信息而非有效信号：
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
documents = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
pooling = HierarchicalTokenPooling(pool_factor=2)
document_embeddings = model.encode_document(documents, token_pooling=pooling)
根据你希望在何时承担这一开销，有三个地方可以应用它：
# 1. Per encode call, as above
document_embeddings = model.encode_document(documents, token_pooling=pooling)
# 2. Standalone, on embeddings you already have saved (e.g. list of [num_tokens, num_dims] tensors)
pooled = pooling.pool(document_embeddings)
# 3. Baked into the model, so every consumer of the checkpoint gets pooled documents
model.append(HierarchicalTokenPooling(pool_factor=2))
model.save_pretrained("my-pooled-colbert")
默认情况下，池化仅应用于文档，因为查询较短，而且是你无法承受失真的那一侧。在之前的 Natural Questions 语料库上，缩减比例与 pool_factor 高度吻合，对全部 608k 个 token 向量进行池化大约耗时 6 秒：
| pool_factor | Token 向量数 | 缩减比例 | float32 索引 |
|---|---|---|---|
| 1（关闭） | 608,414 | 1.00x | 311.5 MB |
| 2 | 305,438 | 1.99x | 156.4 MB |
| 3 | 204,407 | 2.98x | 104.7 MB |
| 4 | 153,936 | 3.95x | 78.8 MB |
聚类中心与查询 token 的匹配度，不如其成员中最佳匹配项；聚类越粗糙，这种差距就越明显。原始实验在 BEIR 上衡量了这一代价，发现其影响微乎其微：在 pool_factor=2 时，平均保留了未池化检索性能的 100.6%，在 pool_factor=3 时为 99.0%。将索引减半却几乎不损失性能，这非常划算，因此 2 是一个合理的起点。不过，这一代价在您的数据上具体有多大，取决于语料库本身，因此在确定因子之前，请使用评估器进行衡量。可运行的对比脚本是 token_pooling.py。
pool_factor 能推到多大，部分也取决于模型本身的特性。LightOn 的分层池化正则化正是为此而训练，它塑造了嵌入空间，使池化代价更低，并报告在 5 倍压缩下保留了 99.4% 的性能。使用该正则化器进行训练目前尚未集成到 Sentence Transformers 中，但由此产生的检查点是标准的 PyLate 模型，因此 lightonai/LateOn-hpool-regularized 可以像其他模型一样加载和池化。
加速推理
多向量模型与 Sentence Transformers 的其他部分运行在相同的后端机制上，因此您可以使用 torch（默认）、onnx 和 openvino，同时支持半精度、Flash Attention 和 torch.compile。
在 GPU 上，fp16 配合 Flash Attention 是我们测得的最高效配置，其吞吐量是 fp32 的 2.44 倍，且检索质量没有可测量的损失。Flash Attention 对多向量模型的帮助比大多数模型都大，因为文档只会被截断而不会被填充到统一长度，因此您的批次包含长度差异很大的序列，这正好可以利用去填充（unpadding）的优势：
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"lightonai/GTE-ModernColBERT-v1",
model_kwargs={"attn_implementation": "flash_attention_2", "dtype": "float16"},
)
GPU
CPU
使用非注意力查询扩展（attend=False，涵盖 Stanford-NLP 的检查点，如 colbert-ir/colbertv2.0 和 answerdotai/answerai-colbert-small-v1）的模型，在加载时会拒绝 Flash Attention。Flash Attention 会移除 attention_mask=0 的位置，因此 MaxSim 评分所依据的 [MASK] 扩展 token 将永远不会收到注意力更新。请为这些模型使用 “sdpa”。
在 CPU 上，只要架构受支持，OpenVINO 就是更好的选择，而 int8 量化能进一步提速，代价是约 0.4% 的准确率。完整的基准测试详情、导出与量化辅助工具，以及选择后端的流程图，请参阅《加速推理》。
评估模型
MultiVectorNanoBEIREvaluator 使用 MaxSim 评分运行 NanoBEIR 套件（包含 13 个小型 BEIR 子集），并且无需你方准备任何数据：
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
model = MultiVectorEncoder("lightonai/GTE-ModernColBERT-v1")
evaluator = MultiVectorNanoBEIREvaluator(batch_size=16)
results = evaluator(model)
print(f"{evaluator.primary_metric}: {results[evaluator.primary_metric]:.4f}")
这也让我们很容易验证本文开头的说法。lightonai/LateOn 和 lightonai/DenseOn 由 LightOn 在相同数据上训练，使用相同的 ModernBERT 主干和相同的 149M 参数，唯一区别在于它们是每个 token 保留一个向量，还是池化到每个文档一个向量。在全部 13 个 NanoBEIR 数据集上运行这两个模型，就能隔离出这一选择带来的差异：
| NanoBEIR 数据集 | LateOn（多向量，128d） | DenseOn（稠密，768d） |
|---|---|---|
| MSMARCO | 0.7194 | 0.6517 |
| NQ | 0.7810 | 0.7511 |
| HotpotQA | 0.9295 | 0.8802 |
| FEVER | 0.9702 | 0.9612 |
| ClimateFEVER | 0.4887 | 0.4846 |
| DBPedia | 0.6836 | 0.6748 |
| QuoraRetrieval | 0.9795 | 0.9687 |
| Touche2020 | 0.5938 | 0.5673 |
| ArguAna | 0.5562 | 0.5660 |
| NFCorpus | 0.3949 | 0.3851 |
| SciFact | 0.7978 | 0.8057 |
| SCIDOCS | 0.4469 | 0.4484 |
| FiQA2018 | 0.5871 | 0.6491 |
| 平均值 | 0.6868 | 0.6764 |
延迟交互在 13 个数据集中的 9 个以及平均值上胜出，大约领先一个 NDCG 点。它输掉的四个数据集（ArguAna、FiQA2018、SCIDOCS 和 SciFact）正是你应该预期的权衡形态：在相同模型规模下获得真实的检索质量提升，代价是索引占用空间更大，而非在每个数据集上都全面获胜。同一对模型在完整的 15 数据集 BEIR 上得分为 57.22 对 56.20，差距相当，因此这一差距并非小型基准测试带来的假象。
除了 NanoBEIR，MultiVectorInformationRetrievalEvaluator、MultiVectorRerankingEvaluator、MultiVectorTripletEvaluator 和 MultiVectorDistillationEvaluator 还覆盖了在你自己的数据上进行评估的常见设置。它们记录在《评估 API 参考》中。
从 PyLate 或 colpali-engine 迁移而来
MultiVectorEncoder 整合了两个库的建模、推理、训练和评估功能。每个 PyLate 检查点都能直接加载，Supported Models 页面列出了 colpali-engine 的检查点，以及仍需要传入的 revision 参数。如果你正在迁移，以下这些调用会发生变化：
| PyLate | Sentence Transformers |
|---|---|
| pylate.models.ColBERT(model_name_or_path=...) | MultiVectorEncoder(...) |
| model.encode(..., is_query=True) | model.encode_query(...) |
| model.encode(..., is_query=False) | model.encode_document(...) |
| pylate.scores.colbert_scores | model.similarity |
| pylate.indexes.PLAID / pylate.retrieve.ColBERT | 无对应项，保留 PyLate 的 PLAID 或参阅 Indexing |
| colpali-engine | Sentence Transformers |
|---|---|
| ColQwen2.from_pretrained(...) + ColQwen2Processor | MultiVectorEncoder(...) |
| processor.process_queries(...) + model(**batch) | model.encode_query(queries) |
| processor.process_images(...) + model(**batch) | model.encode_document(images) |
| processor.score_multi_vector(qs, ds) | model.similarity(query_embeddings, document_embeddings) |
| mask_non_image_embeddings=True | MultiVectorMask(keep_only_token_ids=[...]) |
| HierarchicalTokenPooler | HierarchicalTokenPooling |
| colpali_engine.interpretability | sentence_transformers.multi_vector_encoder.interpretability |
有一个值得指出的差异：在裸（非 ColBERT）检查点上，PyLate 的 ColBERT("bert-base-uncased") 默认应用经典配置，而 MultiVectorEncoder("bert-base-uncased") 则构建一个普通堆栈，将前缀、查询扩展和跳表留作显式选择。训练损失和评估器的对应关系，以及数据处理上的差异，详见迁移指南。
请注意，保存兼容性在所有情况下都是单向的：PyLate、Stanford-NLP ColBERT 和 colpali-engine 的检查点都能加载到 MultiVectorEncoder 中，但 MultiVectorEncoder.save_pretrained 的输出无法被它们中的任何一个加载。
支持的模型
在 Hub 上带有 multi-vector 和 sentence-transformers 标签的模型列表是保持更新的，我们正在努力为所有可用的模型打上这些标签。下面的表格是我们直接测试的对象，所以请将它们视为起点，而非完整集合。特别是对于文本检索，任何 PyLate 或 Stanford-NLP 的 ColBERT 检查点，无论是否已打上标签，都可以加载。
有些条目需要先在它们的仓库中添加一个小的 Sentence Transformers 配置，其中一些在撰写本文时仍是未合并的拉取请求。如果下面列出了某个修订版本，请使用该版本，直到该拉取请求被合并，之后直接使用模型名称即可：
model = MultiVectorEncoder("vidore/colqwen-omni-v0.1", revision="refs/pr/N")
文本检索模型
这些模型会加载其训练好的前缀 token、查询扩展和从保存的配置中恢复的标点跳过列表。
NanoBEIR 列报告了 13 个 NanoBEIR 数据集上的平均 NDCG@10（越高越好），每个数据集是 BEIR 数据集的 50 个查询子样本，作为英语文本检索质量的快速代理指标。我们使用 MultiVectorNanoBEIREvaluator 来计算主要面向英语的模型的得分。`-` 表示该模型未在此项上评估。请注意，NanoBEIR 是一个小型基准，其分数不能替代在您自己的数据上进行评估，而后者始终是选择模型的正确方法。
| 模型 | 参数量 | 维度 | NanoBEIR | 备注 |
|---|---|---|---|---|
| lightonai/LateOn-regularized | 149M | 128 | 0.6897 | - |
| lightonai/LateOn-hpool-regularized | 149M | 128 | 0.6876 | - |
| lightonai/LateOn | 149M | 128 | 0.6868 | - |
| LiquidAI/LFM2.5-ColBERT-350M | 353M | 128 | 0.6864 | 需要 trust_remote_code=True |
| lightonai/mLateOn | 307M | 128 | 0.6851 | - |
| lightonai/GTE-ModernColBERT-v1 | 149M | 128 | 0.6720 | - |
| topk-io/Iso-ModernColBERT | 149M | 128 | 0.6687 | - |
| perplexity-ai/pplx-embed-v1-late-0.6b | 596M | 128 | 0.6662 | 需要 trust_remote_code=True |
| lightonai/ColBERT-Zero | 149M | 128 | 0.6569 | - |
| answerdotai/answerai-colbert-small-v1 | 33M | 96 | 0.6550 | - |
| mixedbread-ai/mxbai-edge-colbert-v0-32m | 32M | 64 | 0.6524 | - |
| LiquidAI/LFM2-ColBERT-350M | 353M | 128 | 0.6441 | - |
| mixedbread-ai/mxbai-edge-colbert-v0-17m | 17M | 48 | 0.6407 | - |
| lightonai/colbertv2.0 | 110M | 128 | 0.6201 | - |
| lightonai/LateOn-Code | 149M | 128 | 0.6169 | - |
| lightonai/Agent-ModernColBERT | 149M | 128 | 0.6164 | - |
| lightonai/Reason-ModernColBERT | 149M | 128 | 0.6078 | - |
| colbert-ir/colbertv2.0 | 110M | 128 | 0.6053 | - |
| VAGOsolutions/SauerkrautLM-EuroColBERT | 212M | 128 | 0.5982 | - |
| antoinelouis/colbert-xm | 853M | 128 | 0.5915 | - |
| VAGOsolutions/SauerkrautLM-Multi-ModernColBERT | 149M | 128 | 0.5886 | - |
| mixedbread-ai/mxbai-colbert-large-v1 | 335M | 128 | 0.5733 | revision="refs/pr/4" |
| lightonai/LateOn-Code-edge | 17M | 48 | 0.5274 | - |
| VAGOsolutions/SauerkrautLM-Multi-Reason-ModernColBERT | 149M | 128 | 0.5267 | - |
| VAGOsolutions/SauerkrautLM-Reason-EuroColBERT | 212M | 128 | 0.4479 | - |
| NeuML/biomedbert-base-colbert | 110M | 128 | 0.4320 | - |
| yjoonjang/colbert-ko-v1 | 149M | 128 | - | - |
| ytu-ce-cosmos/turkish-colbert | 111M | 256 | - | - |
| samheym/GerColBERT | 110M | 128 | - | - |
视觉文档检索模型
ColPali 风格的模型将页面图像嵌入为文档，将文本嵌入为查询。
NanoViDoRe 列报告了 NanoViDoRe v3 上的平均 NDCG@10（越高越好）。NanoViDoRe 是一个紧凑的视觉文档检索基准，涵盖 8 个子集（计算机科学、能源、金融（英语和法语）、人力资源、工业、制药和物理）。与 NanoBEIR 一样，NanoViDoRe 是一个小型基准，不应取代在您自己数据上的评估。
| 模型 | 参数量 | 维度 | NanoViDoRe | 备注 |
|---|---|---|---|---|
| webAI-Official/webAI-ColVec1.1-8b | 8.4B | 640 | 0.6580 | 需要 trust_remote_code=True |
| webAI-Official/webAI-ColVec1.1-4b | 4.5B | 640 | 0.6520 | 需要 trust_remote_code=True |
| tencent/EVIE-Preview-4.5B | 4.54B | 128 | 0.6405 | - |
| TomoroAI/tomoro-colqwen3-embed-8b | 8.8B | 320 | 0.6206 | 需要 trust_remote_code=True |
| TomoroAI/tomoro-colqwen3-embed-4b | 4.4B | 320 | 0.6019 | 需要 trust_remote_code=True |
| vidore/colqwen2.5-v0.2 | 3.8B | 128 | 0.5402 | - |
| vidore/colqwen2.5-v0.1 | 3.8B | 128 | 0.5395 | - |
| vidore/colqwen-omni-v0.1 | 4.4B | 128 | 0.5309 | - |
| vidore/colpali-v1.3 | 2.9B | 128 | 0.4802 | - |
| vidore/colpali-v1.3-hf | 2.9B | 128 | 0.4793 | - |
| vidore/colpali-v1.2 | 2.9B | 128 | 0.4691 | - |
| vidore/colqwen2-v1.0 | 2.2B | 128 | 0.4685 | - |
| vidore/colqwen2-v0.1 | 2.2B | 128 | 0.4526 | - |
| vidore/colpali | 2.9B | 128 | 0.4516 | - |
| vidore/colpali-v1.1 | 2.9B | 128 | 0.4314 | - |
| vidore/colsmolvlm-v0.1 | 2.1B | 128 | 0.4054 | - |
| vidore/colpali-hard-v1.1 | 2.9B | 128 | 0.3949 | - |
| vidore/colSmol-500M | 507M | 128 | 0.3459 | - |
| vidore/colSmol-256M | 256M | 128 | 0.2673 | - |
| ModernVBERT/colmodernvbert | 252M | 128 | 0.2632 | - |
| vidore/colpali-v1.2-hf | 2.9B | 128 | - | - |
| vidore/colqwen2-v1.0-hf | 2.2B | 128 | - | - |
其中大多数是 LoRA 适配器仓库，适配器在加载时直接应用到其基础模型上。有些在 Hub 上还有 -merged 的姊妹版本（例如 vidore/colpali-v1.3-merged），其中适配器已合并到权重中。
这三个带 -hf 后缀的条目是 transformers 原生的 *ForRetrieval 移植版本。它们无需任何配置即可加载，但更多使用 transformers 的建模代码，较少依赖 sentence_transformers。一般来说，更推荐使用原始模型，因为这些移植版本得分大致相同。
致谢
Sentence Transformers 中的晚期交互（late interaction）建立在大量前期工作之上。感谢 Omar Khattab 和 Matei Zaharia 提出 ColBERT，本文的一切都源于此；也感谢 LightOn 团队（Antoine Chaffin、Raphael Sourty、Paulo Moura 和 Amélie Chatelain）开发了 PyLate 和 fast-plaid，这些项目多年来一直支撑着晚期交互的发展，并塑造了上述 API 的很大一部分设计。
感谢 ColPali 团队（Manuel Faysse、Hugues Sibille、Tony Wu、Bilel Omrani、Gautier Viaud、Céline Hudelot 和 Pierre Colombo）提出 ColPali 和 colpali-engine，将晚期交互引入页面图像领域；也感谢 Benjamin Clavié、Antoine Chaffin 和 Griffin Adams 在 token 池化方面的工作。
同样感谢核心 MTEB 团队，包括 Kenneth Enevoldsen 和 Roman Solomatin 等众多成员，感谢他们维护 MTEB，以及那些支撑信息检索研究持续运转的幕后工作。
还要感谢所有训练并发布了“支持的模型”中所列检查点的人。没有他们，这篇文章将没有任何可供评测的对象。
更多资源
文档
- 多向量编码器 > 用法
- 多向量编码器 > 预训练模型
- 多向量编码器 > 创建自定义模型
- 多向量编码器 > 加速推理
- 多向量编码器 > API 参考
- 安装
- 迁移指南
示例脚本
- 语义搜索
- 检索与重排
- Token 池化
- ColPali 热力图
- 文本相似度映射
- NanoBEIR 评测
训练
要了解如何在您自己的数据上训练或微调这些模型：
- 多向量编码器 > 训练概览
- 多向量编码器 > 损失函数概览
- 多向量编码器 > 训练示例
- LateOn 和 mLateOn 训练脚本：LightOn 的 PyLate 配方，涵盖 LateOn、mLateOn、DenseOn 和 mDenseOn，其中微调脚本展示了实际细节，例如如何将包含 16,384 个样本的批次拆分为 16 个样本的小批次。
Hugging Face Hub
- Hub 上的多向量模型
- Hub 上的 Sentence Transformers 数据集
配套博客文章
- 使用 Sentence Transformers 训练和微调嵌入向量模型：面向纯文本稠密嵌入向量模型的通用训练指南。
- 使用 Sentence Transformers 训练和微调重排序模型：交叉编码器训练，这是添加精确第二阶段的另一种方式。
- 使用 Sentence Transformers 训练和微调稀疏嵌入向量模型：SPLADE 及其他稀疏编码器，它们与混合检索中的后期交互结合效果良好。
- 使用 Sentence Transformers 的多模态嵌入向量与重排序模型：单向量多模态模型，是 ColPali 风格检索的稠密对应方案。
- 使用 Sentence Transformers 训练和微调多模态嵌入向量与重排序模型：包含使用单向量模型进行视觉文档检索的完整演练。
- 🪆 Matryoshka 嵌入向量模型入门：按维度压缩稠密嵌入向量，就像 token 池化按数量压缩多向量模型一样。
Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v6.0 update, it gains a fourth model type: MultiVectorEncoder, for ColBERT-style late interaction retrieval. Any PyLate checkpoint and any Stanford-NLP ColBERT checkpoint loads straight into it, and colpali-engine models for visual document retrieval can be used too, through the same familiar API you already use for dense, sparse, and reranker models. Where a regular embedding model compresses a whole text into one vector, a multi-vector model keeps one vector per token and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away, which usually means stronger retrieval at the cost of a bigger index. It's also the state of the art for visual document retrieval, where a text query is matched against page images directly, with no OCR step in between.
In this blogpost, we'll show you how to use these models: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. Everything below runs on a plain pip install -U sentence-transformers.
Table of Contents
What are Multi-Vector Models?
A dense embedding model reads a text and returns a single fixed-size vector. Everything the model noticed has to fit in those 384, 768, or 1024 numbers, and similarity is one dot product between two such summaries. This works remarkably well, but the compression is lossy in a specific way: a rare entity, an exact identifier, or one crucial clause in a long passage all have to compete for room in the same vector. A query with several requirements at once runs into the same wall. For "green sofa with wooden legs and rounded cushions", a single vector has to blend all four into one point, so a green sofa with the wrong legs ends up sitting close to the one you actually asked for.
A multi-vector model (also called a late-interaction or ColBERT-style model, after the ColBERT paper) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.
The interaction between query and document is then deferred until scoring time, which is where the name "late interaction" comes from. A cross-encoder interacts early: both texts go through the model together, which is accurate but leaves nothing to precompute, since every document has to be re-encoded for each new query. A bi-encoder, which is what the dense embedding model above is, barely interacts at all (one dot product between two finished summaries), and that is exactly what lets you encode a collection once and query it fast. Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token, which leaves far more room for the two to interact.
The MaxSim Operator
Scoring uses MaxSim: for each query token, take its highest similarity against any document token, then sum those maxima across the query.
$$ \text{MaxSim} \left(\right. Q , D \left.\right) = \underset{Q_{i} \in Q}{\sum} \underset{D_{j} \in D}{max } Q_{i} \cdot D_{j} $$
MaxSim(Q,D)=Q i∈Q∑D j∈D maxQ i⋅D j
Because the token embeddings are L2-normalized, each of those dot products is a cosine similarity in [-1, 1], so the whole sum lands within [-num_query_tokens, num_query_tokens].
You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document supports the query overall.
The alignment doesn't have to be lexical, since the token embeddings are contextualized. Encode "Where do penguins live?" against "Penguins inhabit Antarctica." with lightonai/mLateOn and the query token live finds its best match on inhabit at 0.94, a word it shares no characters with! That is the thing lexical retrieval cannot do, BM25 and its relatives need the term itself, so synonyms and paraphrases slip past them. Dense embedding models bridge that gap as well, of course. What late interaction adds is that it does so without giving up the other direction: when an exact match is what matters (a product code, a surname, a function name), MaxSim still has that token sitting there on its own, where a single-vector model had to average it in with everything else. It isn't one-to-one either, since several query tokens routinely settle on the same document token.
What You Gain, and What It Costs
You gain retrieval quality, particularly on queries where one specific piece of a document is what makes it relevant, on multi-requirement queries like the sofa above where each requirement gets to find its own evidence, and on out-of-domain data where a dense model's compression was tuned for a different distribution. That compression is learned from the training queries, so the model learns to keep what they needed and drop everything else, which may include exactly what your production queries ask about. The effect grows with document length, since more text has to fit in the same fixed vector.
The cost is index size. One vector per token instead of one vector per document is a lot more vectors, only partly offset by the smaller dimension. Encoding 4,874 Natural Questions passages with lightonai/LateOn produced 608,414 token vectors, an average of 124.8 per passage:
| Representation | Vectors | Dimensions | float32 size |
|---|---|---|---|
Dense, all-MiniLM-L6-v2 | 4,874 | 384 | 7.5 MB |
Dense, gte-modernbert-base | 4,874 | 768 | 15.0 MB |
Multi-vector, LateOn | 608,414 | 128 | 311.5 MB |
That's about 42x the storage of the MiniLM index, or 62 KiB per passage. However, indexes are often compressed, e.g. the same 608,414 vectors take 92 MB as a fast-plaid index, since PLAID stores a centroid id plus a quantized residual per vector rather than the vector itself. For scale, a 4096-dimensional dense model like Qwen3-Embedding-8B would need about 80 MB for these same 4,874 passages, so a compressed multi-vector index sits in the same territory as the dense indexes people already run. Token Pooling cuts the vector count before any of that, and Retrieve and Rerank avoids building an index at all.
PyLate comes up throughout this post, so briefly: Sentence Transformers handled dense and sparse models but not late interaction, so LightOn built PyLate on top of it to close that gap, adding the training, inference, and retrieval pieces these models need. Much of what you'll load below was trained with it, and LightOn built an ecosystem around it too, including fast-plaid, the late-interaction index that turns up in Indexing. With v6.0 those capabilities live in Sentence Transformers itself.
With the tradeoff in mind, let's get a model running.
Installation
Multi-vector models work with a plain install:
pip install -U sentence-transformers
For ColPali-style visual document retrieval, you also need the image dependencies (see Installation for all extras, and Multimodal Embedding & Reranker Models for multimodal support in general):
pip install -U "sentence-transformers[image]"
Sentence Transformers v6.0 requires
transformersv5.x,torch2.2+, andhuggingface-hubv1.x. If you pin any of those lower, plan the upgrade first. See the Migration Guide for the full list of breaking changes.
Loading a Model
Loading a multi-vector model looks exactly like loading any other Sentence Transformers model:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/LateOn")
To find models that work, look for the multi-vector and sentence-transformers tags on the Hub. Any model with those tags loads with the line above, whether it started life as a PyLate checkpoint, a Stanford-NLP ColBERT checkpoint, or a ColPali-family model for visual document retrieval. We're working through the ecosystem to get that tag onto every model that works, so the list keeps growing.
Underneath, MultiVectorEncoder reads each of the formats these checkpoints have been published in over the years, so PyLate and Stanford-NLP checkpoints load directly even where the tag hasn't been added yet:
from sentence_transformers import MultiVectorEncoder
# Native Sentence Transformers checkpoints. PyLate builds on the same schema,
# so any PyLate checkpoint loads identically
model = MultiVectorEncoder("lightonai/LateOn")
model = MultiVectorEncoder("mixedbread-ai/mxbai-edge-colbert-v0-17m")
model = MultiVectorEncoder("LiquidAI/LFM2.5-ColBERT-350M", trust_remote_code=True)
# Any Stanford-NLP ColBERT checkpoint, detected via the `HF_ColBERT` architecture
# marker. The inline projection weight and the recipe come from `artifact.metadata`
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
model = MultiVectorEncoder("answerdotai/answerai-colbert-small-v1")
# A bare transformer: a fresh random projection is appended, so training is required
model = MultiVectorEncoder("answerdotai/ModernBERT-base")
Visual document retrieval models are the exception. ColPali-family checkpoints ship in colpali-engine's own format, which carries no information Sentence Transformers can use, so each one needs a small configuration added to its repository before it loads. Most of that work is done and waiting to be merged. See Supported Models for the current state and how to load them today.
Inspecting What a Checkpoint Configured
Multi-vector models carry a handful of recipe knobs that differ per checkpoint: marker prefixes for queries and documents, length caps, whether queries are padded out with [MASK] tokens, and which tokens are skipped when scoring documents. All of them live in the module configs, so print(model) shows you exactly what you loaded. Here's the original ColBERTv2 checkpoint, which pads every query to exactly 32 tokens and truncates documents at 180:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
print(model)
"""
MultiVectorEncoder(
(0): Transformer({..., 'document_length': 180,
'query_expansion': {'strategy': 'fixed', 'attend': False, 'token': None, 'length': 32}})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, ...})
(2): MultiVectorMask({'skiplist_words': ['!', '"', '#', ...], 'skiplist_tasks': ['document'], ...})
(3): Normalize({...})
)
"""
print(model.prompts)
# {'query': '[unused0] ', 'document': '[unused1] '}
That's the classic ColBERT pipeline: a Transformer producing contextualized token embeddings, a token-level Dense projecting each of them to 128 dimensions, a MultiVectorMask deciding which tokens count during scoring, and a token-level Normalize. Other checkpoints fill in different values. lightonai/GTE-ModernColBERT-v1 uses the same four modules with [Q] and [D] prompts, no query expansion, and caps of 48 and 300.
You rarely need to touch any of this, since every released checkpoint configures its own. It matters when you build a model from a bare backbone, which is covered in Creating Custom Models.
One value is worth checking against your own data, though. document_length truncates, so anything past it never reaches the index. For example, a 662-token passage through LateOn's cap of 300 comes back as 273 vectors, with the rest of the passage simply gone. Most of these checkpoints were trained on short passages, so if your chunks are longer than the cap, you can lift it for a single call with encode_document(..., processing_kwargs={"text": {"max_length": 512}}), keeping in mind that you would be running the model past the length it was trained on and that the index grows roughly in proportion. Multi-vector models tend to tolerate that well. On MLDR, a long-document retrieval benchmark, the multilingual siblings of the pair above show the gap clearly: mLateOn scores 77.92 against mDenseOn's 51.59.
Encoding Queries and Documents
Multi-vector models are asymmetric: queries and documents go through different prefixes, different length caps, and different scoring masks. Unlike many dense models, where the two are interchangeable, encode_query() and encode_document() are required to get correct embeddings:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/mLateOn")
queries = ["What is the capital of France?"]
documents = [
"Paris is the capital of France.",
"Berlin is the capital and largest city of Germany, by both area and population.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape)
# (10, 128)
print(document_embeddings[0].shape, document_embeddings[1].shape)
# (10, 128) (19, 128)
Note what you get back: a list of 2D tensors, one per input, each of shape (num_tokens, embedding_dim). Unlike dense embeddings, you can't stack these into one rectangular tensor, because every input has its own token count. The second document is longer than the first, so it comes back as a taller matrix.
Each call applies the model's own recipe for you. encode_query prepends the query marker, expands the query to a fixed length if the checkpoint asks for it, and caps it at the query length. encode_document prepends the document marker, caps at the document length, and drops any skiplisted tokens (punctuation, for most checkpoints) from the scoring mask.
The usual encode() arguments all still apply, so batch_size, show_progress_bar, convert_to_tensor, device, and multi-process pools work the way you'd expect:
document_embeddings = model.encode_document(
documents,
batch_size=64,
convert_to_tensor=True,
show_progress_bar=True,
)
Scoring with MaxSim
model.similarity() computes the full all-pairs MaxSim matrix:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/LateOn")
query_embeddings = model.encode_query(["Which planet is known as the Red Planet?"])
document_embeddings = model.encode_document([
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
])
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[10.7942, 11.1104, 10.9743, 11.0811]])
Mars wins, as it should. Note how close the runners-up are: Saturn also contains the literal phrase "the Red Planet", and Jupiter is a planet with a red spot, so a token-level operator has plenty to latch onto in all three. The ordering is what matters.
Scores often sit this close together, as GLInt shows by measuring the spread across a full candidate pool. MaxSim takes a maximum per query token, so a document will usually give every query token some decent best match, and scores start from a floor. Contextualized token embeddings are also anisotropic, clustering in a narrow cone rather than spreading out, so even arbitrary token pairs tend to score high.
There is also model.similarity_pairwise(), for when you already have matched pairs and just want the pair scores instead of the full similarity matrix:
scores = model.similarity_pairwise(query_embeddings, document_embeddings[:1])
print(scores)
# tensor([10.7942])
Score Magnitude and MeanMaxSim
MaxSim sums over query tokens, so its magnitude scales with how many query tokens there are, which means you can't compare scores across models with different query recipes. LateOn encodes the Red Planet query above as 12 tokens. Run that same query and those same documents through ColBERTv2, which pads and truncates every query to exactly 32 tokens, and the scores land in a completely different range:
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
# ... same encode_query / encode_document / similarity calls ...
print(scores)
# tensor([[12.7970, 27.1945, 23.8495, 24.5656]])
Within one model the ordering is all you need, but if you want scores on a bounded scale, switch the model's similarity function to MeanMaxSim, which divides by the query token count. Back on LateOn:
model = MultiVectorEncoder("lightonai/LateOn", similarity_fn_name="meanmaxsim")
# or on an already-loaded model: model.similarity_fn_name = "meanmaxsim"
print(model.similarity(query_embeddings, document_embeddings))
# tensor([[0.8995, 0.9259, 0.9145, 0.9234]])
Now every score is an average cosine similarity in [-1, 1], although you'll only see [0, 1] in practice.
Semantic Search
If your corpus is small, exhaustive MaxSim over all of it is the simplest thing that works. Encode the corpus once, then score each query against everything:
import time
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
# Several questions share an answer passage, so drop repeats but keep the order
corpus = list(dict.fromkeys(dataset["answer"])) # 5,000 rows -> 4,874 passages
model = MultiVectorEncoder("lightonai/LateOn")
corpus_embeddings = model.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)
query = "when did richmond last play in a preliminary final"
start = time.perf_counter()
query_embeddings = model.encode_query([query], convert_to_tensor=True)
scores = model.similarity(query_embeddings, corpus_embeddings)[0] # 98ms
top_scores, top_indices = scores.topk(3)
print(f"Search took {(time.perf_counter() - start) * 1000:.1f}ms")
for score, index in zip(top_scores.tolist(), top_indices.tolist()):
print(f"{score:.4f} {corpus[index][:100]}")
"""
Search took 122.7ms
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contest
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fou
"""
Those 4,874 passages encoded in 20 seconds on an RTX 3090, and each search takes about 120ms end to end, most of that the MaxSim scoring against all 608,414 token vectors. This is exact, but it scales linearly in total corpus tokens and keeps every token vector in memory, so reach for it when you have a few thousand documents rather than a few million. The runnable version of this script is semantic_search.py.
Past that size you want a real late-interaction index, which Sentence Transformers doesn't ship. It doesn't need to: these indexes store whatever encode_document produced, so you encode here and hand the token embeddings to something built for them. Indexing has working snippets for four of the options, and the section directly below covers how to skip the index entirely.
Retrieve and Rerank
You can also get late-interaction quality without maintaining a late-interaction index, by using a multi-vector model as your reranker. A fast bi-encoder narrows a large corpus to a handful of candidates, then the multi-vector model rescores only those:
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder, SentenceTransformer
from sentence_transformers.util import semantic_search
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:50000]")
corpus = list(dict.fromkeys(dataset["answer"]))
retriever = SentenceTransformer("jinaai/jina-embeddings-v5-text-nano-retrieval")
reranker = MultiVectorEncoder("perplexity-ai/pplx-embed-v1-late-0.6b", trust_remote_code=True)
# First stage: index the corpus once with a fast bi-encoder
corpus_embeddings = retriever.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)
# Retrieve the top 50
query = "when did richmond last play in a preliminary final"
hits = semantic_search(retriever.encode_query([query], convert_to_tensor=True), corpus_embeddings, top_k=50)[0]
candidates = [corpus[hit["corpus_id"]] for hit in hits]
# Second stage: rescore just those candidates with MaxSim
query_embeddings = reranker.encode_query([query])
document_embeddings = reranker.encode_document(candidates)
scores = reranker.similarity(query_embeddings, document_embeddings)[0]
for index in scores.argsort(descending=True)[:3].tolist():
print(f"{scores[index].item():.4f} {candidates[index][:100]}")
Only the 50 candidates are ever encoded as multi-vectors, so your index stays a normal dense index and the token vectors are transient. This is the same role a cross-encoder plays in a retrieve-and-rerank stack, but a multi-vector model is considerably cheaper per candidate. You encode the documents in one batch and score them with a matrix multiplication, instead of one forward pass per query-document pair. The runnable script is retrieve_rerank.py, which prints the timings of both stages.
Indexing
Several vector databases index and score multi-vectors natively: Qdrant since v1.10, Weaviate since v1.29, Vespa for years now, LanceDB since v0.15.0, and VectorChord, which adds a MaxSim operator to Postgres that plain pgvector doesn't have. Milvus joined them in v2.6.4, under array-of-structs rather than the unrelated feature it calls multi-vector search. If you would rather not run a server at all, LightOn's fast-plaid is a pip install away and implements PLAID directly, and PyLate wraps it in a fuller retrieval stack.
A few others get you partway. OpenSearch and Elasticsearch can rescore candidates with MaxSim but not retrieve on it, and the Elasticsearch field is additionally in technical preview and Enterprise-tier. turbopuffer has late-interaction indexing in private beta.
The snippets below index text, but nothing in them is text-specific. encode_document hands back the same list of token-vector matrices whether the document was a passage, a page image, an audio clip, or a video, so the ColPali-style models from Visual Document Retrieval go into any of these unchanged. There are simply more vectors per document, which is what makes Token Pooling worth reaching for sooner there.
fast-plaid, Qdrant, Weaviate, and Vespa all take exactly what encode_document returns, so the code is the same up to the client library. Here's a working snippet for each, run against the 4,874 passages and 608,414 token vectors from the Semantic Search example. Each one carries the ingestion and query times it produced on one machine (RTX 3090, i7-13700K), with no tuning beyond what the code shows, to give a sense of the shape of the work. All four answer the query faster than the 98ms model.similarity took in that section, and three of them do it on the CPU, since fast-plaid is the only one here using the GPU.
All four returned the same three passages in the same order as the exhaustive PyTorch MaxSim earlier in this post, and the three databases reproduce its scores to four decimals! That is because their snippets score every document, which is affordable at this size and removes approximation as a variable. fast-plaid is approximate by design, so its scores differ slightly. The notes under each one say what changes when you switch to an approximate index, which is where rankings start to drift.
fast-plaid fast-plaid is LightOn's Rust implementation of PLAID, the index ColBERT was originally built around. There's no server to start, and it reads the tensors encode_document hands back without any conversion.
# pip install sentence-transformers datasets fast-plaid
from datasets import load_dataset
from fast_plaid import search
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32, convert_to_tensor=True)
query_embedding = model.encode_query(query, convert_to_tensor=True)
fast_plaid = search.FastPlaid(index="natural-questions", device="cuda")
# 4,874 documents (608,414 token vectors) indexed in 5s
fast_plaid.create(documents_embeddings=document_embeddings)
results = fast_plaid.search(queries_embeddings=query_embedding.unsqueeze(0), top_k=3) # 11ms
for index, score in results[0]:
print(f"{score:.4f} {corpus[index][:90]}")
"""
11.8828 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7676 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6758 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
The index argument is a directory, not just a label, so the index is written to disk as it is built. Pointing a new FastPlaid at the same path reopens it for searching or for adding more documents, instead of rebuilding from the embeddings each time. On this corpus it occupies 92 MB, against 311.5 MB for the raw float32 vectors.
This is the only one of the four that is approximate, and it is the one place in this section where the scores do not match the exhaustive MaxSim. PLAID prunes with centroids and stores quantized residuals, so the three scores drift by a few hundredths in both directions against the 11.9192 / 11.7591 / 11.6710 computed earlier. The ranking is unaffected here, and that is the trade PLAID is making: it was designed for corpora far larger than this one, where scanning everything is not an option.
Qdrant Qdrant needs a server: docker run -p 6333:6333 qdrant/qdrant. The client also has a local mode (QdrantClient(":memory:")) that needs no server, but it's a pure-Python reimplementation, so use it for trying things out rather than for timing them.
# pip install sentence-transformers datasets qdrant-client
from datasets import load_dataset
from qdrant_client import QdrantClient, models
from sentence_transformers import MultiVectorEncoder
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
client = QdrantClient("http://localhost:6333")
client.create_collection(
collection_name="natural-questions",
vectors_config=models.VectorParams(
size=model.get_embedding_dimension(),
distance=models.Distance.COSINE,
multivector_config=models.MultiVectorConfig(
comparator=models.MultiVectorComparator.MAX_SIM
),
# MaxSim never walks the HNSW graph, so skip building one
hnsw_config=models.HnswConfigDiff(m=0),
),
)
# 4,874 documents (608,414 token vectors) ingested in 26.3s
client.upload_points(
collection_name="natural-questions",
points=[
models.PointStruct(id=idx, vector=embedding, payload={"text": text})
for idx, (embedding, text) in enumerate(zip(document_embeddings, corpus))
],
batch_size=64,
)
results = client.query_points(
collection_name="natural-questions",
query=query_embedding,
limit=3,
with_payload=True,
).points # 18ms
for result in results:
print(f"{result.score:.4f} {result.payload['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
MAX_SIM is the only comparator Qdrant offers, and hnsw_config=HnswConfigDiff(m=0) is their recommendation for late-interaction fields, since the vectors are used for rescoring rather than graph traversal. Note that Qdrant themselves suggest reserving late interaction for reranking a few hundred candidates rather than scanning a whole collection, which is the Retrieve and Rerank pattern. At 4,874 documents the full scan costs 18ms and is exact, but that doesn't extrapolate.
Weaviate Weaviate needs a server too: docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:1.34.0. Multi-vector support needs 1.29 or newer, and the embedded mode isn't available on Windows.
# pip install sentence-transformers datasets weaviate-client
import weaviate
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from weaviate.classes.config import Configure, DataType, Property
from weaviate.classes.query import MetadataQuery
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
client = weaviate.connect_to_local()
collection = client.collections.create(
"Documents",
# self_provided turns on MaxSim late interaction
vector_config=[Configure.MultiVectors.self_provided(name="colbert")],
properties=[Property(name="text", data_type=DataType.TEXT)],
)
# 4,874 documents (608,414 token vectors) ingested in 41s
with collection.batch.fixed_size(batch_size=64) as batch:
for text, embedding in zip(corpus, document_embeddings):
batch.add_object(properties={"text": text}, vector={"colbert": embedding.tolist()})
results = collection.query.near_vector(
near_vector=query_embedding.tolist(),
target_vector="colbert",
limit=3,
return_metadata=MetadataQuery(distance=True),
) # 17ms
for result in results.objects:
# Weaviate reports the MaxSim score as a negated distance
print(f"{-result.metadata.distance:.4f} {result.properties['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
client.close()
Defaults are enough here: Weaviate's dynamic ef resolves to 100 for a top-3 query, and this ranking is already exact from about 32 upward. That margin is a property of the embeddings rather than of Weaviate, so it's worth confirming on your own model instead of assuming the defaults hold.
Weaviate also supports MUVERA encoding, which made ingestion 3x faster and queries 1.8x faster in our test. It cost far more accuracy than that speed is worth at this size though: the correct third passage didn't appear even in its top 50.
Vespa Vespa also runs in a container, but pyvespa starts it for you, so there's no separate docker run.
# pip install sentence-transformers datasets pyvespa
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from vespa.deployment import VespaDocker
from vespa.package import (
ApplicationPackage, Document, Field, FirstPhaseRanking, Function, RankProfile, Schema,
)
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
corpus = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
query = "when did richmond last play in a preliminary final"
document_embeddings = model.encode_document(corpus, batch_size=32)
query_embedding = model.encode_query(query)
# "dt" is a mapped dimension over the variable token count, "x" the dense 128-dim vector
package = ApplicationPackage(
name="colbert",
schema=[
Schema(
name="doc",
document=Document(fields=[
Field(name="text", type="string", indexing=["summary"]),
Field(name="colbert", type="tensor<float>(dt{}, x[128])", indexing=["attribute"]),
]),
rank_profiles=[
RankProfile(
name="colbert",
inputs=[("query(qt)", "tensor<float>(qt{}, x[128])")],
functions=[Function(
name="max_sim", # per query token take the best document token, then sum
expression="sum(reduce(sum(query(qt) * attribute(colbert), x), max, dt), qt)",
)],
first_phase=FirstPhaseRanking(expression="max_sim"),
)
],
)
],
)
app = VespaDocker(port=8080).deploy(application_package=package) # ~40s to boot
# Vespa reads a mixed tensor as {token index: vector}, for documents and queries alike
def to_tensor(embedding):
return {str(token): vector for token, vector in enumerate(embedding.tolist())}
# 4,874 documents (608,414 token vectors) ingested in ~80s
app.feed_iterable(
({"id": str(idx), "fields": {"text": text, "colbert": to_tensor(embedding)}}
for idx, (text, embedding) in enumerate(zip(corpus, document_embeddings))),
schema="doc",
)
response = app.query(body={
"yql": "select text from doc where true",
"ranking.profile": "colbert",
"hits": 3,
"input.query(qt)": to_tensor(query_embedding),
}) # ~75ms warm, ~115ms on the first call
for hit in response.hits:
print(f"{hit['relevance']:.4f} {hit['fields']['text'][:90]}")
"""
11.9192 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve
11.7591 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes
11.6710 Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo
"""
Vespa asks for the most upfront structure of the four, because you're declaring a ranking pipeline rather than just an index. In exchange you get to write MaxSim out as a tensor expression and see exactly what it computes. This version puts MaxSim in first-phase over where true, which scores all 4,874 documents and is why the output matches exhaustive MaxSim exactly. It's deliberately not what Vespa recommends at scale: their ColBERT sample app stores int8-binarized vectors and moves MaxSim into second-phase to rerank a cheaper first stage.
Moving to that phased setup needs care: second-phase rescores only the best 100 candidates by default, and here that window left two of the three correct passages unscored entirely. Raising rerank-count to cover your candidate set fixes that, though at this size the phased version still came out slower than simply scanning everything.
Visual Document Retrieval
Late interaction is the state of the art for visual document retrieval: matching a text query against page images, with charts, tables, and layout intact, and no OCR step. This is what the ColPali family of models does, and those checkpoints load and run through the same API, with the revision pinning the open pull request that adds this one's Sentence Transformers configuration (Supported Models has the full list). Image documents are passed as URLs, local paths, or PIL images:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("vidore/colqwen2.5-v0.2")
queries = [
"What is the variable represented on the y-axis of the graph?",
"Total outlay is maximum in which year?",
]
images = [
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (25, 128) (755, 128)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[13.8672, 12.3115, 12.1670, 11.0293],
# [ 7.2012, 14.7207, 6.9414, 6.9746]])
Each query retrieves its own page (the diagonal), and the second query separates much more cleanly than the first, since only one of the four pages is about outlay over time.
The code is unchanged. Underneath, the processor handles the visual prompt and the image patches, and MaxSim scores query text tokens against document image patches. A page holds many separate regions, which is exactly what makes late interaction a natural fit here, since a single vector would have to average a chart, a table, and three paragraphs into one summary. That fidelity costs index space, though. The shapes above are 755 token vectors for one page against 25 for the query, where a Natural Questions passage from earlier averaged about 125, so token pooling is worth reaching for earlier here than it is for text.
These are VLMs, so plan for the memory they need. The table in Supported Models runs from 252M to 8.8B parameters, and the small end of it stays practical on CPU where the multi-billion ones don't.
Page images are the common case, but they're not the only non-text modality. Sentence Transformers accepts text, images, audio, and video, and a checkpoint supports whichever of those its processor does, which model.modalities reports. A single document can combine modalities too, by passing a dict like {"text": ..., "image": ...} in place of a bare value. Multimodal Embedding & Reranker Models covers multimodal models in Sentence Transformers more broadly, and the Usage documentation lists exactly which input formats each modality accepts.
Audio Retrieval
vidore/colqwen-omni-v0.1 is built on Qwen2.5-Omni and takes all four modalities. Retrieving a recorded conversation with it is the same two calls as retrieving a page:
# pip install -U "sentence-transformers[audio,video]"
import torch
from datasets import Audio, load_dataset
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"vidore/colqwen-omni-v0.1",
model_kwargs={"dtype": torch.bfloat16},
)
print(model.modalities)
# ['text', 'image', 'audio', 'video', 'message']
# 20 recorded conversations, averaging 28 seconds each
dataset = load_dataset("eustlb/dailytalk-conversations-grouped", split="train[:20]")
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
audio = [row["array"] for row in dataset["audio"]] # raw mono waveforms, float32 at 16 kHz
query_embeddings = model.encode_query(["medicine for car nausea"])
document_embeddings = model.encode_document(audio, batch_size=2)
scores = model.similarity(query_embeddings, document_embeddings)[0]
top_scores, top_indices = scores.topk(3)
for score, index in zip(top_scores.tolist(), top_indices.tolist()):
print(f"{score:.4f} {' / '.join(dataset[index]['texts'][:2])}")
"""
50.8902 Excuse me? Do you have anything for a carsickness? / Yes, but you look fine.
46.1028 Excuse me, could you tell me where you have got that music book? / Certainly. Let me see. Oh, it's on that shelf.
46.0514 Jeff, I'm going to the supermarket. Do you want to come with me? / I think the supermarket is closed now.
"""
ColQwen-Omni was trained purely on image-text pairs, so its audio retrieval is zero-shot: it never heard a training example, and there is no transcription step anywhere in the pipeline. The query says nausea where the recording says carsickness, and it still picks the pharmacy conversation out of twenty by a wide margin.
Video Retrieval
Video works the same way, but sample the frames or it will eat your VRAM. Its release blogpost is blunt about this, that video "is very memory-intensive, so it's best suited for short clips":
import torch
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"vidore/colqwen-omni-v0.1",
model_kwargs={"dtype": torch.bfloat16},
)
# Sparse, low-resolution frames: 0.5 fps rather than the full frame rate
model[0].processing_kwargs.update(
{"video": {"max_pixels": 32 * 28 * 28, "do_sample_frames": True, "fps": 0.5}}
)
query_embeddings = model.encode_query(["How to cook Mapo Tofu?"])
document_embeddings = model.encode_document([
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/mapo_tofu.mp4",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/zhajiang_noodle.mp4",
], batch_size=1)
print(model.similarity(query_embeddings, document_embeddings))
# tensor([[53.3100, 51.0561]])
At 1 fps and full resolution the same pair of videos produces 8,426 and 5,137 token vectors and peaks at 20.8 GB of VRAM, against 4,240 and 2,446 vectors and 12.5 GB here, for a model that occupies 9.0 GB on its own. The ranking is identical either way. Long audio wants the same treatment, and the release blogpost recommends 30-second chunks, which come to roughly 800 tokens each.
Interpretability
Because MaxSim is a sum of per-query-token maxima, a ranking decomposes exactly: every point of a document's score belongs to one query token and one document token. That lets you answer "why did this rank here?" precisely, rather than by eye.
For image documents, sentence_transformers.multi_vector_encoder.interpretability overlays that decomposition onto the page as the standard ColPali heatmap, either aggregated over the query or one map per query token. Asking "How much was spent on water resources and power?" against the outlays page from above, this is where the water token went:
heatmap.py is the runnable version, including the masking step that lines the document embedding up with the patch grid.
Text documents have no patch grid to overlay, but the same decomposition applies. text_similarity_map.py ranks a corpus and then attributes the top hit's score token by token, here on the Natural Questions corpus from earlier with the 32M-parameter mxbai-edge-colbert-v0-32m:
Query: when did richmond last play in a preliminary final
Top 3 of 4874 documents by exhaustive MaxSim (191.0ms):
12.3489 Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 19
12.1771 2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contested betwee
12.0591 2018 UEFA Champions League Final The 2018 UEFA Champions League Final was the final match of the 201
query token best document token sim share
when since 0.9154 7.4%
did had 0.9675 7.8%
rich rich 0.9764 7.9%
mond mond 0.9856 8.0%
last to 0.9249 7.5%
play game 0.9384 7.6%
in the 0.9732 7.9%
a a 0.9587 7.8%
preliminary preliminary 0.9394 7.6%
final final 0.9654 7.8%
--------------------------------------------------------
3 special tokens 2.8038 22.7%
MaxSim score 12.3489 100.0%
rich, mond, preliminary, and final matched themselves, while when settled on since and play on game. The special tokens are worth noticing too: three of them contribute 22.7% of the score while carrying none of the query's content. Below this table the script prints the passage itself, with the winning tokens highlighted in place.
Token Pooling
If the index footprint worries you, the most effective knob is to store fewer token vectors. HierarchicalTokenPooling implements the token pooling technique from Clavié, Chaffin, and Adams: it clusters each document's token vectors with Ward linkage on cosine distance and replaces each cluster with its mean, keeping roughly 1 / pool_factor of the tokens. Within one document a lot of token vectors end up close to each other, so much of what you drop is redundancy rather than signal:
from datasets import load_dataset
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:5000]")
documents = list(dict.fromkeys(dataset["answer"]))
model = MultiVectorEncoder("lightonai/LateOn")
pooling = HierarchicalTokenPooling(pool_factor=2)
document_embeddings = model.encode_document(documents, token_pooling=pooling)
There are three places to apply it, depending on when you want to pay for it:
# 1. Per encode call, as above
document_embeddings = model.encode_document(documents, token_pooling=pooling)
# 2. Standalone, on embeddings you already have saved (e.g. list of [num_tokens, num_dims] tensors)
pooled = pooling.pool(document_embeddings)
# 3. Baked into the model, so every consumer of the checkpoint gets pooled documents
model.append(HierarchicalTokenPooling(pool_factor=2))
model.save_pretrained("my-pooled-colbert")
By default, pooling applies to documents only, since queries are short and are the side you can't afford to distort. On the Natural Questions corpus from earlier, the reduction tracks pool_factor closely, and pooling all 608k token vectors took about 6 seconds:
pool_factor | Token vectors | Reduction | float32 index |
|---|---|---|---|
| 1 (off) | 608,414 | 1.00x | 311.5 MB |
| 2 | 305,438 | 1.99x | 156.4 MB |
| 3 | 204,407 | 2.98x | 104.7 MB |
| 4 | 153,936 | 3.95x | 78.8 MB |
A cluster mean is a worse match for a query token than the best of its members was, and the coarser the clusters, the more that shows. The original experiments measured that cost on BEIR and found very little of it: 100.6% of the unpooled retrieval performance on average at pool_factor=2, and 99.0% at pool_factor=3. Halving your index for free is a good deal, so 2 is a reasonable place to start. How much it costs on your data is corpus-specific though, so measure it with an evaluator before you settle on a factor. The runnable comparison is token_pooling.py.
How far you can push pool_factor is also partly a property of the model. LightOn's hierarchical pooling regularization trains for exactly that, shaping the embedding space so pooling costs less and reporting 99.4% retention at 5x compression. Training with that regularizer isn't in Sentence Transformers yet, but the resulting checkpoints are ordinary PyLate models, so lightonai/LateOn-hpool-regularized loads and pools like any other.
Speeding Up Inference
Multi-vector models run through the same backend machinery as the rest of Sentence Transformers, so you get torch (default), onnx, and openvino, alongside half precision, Flash Attention, and torch.compile.
On GPU, fp16 with Flash Attention is the best configuration we measured, at 2.44x the throughput of fp32 with no measurable retrieval quality loss. Flash Attention helps multi-vector models more than most, because documents are only truncated and never padded to a shared length, so your batches have widely varying sequence lengths that unpadding can exploit:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"lightonai/GTE-ModernColBERT-v1",
model_kwargs={"attn_implementation": "flash_attention_2", "dtype": "float16"},
)
GPU
CPU
Models with non-attend query expansion (
attend=False, which covers the Stanford-NLP checkpoints likecolbert-ir/colbertv2.0andanswerdotai/answerai-colbert-small-v1) reject Flash Attention at load time. Flash Attention stripsattention_mask=0positions, so the[MASK]expansion tokens that MaxSim scores would never receive an attention update. Use"sdpa"for those models.
On CPU, OpenVINO is your better bet where the architecture is supported, and int8 quantization buys a further speedup at a cost of about 0.4% accuracy. See Speeding up Inference for the full benchmark details, the export and quantization helpers, and a flowchart for picking a backend.
Evaluating a Model
MultiVectorNanoBEIREvaluator runs the NanoBEIR suite of 13 small BEIR subsets with MaxSim scoring, and needs no data preparation on your side:
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
model = MultiVectorEncoder("lightonai/GTE-ModernColBERT-v1")
evaluator = MultiVectorNanoBEIREvaluator(batch_size=16)
results = evaluator(model)
print(f"{evaluator.primary_metric}: {results[evaluator.primary_metric]:.4f}")
This also makes it easy to check the claim from the top of this post. lightonai/LateOn and lightonai/DenseOn were trained by LightOn on the same data with the same ModernBERT backbone and the same 149M parameters, differing only in whether they keep one vector per token or pool down to one per document. Running both over all 13 NanoBEIR datasets isolates what that choice buys:
| NanoBEIR dataset | LateOn (multi-vector, 128d) | DenseOn (dense, 768d) |
|---|---|---|
| MSMARCO | 0.7194 | 0.6517 |
| NQ | 0.7810 | 0.7511 |
| HotpotQA | 0.9295 | 0.8802 |
| FEVER | 0.9702 | 0.9612 |
| ClimateFEVER | 0.4887 | 0.4846 |
| DBPedia | 0.6836 | 0.6748 |
| QuoraRetrieval | 0.9795 | 0.9687 |
| Touche2020 | 0.5938 | 0.5673 |
| ArguAna | 0.5562 | 0.5660 |
| NFCorpus | 0.3949 | 0.3851 |
| SciFact | 0.7978 | 0.8057 |
| SCIDOCS | 0.4469 | 0.4484 |
| FiQA2018 | 0.5871 | 0.6491 |
| Mean | 0.6868 | 0.6764 |
Late interaction wins on 9 of the 13 datasets and on the mean, by roughly one NDCG point. The four it loses (ArguAna, FiQA2018, SCIDOCS, and SciFact) are the shape of the tradeoff you should expect: a real gain in retrieval quality at the same model size, paid for in index footprint, rather than a universal win on every dataset. The same pair scores 57.22 against 56.20 on the full 15-dataset BEIR, a comparable gap, so the margin is not an artifact of the small benchmark.
Alongside NanoBEIR, MultiVectorInformationRetrievalEvaluator, MultiVectorRerankingEvaluator, MultiVectorTripletEvaluator, and MultiVectorDistillationEvaluator cover the usual evaluation setups on your own data. They're documented in the Evaluation API Reference.
Coming from PyLate or colpali-engine
MultiVectorEncoder absorbs the modeling, inference, training, and evaluation of both libraries. Every PyLate checkpoint loads directly, and Supported Models lists the colpali-engine checkpoints along with the revision to pass where one is still needed. If you're migrating, these are the calls that change:
| PyLate | Sentence Transformers |
|---|---|
pylate.models.ColBERT(model_name_or_path=...) | MultiVectorEncoder(...) |
model.encode(..., is_query=True) | model.encode_query(...) |
model.encode(..., is_query=False) | model.encode_document(...) |
pylate.scores.colbert_scores | model.similarity |
pylate.indexes.PLAID / pylate.retrieve.ColBERT | no equivalent, keep PyLate's PLAID or see Indexing |
| colpali-engine | Sentence Transformers |
|---|---|
ColQwen2.from_pretrained(...) + ColQwen2Processor | MultiVectorEncoder(...) |
processor.process_queries(...) + model(**batch) | model.encode_query(queries) |
processor.process_images(...) + model(**batch) | model.encode_document(images) |
processor.score_multi_vector(qs, ds) | model.similarity(query_embeddings, document_embeddings) |
mask_non_image_embeddings=True | MultiVectorMask(keep_only_token_ids=[...]) |
HierarchicalTokenPooler | HierarchicalTokenPooling |
colpali_engine.interpretability | sentence_transformers.multi_vector_encoder.interpretability |
One difference worth calling out: on a bare (non-ColBERT) checkpoint, PyLate's ColBERT("bert-base-uncased") applies the classic recipe by default, while MultiVectorEncoder("bert-base-uncased") builds a plain stack and leaves the prefixes, query expansion, and skiplist as explicit choices. The training loss and evaluator equivalents, and the data-handling differences, are in the Migration Guide.
Note that save compatibility is one-way in every case: PyLate, Stanford-NLP ColBERT, and colpali-engine checkpoints all load into MultiVectorEncoder, but MultiVectorEncoder.save_pretrained output isn't loadable by any of them.
Supported Models
Models carrying the multi-vector and sentence-transformers tags on the Hub are the list that stays current, and we're working to get those tags onto every model that works. The tables below are what we test against directly, so treat them as a starting point rather than the full set. For text retrieval in particular, any PyLate or Stanford-NLP ColBERT checkpoint loads whether or not it carries the tag yet.
Some entries need a small Sentence Transformers configuration added to their repository first, and several of those are still open pull requests at the time of writing. Where a revision is listed below, pass it until that pull request is merged, after which the plain model name is enough:
model = MultiVectorEncoder("vidore/colqwen-omni-v0.1", revision="refs/pr/N")
Text Retrieval Models
These load with their trained prefix tokens, query expansion, and punctuation skiplist recovered from the saved configuration.
The NanoBEIR column reports the mean NDCG@10 (higher is better) across the 13 NanoBEIR datasets, each a 50-query subsample of a BEIR dataset, as a fast proxy for English text retrieval quality. We used the MultiVectorNanoBEIREvaluator to compute the scores for the primarily-English models. A - means the model was not evaluated on it. Note that NanoBEIR is a small benchmark, and its scores aren't a substitute for evaluating on your own data, which is always the right way to pick a model.
Visual Document Retrieval Models
ColPali-style models embed page images as documents and text as queries.
The NanoViDoRe column reports the mean NDCG@10 (higher is better) across NanoViDoRe v3, a compact visual document retrieval benchmark spanning 8 subsets (computer science, energy, finance in English and French, HR, industrial, pharmaceuticals, and physics). Like with NanoBEIR, NanoViDoRe is a small benchmark which shouldn't replace evaluation on your own data.
| Model | Parameters | Dimensionality | NanoViDoRe | Notes |
|---|---|---|---|---|
| webAI-Official/webAI-ColVec1.1-8b | 8.4B | 640 | 0.6580 | needs trust_remote_code=True |
| webAI-Official/webAI-ColVec1.1-4b | 4.5B | 640 | 0.6520 | needs trust_remote_code=True |
| tencent/EVIE-Preview-4.5B | 4.54B | 128 | 0.6405 | - |
| TomoroAI/tomoro-colqwen3-embed-8b | 8.8B | 320 | 0.6206 | needs trust_remote_code=True |
| TomoroAI/tomoro-colqwen3-embed-4b | 4.4B | 320 | 0.6019 | needs trust_remote_code=True |
| vidore/colqwen2.5-v0.2 | 3.8B | 128 | 0.5402 | - |
| vidore/colqwen2.5-v0.1 | 3.8B | 128 | 0.5395 | - |
| vidore/colqwen-omni-v0.1 | 4.4B | 128 | 0.5309 | - |
| vidore/colpali-v1.3 | 2.9B | 128 | 0.4802 | - |
| vidore/colpali-v1.3-hf | 2.9B | 128 | 0.4793 | - |
| vidore/colpali-v1.2 | 2.9B | 128 | 0.4691 | - |
| vidore/colqwen2-v1.0 | 2.2B | 128 | 0.4685 | - |
| vidore/colqwen2-v0.1 | 2.2B | 128 | 0.4526 | - |
| vidore/colpali | 2.9B | 128 | 0.4516 | - |
| vidore/colpali-v1.1 | 2.9B | 128 | 0.4314 | - |
| vidore/colsmolvlm-v0.1 | 2.1B | 128 | 0.4054 | - |
| vidore/colpali-hard-v1.1 | 2.9B | 128 | 0.3949 | - |
| vidore/colSmol-500M | 507M | 128 | 0.3459 | - |
| vidore/colSmol-256M | 256M | 128 | 0.2673 | - |
| ModernVBERT/colmodernvbert | 252M | 128 | 0.2632 | - |
| vidore/colpali-v1.2-hf | 2.9B | 128 | - | - |
| vidore/colqwen2-v1.0-hf | 2.2B | 128 | - | - |
Most of these are LoRA adapter repositories, with the adapter applied directly onto its base at load time. Some also have a -merged sibling on the Hub (e.g. vidore/colpali-v1.3-merged) with the adapter already folded into the weights.
The three -hf entries are the transformers-native *ForRetrieval ports. They load without any configuration, but use more modeling from transformers and less from sentence_transformers. Generally, it's preferable to use the original models instead, as the ports score approximately the same.
Acknowledgements
Late interaction in Sentence Transformers rests on a lot of earlier work. Thanks to Omar Khattab and Matei Zaharia for ColBERT, which everything here descends from, and to the LightOn team (Antoine Chaffin, Raphael Sourty, Paulo Moura, and Amélie Chatelain) for PyLate and fast-plaid, which carried late interaction for years and shaped a good deal of the API described above.
Thanks to the ColPali team (Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo) for ColPali and colpali-engine, which brought late interaction to page images, and to Benjamin Clavié, Antoine Chaffin, and Griffin Adams for token pooling.
Thanks as well to the core MTEB team, Kenneth Enevoldsen and Roman Solomatin among many others, for MTEB and for the kind of hidden work that keeps information retrieval research running.
And thanks to everyone who trained and released the checkpoints in Supported Models. Without them this post would have had nothing to measure.
Additional Resources
Documentation
- Multi-Vector Encoder > Usage
- Multi-Vector Encoder > Pretrained Models
- Multi-Vector Encoder > Creating Custom Models
- Multi-Vector Encoder > Speeding up Inference
- Multi-Vector Encoder > API Reference
- Installation
- Migration Guide
Example Scripts
- Semantic Search
- Retrieve and Rerank
- Token Pooling
- ColPali Heatmaps
- Text Similarity Maps
- NanoBEIR Evaluation
Training
To learn how to train or finetune these models on your own data:
- Multi-Vector Encoder > Training Overview
- Multi-Vector Encoder > Loss Overview
- Multi-Vector Encoder > Training Examples
- LateOn and mLateOn training scripts: LightOn's PyLate recipes for LateOn, mLateOn, DenseOn, and mDenseOn, where the finetuning scripts show practical details like splitting a 16,384-example batch into mini-batches of 16.
Hugging Face Hub
Companion Blogposts
- Training and Finetuning Embedding Models with Sentence Transformers: the general training guide for text-only dense embedding models.
- Training and Finetuning Reranker Models with Sentence Transformers: Cross Encoder training, the other way to add a precise second stage.
- Training and Finetuning Sparse Embedding Models with Sentence Transformers: SPLADE and other sparse encoders, which combine well with late interaction in hybrid search.
- Multimodal Embedding & Reranker Models with Sentence Transformers: single-vector multimodal models, the dense counterpart to ColPali-style retrieval.
- Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: includes a Visual Document Retrieval walkthrough with single-vector models.
- 🪆 Introduction to Matryoshka Embedding Models: shrink dense embeddings by dimension, the way token pooling shrinks multi-vector ones by count.