TL;DR — Falcon Perception 是一个 0.6B 参数的早期融合 Transformer 架构，用于从自然语言提示词进行开放词汇定位和分割。该模型使用混合注意力掩码，将图像块和文本作为一个序列进行处理，并通过一个紧凑的结构化 token 接口和轻量级输出头生成可变数量的实例。在 SA-Co 上，Falcon Perception 达到了 68.0 的 Macro-F1（而 SAM 3 为 62.3），主要差距在于存在性校准（MCC 0.64 对比 0.82）。我们还推出了 PBench，这是一个诊断性基准，按能力（属性、OCR 引导的消歧、空间约束、关系）以及密集长上下文拥挤场景来分解性能。我们还发布了 Falcon OCR，这是一个 0.3B 参数的模型，在 olmOCR 基准和 OmniDocBench 上分别达到了 80.3 和 88.6 的分数，同时拥有所有开源 OCR 模型中最高的吞吐量。
这篇文章是一份简短、实用的总结，介绍了我们构建了什么、为什么这样构建，以及我们在此过程中学到了什么。
问题：为什么感知系统最终都变成了流水线？
许多开放词汇感知系统被构建为模块化流水线：一个（通常是冻结的）视觉骨干网络提取特征，一个独立的融合/解码器阶段将这些特征与语言结合，再由额外的组件处理匹配和后处理。这类设计在许多场景下表现良好，但也伴随着权衡：它可能难以干净地扩展，难以将改进归因于正确的组件，并且随着我们为每种失败模式添加新的修复，容易积累复杂性。
我们提出了一个更简单的问题：如果选择了正确的注意力模式、输出接口和训练信号，一个单一的早期融合 Transformer 骨干网络能否同时处理感知和语言建模？
在我们的实验中，答案基本上是肯定的。本文的其余部分将描述主要的设计选择及其背后的证据。
架构：早期融合、混合注意力与高效密集接口
一个自回归 Transformer 架构处理由图像块、文本和任务 token 组成的统一序列。模型按固定顺序预测物体属性：<coord> → <size> → <seg>。边界框坐标和尺寸通过专用解码头解码，并作为傅里叶特征重新注入。高分辨率分割掩码通过 <seg> token 与上采样图像特征的点积生成。
单一主干，两种行为
Falcon Perception 的核心是一个密集 Transformer，从第一层起就在共享参数空间中处理图像块和文本 token。我们没有采用独立的视觉主干加后期融合解码器的方案，而是保留单一主干，并依靠掩码机制和轻量级输出接口，使密集预测问题变得可处理。
图像和文本具有不同的结构：像素是二维的，受益于双向上下文，而预测接口本质上是序列化的。我们通过混合注意力掩码来解决这一问题：
- 图像 token 双向关注所有其他图像 token，构建全局视觉上下文（类似于视觉编码器的功能）。
- 文本和任务 token 以因果方式关注它们之前的所有内容——即完整的视觉前缀加上前面的文本。
这使得同一主干在图像 token 上能像双向视觉编码器一样工作，同时仍支持对任务 token 进行自回归预测。
感知链：面向密集输出的由粗到细监督
密集感知并非固定尺寸的预测问题：一张图像可能包含零个或数百个实例。自回归生成提供了清晰的变长接口，但完全自回归的密集生成（例如逐 token 生成多边形或高分辨率掩码）很快就会变得代价高昂。
我们使用一个紧凑的结构化接口——感知链，它将每个实例分解为三个步骤：
<coord> → <size> → <seg>
- 坐标 token：模型首先预测实例的中心——确定它正在谈论哪个物体。
- 尺寸 token：然后预测空间范围——确定它有多大。
- 分割 token：最后生成一个单一嵌入，该嵌入与上采样图像特征进行点积运算后，即可生成全分辨率二值掩码。
这种排序是经过深思熟虑的。先确定几何信息可以减少歧义（“哪个实例？”），并使掩码预测步骤更接近于在已解析对象条件下的像素级细化。
专用头部，最小开销
主干网络是共享的，而解码则使用针对输出类型量身定制的轻量级头部：
坐标与尺寸头部使用傅里叶特征编码：通过随机高斯投影将连续坐标映射到高维正弦空间。这克服了神经网络的频谱偏差，比单纯的离散分箱法实现了更精确的定位。解码后的坐标会重新注入序列中，作为后续 token 的条件信息。
分割头部计算 `<seg>` token 的隐藏状态与内容感知上采样图像特征之间的点积。由于 `<seg>` token 是在几何信息之后生成的，并且能够访问早期融合的视觉上下文，因此我们可以避免基于解码器的实例分割训练中常见的独立掩码查询机制和匈牙利匹配算法。
PBench：一个旨在隔离缺失能力的基准测试
现有的指代表达基准测试（如 RefCOCO）已经饱和——模型通常能达到 90% 以上的准确率——并且它们混淆了问题的根源。模型失败是因为无法读取文本？无法理解空间关系？还是无法处理拥挤场景？
我们引入了 PBench，这是一个诊断性基准测试，它根据所需的主要能力对样本进行分离：
| 等级 | 能力 | 示例提示词 |
|---|---|---|
| L0 | 简单物体 | “汽车” |
| L1 | 属性与子类型 | “红色汽车”、“破损的栅栏” |
| L2 | OCR 引导识别 | “健怡可乐瓶”、“耐克鞋” |
| L3 | 空间理解 | “左边的汽车”、“左边数第三个窗户” |
| L4 | 关系与交互 | “打伞的人”、“最高的建筑” |
| 密集 | 拥挤度压力测试 | 每张图像数百个实例 |
每个样本针对一种主要能力：OCR 提示词避免空间限定词，空间提示词避免图像内文本消歧器。这产生的是能力画像，而非单一的模糊分数，并且更容易决定下一步的投资方向（数据、训练课程或后训练）。
训练：知识蒸馏、大规模数据与三阶段方案
多教师知识蒸馏
猎鹰感知模型并非从随机权重开始训练（在我们的消融实验中，随机初始化对分割任务不稳定），而是通过多教师知识蒸馏进行初始化。两位强大的视觉教师模型贡献了互补的信号：
- DINOv3（ViT-H）：对分割至关重要的强局部特征
- SigLIP2：用于开放词汇理解的语言对齐特征
经过蒸馏初始化后，模型在 ImageNet-1k 上达到了 74.25% 的零样本准确率，在 Pascal VOC 上线性探测 mIoU 达到 85.11%，为后续感知专用训练奠定了坚实的视觉基础。
数据：5400 万张图像，1.95 亿条正向描述，4.88 亿条困难负样本
我们通过一个多阶段流水线构建训练集：
- 利用 DINOv3 嵌入对网络爬取的图像进行层次聚类，以确保概念覆盖的均匀性。
- 基于 VLM 的列表生成功能为每张图像生成密集的对象描述，并按 PBench 复杂度级别分类（60% 基础级，40% 高级）。
- 负样本挖掘生成语义、视觉和细粒度的困难负样本，以对抗模型幻觉。
- 集成共识——SAM 3、Qwen3-VL-30B 和 Moondream3 必须达成一致（IoU > 0.8）才能自动接受。
- 人工验证——存在分歧的样本交由标注员处理，以恢复那些令自动化系统困惑的困难样本。
我们严格保持正负样本 1:1 的比例。这使得存在性校准成为一个首要目标：模型应当可靠地判断“不存在”，而不仅仅是在有把握时才绘制掩码。
三个阶段（总计 700 GT）
第一阶段——上下文列表生成（450 GT）：模型学习自回归地列举场景清单——预测文本描述及其位置。查询之间采用完全因果注意力机制，使模型能够学习对象的共现关系（“叉子，然后是刀子，然后是盘子”）。这构建了广泛的场景理解能力。
第二阶段——任务对齐（225 GT）：修改注意力掩码，使查询之间不再相互可见，模拟推理时的独立查询。文本 token 的损失被屏蔽，将梯度信号完全聚焦于存在性分类和定位。这一阶段从“场景理解”过渡到“回答这个具体问题”。
阶段三——长上下文微调（10 GT）：这是一个短阶段，将每个表达式的掩码上限提升至 600，并使用极小的恒定学习率。这使得模型能够适应极端人群密度场景，同时不会遗忘先前的能力。
通过消融实验验证的关键设计选择：
- 针对专用头使用 Muon 优化器（相对于 AdamW）——在 SA-Co 检测上带来 +4.8 个百分点的提升
- 实例的栅格排序（相对于随机/按大小排序）——在 SA-Co 上相比随机排序提升 +10 个百分点
- 语法特征正则化——防止从蒸馏特征中发生漂移，将分割性能提升 +1.5 个百分点
- 跨秩的全局损失归一化——纠正 FSDP 中因变长打包序列导致的偏差
结果
SA-Co：同类最佳掩码质量
在 SA-Co 开放词汇分割基准上，Falcon Perception（0.6B 参数）达到了 68.0 的 Macro-F1 分数，而 SAM 3 为 62.3，在属性密集（+8.2）、食品与饮料（+12.2）以及运动器材（+4.0）等细分领域取得了大幅提升。与此同时，Falcon Perception 在存在性校准方面落后于 SAM 3（MCC：0.64 对比 0.82），这是目前最明确的改进方向。
以下是一个输出示例——提示词“Falcon”生成了精确的实例掩码：
Falcon Perception 在指代表达方面也表现良好，能够正确分割视频每一帧中带有黑色面包的汉堡：
PBench：随提示词复杂度扩展
这正是早期融合设计展现出最大差异的地方：
| 能力 | SAM 3 | Falcon Perception | 差距 |
|---|---|---|---|
| L0：简单物体 | 64.3 | 65.1 | +0.8 |
| L1：属性 | 54.4 | 63.6 | +9.2 |
| L2：OCR 引导 | 24.6 | 38.0 | +13.4 |
| L3：空间 | 31.6 | 53.5 | +21.9 |
| L4：关系 | 33.3 | 49.1 | +15.8 |
| 密集 | 58.4 | 72.6 | +14.2 |
在简单物体上，差距不大。随着提示词变得更具组合性——需要 OCR 引导的消歧、空间约束或关系绑定——差距逐渐扩大。
在我们的 PBench 密集子集中，Falcon Perception（0.6B）大幅超越了通用 VLM 基线（例如，在我们的评估设置中，72.6 对比 Qwen3-VL-30B 的 8.9），并在空间和关系层级上达到或超越了 8B 模型。
定性结果：OCR、空间、关系和密集场景
随着提示词变得越来越组合化——需要OCR引导的消歧、空间约束、关系绑定，或扩展到数百个实例——早期融合的优势在视觉上变得清晰可见：
- OCR引导的定位（第2级）：当区分信号是物体上写的文字时，Falcon Perception能正确读取，而SAM 3无法区分。
- 空间理解（第3级）：当提示词指定空间关系时，Falcon Perception会形成连贯的二维场景地图。
- 关系推理（第4级）：当目标通过交互而非外观来定义时，Falcon Perception能理解场景图。
- 密集场景：扩展到数百个实例：自回归接口在场景极其拥挤时特别有用，而固定查询解码器可能会遇到实际限制。
第2级——OCR引导的定位：Falcon Perception读取物体上的文字来进行消歧；SAM 3无法做到。
“168葡萄酒瓶”：Falcon Perception识别出标有“168”的瓶子，而SAM 3高亮了所有瓶子。“檀香山方向指示牌”：Falcon读取文字以找到正确的指示牌。
第3级——空间理解：Falcon Perception能解析空间约束；SAM 3会产生误报。
“左侧烤架上的下方肉串”、“底部红色汽车右边的黑色汽车”、“左侧的比利时国旗”——Falcon Perception根据空间约束解析出正确的实例。SAM 3对多个候选对象预测了误报。
第4级——关系推理：Falcon Perception能理解交互关系；SAM 3忽略了关系约束。
“棕色圆面包旁边的糕点”、“正在使用手机的人”、“手里拿着头盔的人”——Falcon Perception识别出正在交互的实例。SAM 3高亮了该物体类别的所有实例，忽略了关系约束。
密集场景：Falcon Perception可扩展到数百个实例；SAM 3的解码器会耗尽查询token。
“雪雁”、“鸽子”、“彩色罐装饮料”——Falcon Perception以自回归方式分割数百个实例。SAM 3的固定大小解码器在超过约200个实例时会耗尽查询token。
Falcon OCR：将早期融合扩展到文档理解领域
现代 OCR 早已超越从清晰扫描件中提取文本的阶段。如今的系统必须一次性处理多栏布局、数学公式、表格、图表以及多语言内容。大多数有竞争力的 OCR 视觉语言模型都采用一种熟悉的方案：一个视觉编码器连接一个独立的文本解码器，再加上特定任务的粘合层。这些系统虽然有效，但往往体积庞大（10 亿到 30 亿以上参数）。
我们走了一条不同的路：复用 Falcon Perception 中相同的早期融合密集 Transformer 架构，但专门为 OCR 从头训练一个更小的 0.3B 参数变体。由此诞生了 Falcon OCR——一个单一骨干网络，在共享参数空间内，使用相同的混合注意力掩码（图像 token 用双向掩码，文本 token 用因果掩码）处理图像块和文本 token，并通过提示词而非额外模块来切换任务。
我们从头训练（没有使用多教师知识蒸馏），因为 OCR 所需的视觉特征——细粒度字形识别、笔画级区分——与分割任务所需的物体级特征有本质区别。从头开始训练让骨干网络能够从零开始构建针对文本优化的表征。
训练
我们在一个精选的英文混合数据集上进行训练，涵盖三个核心任务：通用文档文本解析（数字 PDF、老旧扫描件、打字机文档）、数学与科学公式识别，以及表格结构识别。该混合数据集还包括手写文本、真实场景文字，以及由渲染后的 LaTeX 和 HTML 源生成的合成样本。训练目标是对结构化文本输出进行纯粹的下一 token 预测。
训练分两个阶段进行：首先是在恒定学习率下的长时间预训练阶段，模型在此阶段学习所有元素类型的核心 OCR 能力；随后是短暂的余弦衰减微调阶段，学习率在此阶段退火至接近零。
基准测试结果
我们在 olmOCR（跨多种输入的二元正确性检查）和 OmniDocBench（整页解析的连续指标）上进行了评估。所有对比模型都显著更大，和/或使用了专有基础设施。Falcon OCR 在仅有 0.3B 参数的情况下，在 olmOCR 上达到了 80.3% 的成绩，与排名第一的系统仅差 1.7 个百分点，并在多列（87.1%）和表格（90.3%）方面领先所有模型。在 OmniDocBench 上，它获得了 88.64 的总分，领先于 DeepSeek OCR v2、GPT 5.2 和 Mistral OCR 3。
服务吞吐量
在 0.3B 参数下，Falcon OCR 比 0.9B 级别的 OCR 视觉语言模型大约小 3 倍，这直接转化为更高的服务吞吐量。在单张 A100-80GB 上使用 vLLM 在高并发下测量：
| 模式 | token/秒 | 图片/秒 | 描述 |
|---|---|---|---|
| 布局 + OCR | 5,825 | 2.9 | 完整流程：布局检测 → 裁剪 → 按区域 OCR |
紧凑的模型体积和 vLLM 集成（连续批处理、PagedAttention、优化的 CUDA 内核）使其非常适合需要处理数百万页的大规模文档数字化场景。
我们在结果中看到了什么
更广泛地说，这些结果表明，早期融合的单栈 Transformer 是“视觉编码器加文本解码器”这一 OCR 方案的一个可行替代方案。一个主干网络、共享参数空间、一个解码接口，以及更好的数据和训练信号，而不是日益复杂的流水线。我们希望这能鼓励更多朝着这个方向的研究工作。
定性示例
Falcon OCR 处理在具有挑战性的真实世界条件下捕获的图像，这些图像具有不同的光照、多样的文本语义（数学公式、结构化表格、手写笔记）和复杂的文档布局，以生成结构化的文本输出。
点击下方每个类别以展开。
手写和真实世界图像：对手写文本以及在不利条件下野外捕获的图像进行准确的转录。
Falcon OCR 从手写文档和真实世界照片中提取文本，这些照片具有可变的光照、方向和内容复杂度。
表格提取：跨多种格式忠实再现表格结构和单元格内容。
Falcon OCR 能够准确再现来自不同格式和复杂度的表格中的单元格条目和结构布局。
数学公式：对不同符号复杂程度的方程进行精确识别。
Falcon OCR 能够正确转录从简单方程到包含嵌套运算符的多行推导等各类数学表达式。
复杂文档布局：从多栏、混合内容文档中忠实提取文本。
Falcon OCR 在从具有多栏布局、图片和脚注的文档中提取文本时，能够保留阅读顺序和结构保真度。
推理：快速、实用且开源
本次发布包含一个基于 PyTorch 的 FlexAttention 构建的推理栈，这使得表达自定义注意力模式以及高效处理打包的可变长度序列变得切实可行。
分页推理引擎
- 带有虚拟页表的分页 KV 缓存（无填充导致的内存浪费）
- 连续批处理：新序列在生成中途进入，已完成序列立即释放页面
- 针对解码循环的 CUDA 图捕获
- 后台分词与 GPU 计算重叠进行
- 高分辨率特征缓存：使用固定内存缓冲区的 LRU 缓存，用于上采样图像特征的异步 GPU-CPU 传输——对同一图像的后续查询将跳过昂贵的上采样步骤
在我们基于 H100 的设置中，典型延迟约为：预填充 ~100ms，上采样 ~200ms（若已缓存则为 0ms），以及针对少量实例的解码 ~50ms。（这些数字取决于分辨率、序列长度和预测实例的数量。）
Falcon-OCR 的 Docker 和 MLX 集成
对于 Falcon-OCR 模型，我们还提供了用于快速部署的 vLLM Docker 服务器，以及用于 Apple Silicon 的 MLX 集成。
详情请查看 GitHub 仓库。
更广阔的图景：感知领域的“苦涩教训”
Falcon Perception 的设计有意保持极简：一个主干网络、一组目标函数，以及仅在输出为连续且密集时才使用的小型头部。其基本假设是，大部分性能提升应来自数据、算力和训练信号，而非不断用专用模块扩展流水线。
该架构并未阻塞任何明显的扩展路径：增加更多图像和更难的提示词以提升基础能力，混入纯文本数据以改善语言能力，延长上下文窗口以处理更密集的场景。它本质上仍然只是一个序列模型。
Falcon Perception 由阿联酋阿布扎比技术创新研究所（TII）的 Falcon Vision 团队开发。
引用
如果您使用了 Falcon-Perception，请引用
@article{bevli2026falcon,
title = {Falcon Perception},
author = {Bevli, Aviraj and Chaybouti, Sofian and Dahou, Yasser and Hacid, Hakim and Huynh, Ngoc Dung and Le Khac, Phuc H. and Narayan, Sanath and Para, Wamiq Reyaz and Singh, Ankit},
journal = {arXiv preprint arXiv:2603.27365},
year = {2026},
url = {https://arxiv.org/abs/2603.27365}
}
TL;DR — Falcon Perception is a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural language prompts. The model processes image patches + text in one sequence using a hybrid attention mask, and produces variable numbers of instances with a small, structured token interface and lightweight output heads. On SA-Co, Falcon Perception reaches 68.0 Macro-F1 (vs. 62.3 for SAM 3) with the main remaining gap being presence calibration (MCC 0.64 vs. 0.82). We also introduce PBench, a diagnostic benchmark that breaks down performance by capability (attributes, OCR-guided disambiguation, spatial constraints, relations) and by dense long-context crowded scenes. We also relase Falcon OCR, a 0.3B-parameter model which reaches a score of 80.3 and 88.6 on the olmOCR benchmark and OmniDocBench respectively, while having the highest throughput of any open source OCR model.
This post is a brief, practical write-up of what we built, why we built it this way, and what we learned along the way.
The problem: why do perception systems end up as pipelines?
Many open-vocabulary perception systems are built as modular pipelines: a (often frozen) vision backbone extracts features, a separate fusion/decoder stage combines them with language, and additional components handle matching and post-processing. This family of designs works well in many settings, but it comes with trade-offs: it can be hard to scale cleanly, hard to attribute improvements to the right component, and easy to accumulate complexity as we add a new fix for each failure mode.
We asked a simpler question: can a single early-fusion Transformer backbone handle both perception and language modeling, if we choose the right attention pattern, output interface, and training signal?
In our experiments, the answer is largely yes. The rest of this post describes the main design choices and the evidence behind them.
The architecture: early fusion, hybrid attention, and an efficient dense interface
A single autoregressive Transformer processes a unified sequence of image patches, text, and task tokens. The model predicts object properties in a fixed order: <coord> → <size> → <seg>. Bounding box coordinates and sizes are decoded via specialized heads and re-injected as Fourier features. High-resolution segmentation masks are generated by a dot product between the <seg> token and upsampled image features.
One Backbone, Two Behaviors
At its core, Falcon Perception is a dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer. Instead of a separate vision backbone followed by a late-fusion decoder, we keep a single backbone and rely on masking and a lightweight output interface to make the dense prediction problem tractable.
Images and text have different structure: pixels are 2D and benefit from bidirectional context, while the prediction interface is naturally sequential. We address this with a hybrid attention mask:
- Image tokens attend to all other image tokens bidirectionally, building a global visual context (like a vision encoder would).
- Text and task tokens attend causally to everything before them — the full visual prefix plus preceding text.
This allows the same backbone to behave like a bidirectional visual encoder on image tokens, while still supporting autoregressive prediction over task tokens.
Chain-of-Perception: coarse-to-fine supervision for dense outputs
Dense perception is not a fixed-size prediction problem: an image may contain zero instances or hundreds. Autoregressive generation gives a clean variable-length interface, but fully autoregressive dense generation (e.g., polygons or high-resolution masks token-by-token) quickly becomes expensive.
We use a small structured interface, Chain-of-Perception, which decomposes each instance into three steps:
<coord> → <size> → <seg>
- Coordinate token: The model first predicts the center of the instance — resolving which object it's talking about.
- Size token: Then the spatial extent — resolving how big it is.
- Segmentation token: Finally, a single embedding that, when dot-producted with upsampled image features, produces a full-resolution binary mask.
This ordering is deliberate. Committing to geometry first reduces ambiguity (“which instance?”), and makes the mask prediction step closer to pixel refinement conditioned on the resolved object.
Specialized Heads, Minimal Overhead
The backbone is shared, while decoding uses lightweight heads tailored to the output type:
Coordinate & Size Heads use Fourier feature encoding : mapping continuous coordinates through a random Gaussian projection into a high-dimensional sinusoidal space. This overcomes the spectral bias of neural networks, yielding more precise localization than discrete binning alone. Decoded coordinates are re-injected into the sequence as conditioning for subsequent tokens.
Segmentation Head computes a dot product between the
<seg>token’s hidden state and content-aware upsampled image features. Because the<seg>token is produced after geometry and has access to early-fused visual context, we can avoid the separate mask-query machinery and Hungarian matching that often appears in decoder-based instance segmentation training.
PBench: a benchmark designed to isolate what is missing
Existing referring-expression benchmarks like RefCOCO are saturated — models routinely hit 90%+ — and they conflate what went wrong. Did the model fail because it can't read text? Can't understand spatial relationships? Can't handle a crowd?
We introduce PBench, a diagnostic benchmark that separates samples by the dominant capability required:
| Level | Capability | Example Prompt |
|---|---|---|
| L0 | Simple objects | "car" |
| L1 | Attributes & subtypes | "red car", "broken fence" |
| L2 | OCR-guided identification | "Diet Coke bottle", "Nike shoes" |
| L3 | Spatial understanding | "car on the left", "third window from left" |
| L4 | Relations & interactions | "person holding umbrella", "tallest building" |
| Dense | Crowdedness stress test | Hundreds of instances per image |
Each sample targets one dominant capability: OCR prompts avoid spatial qualifiers, and spatial prompts avoid in-image text disambiguators. This yields a capability profile rather than a single opaque score, and makes it easier to decide where to invest next (data, training curriculum, or post-training).
Training: distillation, large-scale data, and a three-stage recipe
Multi-Teacher Distillation
Rather than training from random weights (which in our ablations was unstable for segmentation), Falcon Perception initializes via multi-teacher distillation. Two strong vision teachers contribute complementary signals:
- DINOv3 (ViT-H): strong local features critical for segmentation
- SigLIP2: language-aligned features for open-vocabulary understanding
The distilled initialization achieves 74.25% zero-shot accuracy on ImageNet-1k and 85.11% linear-probe mIoU on Pascal VOC, providing a strong visual foundation before perception-specific training.
Data: 54M Images, 195M Positive Expressions, 488M Hard Negatives
We build the training set through a multi-stage pipeline:
- Hierarchical clustering of web-scraped images via DINOv3 embeddings to ensure uniform concept coverage.
- VLM-driven listing generates dense object descriptions per image, categorized by PBench complexity level (60% basic, 40% advanced).
- Negative mining produces semantic, visual, and fine-grained hard negatives to combat hallucination.
- Ensemble consensus — SAM 3, Qwen3-VL-30B, and Moondream3 must agree (IoU > 0.8) for automatic acceptance.
- Human verification — disagreements go to annotators, recovering hard samples that confuse automated systems.
We maintain a strict 1:1 ratio of positive to negative samples. This makes presence calibration a first-class target: the model should reliably say “absent,” not only draw masks when confident.
The Three Stages (700 GT Total)
Stage 1 — In-Context Listing (450 GT): The model learns to autoregressively list scene inventories — predicting text expressions and their locations. Full causal attention between queries enables learning of object co-occurrence ("fork, then knife, then plate"). This builds broad scene understanding.
Stage 2 — Task Alignment (225 GT): The attention mask is modified so queries can no longer see each other, simulating independent queries at inference time. Loss on text tokens is masked, focusing gradient signal entirely on presence classification and localization. This stage transitions from "scene understanding" to "answer this specific question."
Stage 3 — Long-Context Finetuning (10 GT): A short phase with the mask limit raised to 600 per expression and a minimal constant learning rate. This adapts the model for extreme crowd density without forgetting earlier capabilities.
Key design choices validated through ablations:
- Muon optimizer for the specialized heads (vs. AdamW) — yields +4.8 points on SA-Co detection
- Raster ordering of instances (vs. random/size) — +10 points over random ordering on SA-Co
- Gram feature regularization — prevents drift from the distillation features, improving segmentation by +1.5 points
- Global loss normalization across ranks — corrects bias from variable-length packed sequences in FSDP
Results
SA-Co: Best-in-Class Mask Quality
On the SA-Co open-vocabulary segmentation benchmark, Falcon Perception (0.6B parameters) achieves 68.0 Macro-F1, compared to 62.3 for SAM 3, with large gains on attribute-heavy (+8.2), food & drink (+12.2), and sports equipment (+4.0) splits. At the same time, Falcon Perception lags SAM 3 on presence calibration (MCC: 0.64 vs 0.82), which is the clearest remaining improvement axis.
Here's an example output — the prompt "Falcon" produces precise instance masks:
Falcon Perception also performs well for reffering expressions, able to correctly segment the burger with a black bun in each frame of the video:
PBench: Scaling with Prompt Complexity
This is where the early-fusion design shows the largest differences:
| Capability | SAM 3 | Falcon Perception | Gap |
|---|---|---|---|
| L0: Simple objects | 64.3 | 65.1 | +0.8 |
| L1: Attributes | 54.4 | 63.6 | +9.2 |
| L2: OCR-guided | 24.6 | 38.0 | +13.4 |
| L3: Spatial | 31.6 | 53.5 | +21.9 |
| L4: Relations | 33.3 | 49.1 | +15.8 |
| Dense | 58.4 | 72.6 | +14.2 |
On simple objects, the gap is modest. As prompts become more compositional—requiring OCR-guided disambiguation, spatial constraints, or relational binding—the gap widens.
In our PBench Dense split, Falcon Perception (0.6B) substantially outperforms generalist VLM baselines (e.g., 72.6 vs 8.9 for Qwen3-VL-30B in our evaluation setup), and matches or exceeds the 8B model on spatial and relational tiers.
Qualitative Results: OCR, Spatial, Relational, and Dense
As prompts grow more compositional — requiring OCR-guided disambiguation, spatial constraints, relational binding, or scaling to hundreds of instances — the early-fusion advantage becomes visually clear:
- OCR-Guided Grounding (Level 2): When the distinguishing signal is text written on an object, Falcon Perception reads it correctly while SAM 3 cannot differentiate.
- Spatial Understanding (Level 3): When prompts specify spatial relationships, Falcon Perception forms a coherent 2D scene map.
- Relational Reasoning (Level 4): When the target is defined through interactions rather than appearance, Falcon Perception understands the scene graph.
- Dense Scenes: Scaling to Hundreds of Instances: The autoregressive interface is particularly useful when scenes are extremely crowded, where fixed-query decoders can run into practical limits.
Level 2 — OCR-Guided Grounding: Falcon Perception reads text on objects to disambiguate; SAM 3 cannot. 
"168 wine bottles": Falcon Perception identifies the bottles labeled "168", while SAM 3 highlights every bottle. "Honolulu direction sign": Falcon reads the text to find the right sign.
Level 3 — Spatial Understanding: Falcon Perception resolves spatial constraints; SAM 3 returns false positives. 
"Lower meat skewer on left grill," "black car to the right of red car at bottom," "Belgian flag on the left" — Falcon Perception resolves the correct instance from spatial constraints. SAM 3 predicts false positives for multiple candidates.
Level 4 — Relational Reasoning: Falcon Perception understands interactions; SAM 3 ignores relational constraints. 
"Pastry next to brown round bread," "person using phone," "person holding helmet in hand" — Falcon Perception identifies the interacting instance. SAM 3 highlights all instances of the object class, ignoring the relational constraint.
Dense Scenes: Falcon Perception scales to hundreds of instances; SAM 3's decoder runs out of query tokens. 
"Snow goose," "pigeon," "colorful canned drinks" — Falcon Perception autoregressively segments hundreds of instances. SAM 3's fixed-size decoder runs out of query tokens beyond ~200 instances.
Falcon OCR: extending early fusion to document understanding
Modern OCR has moved well beyond extracting text from clean scans. Today's systems must handle multi-column layouts, mathematical formulas, tables, charts, and multilingual content — all in one pass. Most competitive OCR VLMs tackle this with a familiar recipe: a vision encoder feeding a separate text decoder, plus task-specific glue. These systems work, but they tend to be large (1B–3B+ parameters).
We took a different path: reuse the same early-fusion dense Transformer from Falcon Perception, but train a smaller 0.3B-parameter variant from scratch specifically for OCR. The result is Falcon OCR — a single backbone that processes image patches and text tokens in a shared parameter space with the same hybrid attention mask (bidirectional for image tokens, causal for text tokens), and switches tasks through prompts rather than additional modules.
We trained from scratch (no multi-teacher distillation) because the visual features OCR needs — fine-grained glyph recognition, stroke-level discrimination — differ substantially from the object-level features useful for segmentation. Starting fresh lets the backbone develop text-optimized representations from the ground up.
Training
We train on a curated English-language mixture spanning three core tasks: general document text parsing (digital PDFs, old scans, typewritten documents), mathematical and scientific formula recognition, and table structure recognition. The mixture also includes handwriting, real-world scene text, and synthetic samples generated from rendered LaTeX and HTML sources. The training objective is pure next-token prediction on structured text outputs.
Training proceeds in two phases: a long pre-training phase at constant learning rate where the model learns core OCR capabilities across all element types, followed by a short cosine-decay finetuning phase where the learning rate is annealed to near zero.
Benchmark results
We evaluate on olmOCR (binary correctness checks across diverse inputs) and OmniDocBench (continuous metrics over full-page parses). All comparison models are significantly larger and/or use proprietary infrastructure. At 80.3% on olmOCR with only 0.3B parameters, Falcon OCR is within 1.7 points of the top system and leads all models on Multi-Column (87.1%) and Tables (90.3%). On OmniDocBench it scores 88.64 overall, ahead of DeepSeek OCR v2, GPT 5.2, and Mistral OCR 3.
Serving throughput
At 0.3B parameters, Falcon OCR is roughly 3x smaller than 0.9B-class OCR VLMs, which translates directly into higher serving throughput. Measured on a single A100-80GB with vLLM at high concurrency:
| Mode | tok/s | img/s | Description |
|---|---|---|---|
| Layout + OCR | 5,825 | 2.9 | Full pipeline: layout detection → crop → per-region OCR |
The compact footprint and vLLM integration (continuous batching, PagedAttention, optimized CUDA kernels) make it practical for large-scale document digitization where millions of pages need processing.
What we see in the results
More broadly, these results suggest that the early-fusion single-stack Transformer is a viable alternative to the "vision encoder plus text decoder" recipe for OCR. One backbone, shared parameter space, one decoding interface, and better data and training signals rather than increasingly complex pipelines. We hope this encourages more work in this direction.
Qualitative examples
Falcon OCR processes images captured under challenging real-world conditions with varying lighting, diverse text semantics (mathematical formulae, structured tables, handwritten notes), and complex document layouts, to produce structured text output.
Click each category below to expand.
Handwriting and Real-world Images: Accurate transcription of handwritten text and in-the-wild captures under adverse conditions. 
Falcon OCR extracts text from handwritten documents and real-world photographs with variable lighting, orientation, and content complexity.
Table Extraction: Faithful reproduction of tabular structure and cell content across diverse formats. 
Falcon OCR accurately reproduces cell entries and structural layout from tables of varying formats and complexity.
Mathematical Formulae: Accurate recognition of equations across varying levels of symbolic complexity. 
Falcon OCR correctly transcribes mathematical expressions ranging from simple equations to multi-line derivations with nested operators.
Complex Document Layouts: Faithful text extraction from multi-column, mixed-content documents. 
Falcon OCR preserves reading order and structural fidelity when extracting text from documents with multi-column layouts, figures, and footnotes.
Inference: Fast, Practical, and Open
The release includes an inference stack built on PyTorch’s FlexAttention, which makes it practical to express the custom attention patterns and efficiently serve packed variable-length sequences.
Paged Inference Engine
- Paged KV cache with virtual page tables (no wasted memory from padding)
- Continuous batching: new sequences enter mid-generation, finished ones release pages immediately
- CUDA graph capture for the decode loop
- Background tokenization overlapped with GPU compute
- HR feature cache: LRU cache with pinned-memory buffers for async GPU-CPU transfer of upsampled image features — subsequent queries on the same image skip the expensive upsampling step
In our setup on an H100, typical latencies are on the order of ~100ms prefill, ~200ms upsampling (0ms if cached), and ~50ms decode for a handful of instances. (These numbers depend on resolution, sequence length, and the number of predicted instances.)
Docker and MLX Integration for Falcon-OCR
For the Falcon-OCR model, we also provide a vLLM docker server for fast deployment and MLX integration for Apple-Silicon
Please check out the github repo for details.
The Bigger Picture: A "Bitter Lesson" for Perception
Falcon Perception is intentionally minimal: one backbone, one objective family, and small heads only where outputs are continuous and dense. The working assumption is that most gains should come from data, compute, and training signals, rather than continually expanding the pipeline with specialized modules.
The architecture doesn't block any obvious scaling path: add more images and harder prompts for better grounding, mix in text-only data for better language, increase context length for denser scenes. It's still just one sequence model.
Falcon Perception is developed by the Falcon Vision Team at the Technology Innovation Institute (TII), Abu Dhabi, UAE.
Citation
If you use Falcon-Perception, please cite
@article{bevli2026falcon,
title = {Falcon Perception},
author = {Bevli, Aviraj and Chaybouti, Sofian and Dahou, Yasser and Hacid, Hakim and Huynh, Ngoc Dung and Le Khac, Phuc H. and Narayan, Sanath and Para, Wamiq Reyaz and Singh, Ankit},
journal = {arXiv preprint arXiv:2603.27365},
year = {2026},
url = {https://arxiv.org/abs/2603.27365}
}