我们近期推出了 VAKRA，这是一个基于工具、可执行的基准测试，用于评估 AI 智能体在企业级环境中进行推理和行动的能力。
与测试单一技能的传统基准不同，VAKRA 衡量的是跨 API 和文档的组合推理能力，它利用完整的执行轨迹来评估智能体能否可靠地完成多步骤工作流。
VAKRA 提供了一个可执行环境，智能体在其中与超过 8000 个本地托管的 API 进行交互，这些 API 背后是覆盖 62 个领域的真实数据库，同时还配有与领域对齐的文档集合。任务可能需要 3 到 7 步的推理链，在自然语言工具使用约束下，将结构化 API 交互与非结构化检索结合起来。
如下所示，模型在 VAKRA 上的表现不佳——在这篇博客中，我们提供了关于 VAKRA 中任务的额外数据集细节，并分析了我们在不同任务上观察到的失败模式。
任务描述
如下所示，VAKRA 基准测试包含四个任务，每个任务测试不同的能力集。
图 1：VAKRA 基准测试中各项能力的代表性示例
能力 1：使用商业智能 API 进行 API 链式调用
该能力包含覆盖 54 个领域的 2077 个测试实例，需要使用来自 SLOT-BIRD 和 SEL-BIRD 集合中的工具（Elder 等人，2026 年）。与 Elder 等人的设置相比，SLOT-BIRD 和 SEL-BIRD 中的工具范围通过纳入更多领域而得到了扩展。每个领域仅限于一个工具集合，任务涉及链式调用 1 到 12 次工具以得出最终答案。
{
"query": "Which football team has a build-up play speed of 31, build-up plan dribbling of 53, and build-up play passing of 32?",
"tool_calls":[
{
"name": "get_data",
"arguments":{"tool_universe_id="486ea46224d1-aeb8037c5e78"},
"label": "retrieved_data_1"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"retrieved_data_1","key_name":"play_speed","value":31},
"label": "FILTERED_DF_0"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"FILTERED_DF_0","key_name":"play_dribble","value":53},
"label": "FILTERED_DF_1"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"FILTERED_DF_1","key_name":"play_passing","value":32},
"label": "FILTERED_DF_2"
},
{"name":{get_team_name},"arguments":{"data_label":"FILTERED_DF_2","n":1}}}],
"answer": "FC Barcelona"
}
图 2：来自 SEL-BIRD 集合的数据样本
如上所示，每个实例都有一个关联的 JSON 数据源，答案必须从中推导得出。支持此任务的 MCP 服务器包含一个名为 `get_data(tool_universe_id=id)` 的特殊工具，该工具必须在每个实例开始时调用。此工具会初始化数据源，返回数据的轻量级预览（见下方图 3），并将完整数据集存储在服务器端，以避免大量数据传输。这防止了通过 MCP 协议低效传输大数据。该调用还会根据 `tool_universe_id` 配置 MCP 服务器，以暴露相应的工具集，并将数据源与该实例的特定领域数据库对齐。
SLOT-BIRD 集合提供了一组共 7 个用于通用数据操作（例如过滤、排序）的全局工具，其灵感来源于 Tableau 和 Google Analytics 等系统。SEL-BIRD 集合在此基础上进行了扩展，引入了更专门的工具：其中一些工具与 SLOT-BIRD 共享，而另一些则是通过将分类参数扁平化为独立函数而衍生出来的（例如，带有参数 `ascending: bool = False` 的 `sort_data` 变成了 `sort_data_ascending` 和 `sort_data_descending`）。此外，SLOT-BIRD 中的通用 `retrieve_data` 函数被替换为特定于查询的 getter 函数。给定实例数据中的每个键都有一个关联的 `get` 函数（`get_KEY_NAME`），平均每个实例有 4 个 get 函数。
{
"handle": "retrieved_data_1",
"num_records": 2,
"key_details": [
{"name": "team_name", "dtype": "str", "first_3_values": ["FC Barcelona", "Manchester City"]},
{"name": "play_speed", "dtype": "int32", "first_3_values": [31, 40]},
{"name": "play_dribble", "dtype": "int32", "first_3_values": [53, 30]},
{"name": "play_passing", "dtype": "int32", "first_3_values": [32, 16]}
]}
图 3：从 `get_data` 函数获取的数据预览
能力 2：使用仪表盘 API 进行工具选择
此能力包含横跨 17 个领域的 1,597 个实例，需要使用来自扩展后的 REST-BIRD 集合（Elder 等人）中的工具。这些工具采用端点式接口，提供高度具体、与查询对齐的端点，封装了大部分计算逻辑。它们作为运行在 FastAPI 服务器中的 REST API 提供服务，该服务器由 MCP 服务器封装。此任务需要从特定领域的工具集中选择正确的 API（如图 1 示例所示）。每个领域包含最少 6 个、最多 328 个工具（平均 116 个工具）。与上一个任务类似，`get_data` 工具会配置 MCP 服务器，使其仅暴露相关的领域特定 API。
OpenAI API 规范将工具列表输入的最大长度限制为 128 个工具。这一限制要求使用该 API 的智能体构建者通过精简机制直接管理工具列表的长度。在我们仓库中的基线智能体中，一个简单的精简功能即可应对这一挑战。
能力 3：基于仪表盘 API 的多跳推理
基准测试的“能力 3”部分包含来自 38 个学科领域的 869 个测试实例。这些实例再次依赖 REST-BIRD API 集合，但在挑战中增加了多跳推理（参见图 1 中的示例）。多跳问题需要提取并组合多条支撑证据才能得出答案。该部分的实例需要 1 到 5 次逻辑跳转才能回答一个查询。测试数据集中查询的问题类型分布如下方图 4 所示。
图 4：能力 3（多跳）的 API 跳转类型分布，以及能力 4（多跳多源推理）的混合跳转类型分布
能力 4：多跳、多源推理与策略遵循
能力 4 包含来自 41 个领域的 644 个实例，同样基于 REST-BIRD API 集合构建。上方的图 4 展示了无策略测试查询的混合跳转分布。该部分包含最复杂的查询，具有以下特征：
多源：该部分为每个领域增加了文档索引。此能力中的查询可能需要同时从这些文档索引和 API 调用中获取信息。与能力 3 类似，该任务也包含多跳查询。所需的信息源在每一跳级别上应用，例如，一个问题可能涉及三次逻辑跳转，其来源依次为：API - RAG（文档检索）- API。为确保推理正确，在数据生成过程中对来源进行了去污染处理，即给定一跳所需的信息仅在一个来源中可用。例如，如果某一跳需要通过 API 来回答，则文档索引会通过移除可能包含回答问题所需信息的文档来构建。
多轮对话：数据集的这一部分还向场景中加入了多轮对话。每个实例都是一个包含多轮交互的对话。数据以上下文-回复对的形式发布，其中上下文编码了当前的对话历史，而智能体仅负责回答当前轮次的问题。
工具使用策略：这些实例中的一部分包含了智能体需要遵循的工具使用策略。这些策略以纯文本指令的形式呈现，规定了智能体允许访问的知识来源以及在何种情况下可以访问。例如：
If a user's query pertains to Technology & Software, which is/are about Topics focusing on codebases,
software platforms, applications, and user interactions in tech, make sure you try answering them by
only using document retrievers. Do not use other types of tools.
项目仓库中的基线智能体通过在提示词中简单添加以下内容来强制遵守这些策略：“你是一个可以访问工具的有用助手。\n 工具使用约束：{additional_instructions}。”当然，智能体构建者可以自由选择任何约束执行机制。
评估框架
VAKRA 在工具环境中评估智能体，其成功与否取决于执行连贯的多步骤工作流的能力以及答案的正确性。我们引入了一个以执行为中心的评估框架，该框架不仅评估最终输出，还评估完整的工具执行轨迹，包括工具调用、输入和中间结果。
评估指标
VAKRA 评估器针对每个样本处理两个关键输入：预测的最终回复和相应的工具调用轨迹。预测轨迹中的工具调用在与真实情况相同的环境中执行，以验证中间工具输出。
评估遵循一个瀑布式流程（图 6），其中后续阶段取决于前一个阶段的成功：
- 对于能力 4 的任务，首先通过编程方式验证策略遵守情况（此步骤不适用于其他能力）。
- 然后，将预测的工具调用序列与真实序列进行比较。
- 只有轨迹有效的样本才会进入最终回复评估阶段。
图 6：瀑布式评估流程
工具序列对比 由于存在可执行环境，智能体可以探索环境，有时会通过调用一组与我们识别出的不同的 API 来返回答案。为了支持替代但有效的工具调用和推理路径，正确性评估是通过执行每个预测的工具，并将工具响应集与真实标注的响应集进行比较来完成的（而不是强制执行严格的步骤级匹配）。
具体来说，我们首先执行程序化检查，验证预测的工具响应是否包含了真实标注工具响应中的所有信息。在涉及部分匹配、语义等价或表示方式差异（例如顺序、聚合或格式）的情况下，此检查可能无法得出明确结论。在这种情况下，我们采用基于大语言模型的二次评估，该方法改编自 CRAG 框架（Yang 等人，2024），以确定预测的轨迹是否检索到了所有必要信息，即使存在结构差异。此步骤使用一个改编后的提示词，来判断预测的轨迹是否捕获了所有必要信息，即使是通过不同的工具调用序列获得的。
最终响应评估 对于通过上一检查的轨迹，最终响应将使用基于大语言模型的评判器进行评估。此步骤确保响应（i）基于预测的工具输出，并且（ii）在事实上与真实标注答案保持一致，同时考虑到措辞或结构上的潜在差异。
这种设计确保智能体不仅因生成正确答案而获得奖励，也因其通过有效且完整的推理过程获得答案而获得奖励。
评分
每项能力权重相同，以得出最终的排行榜分数
$$ \text{排行榜分数} = \frac{1}{4} \sum_{n = 1}^{4} \text{能力}_{n} $$
排行榜分数 = 1/4 * (能力1 + 能力2 + 能力3 + 能力4)
为了获得能力分数，对于能力 1 到 3，每项能力内的每个样本权重相同。
$$ \text{能力}_{1 - 3} = \frac{\text{正确查询数}}{\text{总查询数}} $$
能力 1−3 = #总查询数 / #正确查询数
对于能力 4，我们对异构查询赋予更高权重：
$$ \text{能力}_{4} = \frac{# \text{ 正确多源查询} \times 2 + # \text{ 正确仅 API 或仅 RAG 查询}}{# \text{ 总多源查询} \times 2 + # \text{ 总仅 API 或仅 RAG 查询}} $$
能力 4 = #总多源查询 × 2 + #总仅 API 或仅 RAG 查询 / #正确多源查询 × 2 + #正确仅 API 或仅 RAG 查询
错误分析
现在我们针对 VAKRA 的四项能力进行详细的错误分析。为便于分析，我们采用分阶段错误分类法，将每次失败归因于首次出现问题的环节。具体来说，我们按顺序评估：(i) 是否选择了正确的工具，(ii) 是否提供了必需的参数且无遗漏或幻觉，(iii) 参数值是否正确，以及 (iv) 最终回答是否准确且基于工具输出。
失败阶段隔离
由于单个样本可能在不同步骤出现多个错误，我们按顺序将每个实例归类到最早失败的阶段（例如，工具选择错误优先于参数错误）。这避免了重复计数，并使错误类别可被解释为数据集中互不相交的部分。虽然可能存在更细粒度的指标（例如，工具使用上的精确率/召回率）（Elder 等人，2026），但我们发现这种表述方式能提供一种简单且可解释的智能体失败分解。
基准测试的这一部分中的实例需要选择并排序多个工具以完成单个任务。我们在此能力中有 2077 个样本。这对所有模型都具有挑战性，但 GPT-OSS-120B 在该基准测试的这一部分表现最佳。
- GPT-OSS-120B 以较大优势超越了其他模型，这主要归功于其对工具模式（tool schemas）有更好的理解。
- 基准测试这一部分涉及的工具包含大量参数，其中许多是可选的，而与其他模型相比，GPT-OSS-120B 在正确选择要填充的参数方面表现尤为稳健。
- 总体而言，在正确完成所有工具调用后合成正确答案，在这一部分基准测试中挑战性较低，这很可能是因为工具调用顺序使得工具选择问题比仪表盘 API 能力更不容易通过猜测来解决。
图 7：SEL-BIRD 与 SLOT-BIRD 错误类型分析
商业智能（BI）API 能力包含两组 API，分别来自 SLOT-BIRD 和 SEL-BIRD 工具集合。该基准测试的 SEL 部分有 600 个样本，而 SLOT 部分有 1477 个样本。这两个集合被归入 BI API 能力之下，但特征略有不同。SLOT-BIRD 集合包含数量较少但通用性较强的工具，每个工具需要填充大量参数值；而 SEL-BIRD 集合则包含更多工具，但每个工具的参数较少。这种侧重点反映在模型使用这两个工具集合时产生的相对错误上。
- 在使用 SLOT-BIRD 时，除 GPT-OSS-120b 之外的所有模型在正确生成工具参数名称方面都出现了大量错误。这很大程度上是 GPT-OSS-120b 在该基准测试部分整体表现如此出色的原因。
- 由于需要填充的参数较少，同样的模型在使用 SEL-BIRD 工具集合时几乎未出现此类错误，但它们在正确选择工具方面犯了更多错误，这反映出从更大（且动态）的工具集中进行选择的难度增加了。
- 如上所示，在工具选择能力的 1597 个样本中，Gemini-3-flash-preview 在所有错误类别上的表现均优于其他受测模型。
- 正如预期的那样，由于仪表盘 API 实例要求模型从大量工具选项中进行选择，但每个工具只需要少量参数，因此在工具选择和参数值选择方面出现了大量错误。
- 在模型幻觉或跳过必需参数方面似乎问题不大。然而，即使所有工具调用都正确执行，模型（尤其是 Gemini-3-flash-preview 和 Claude-Sonnet-4-5）仍然难以从工具响应中综合出正确答案，图表右侧的大幅下降趋势就证明了这一点。
多跳推理：跳数深度对模型性能的影响
图 8：不同模型按跳数深度的准确率对比
多跳推理要求模型成功回答多个隐式关联的问题，每个问题都需要选择并调用正确的 API，从而增加了原始任务的难度。正如预期，所有模型在仅需单次逻辑跳转的问题上表现最佳，而在 2 跳问题上的性能出现下降，在 3 跳及以上问题上性能进一步退化。
多跳多源推理：混合跳数对模型性能的影响
图 9：按交互类型（API、文档检索器、混合）划分的模型准确率
数据集的最后一部分除了其他部分中的工具/API 来源外，还包含了文档来源。这导致实例需要单次或多次 API 调用、单次或多次文档搜索，或者 API 调用与文档搜索的某种组合。
- 与之前一样，需要单次 API 调用（1 跳 API）的实例与需要多次 API 调用（2 跳 API）的实例在性能上存在显著差异，而加入文档检索器则使任务更具挑战性（RAG 跳数和混合跳数）。
- 有趣的是，我们发现对于需要单次文档检索器调用（1 跳 RAG）的问题，GPT-OSS-120B 试图直接从参数知识中返回答案，但当问题似乎需要多跳时，它则会回答问题。我们推测，由于 1 跳 RAG 的问题高度集中于维基百科实体，模型跳过了工具调用（我们在 1 跳 API 中未发现此问题，因为后端数据库特定的实体/事实可能更频繁地出现在问题中）。
- 有趣的是，Gemini-3-flash-preview 在 2 跳 API-RAG 上的性能相比其他混合跳数模式有显著提升。这很可能是因为 Gemini-3-flash-preview 在仪表盘 API（工具选择能力）上表现相对较强，因此，一旦通过工具调用确定了正确的中间答案，检索查询的成功率也会更高。
策略对模型性能的影响
图 10：按策略类型划分的模型准确率
策略在多跳、多源推理的基础上增加了额外的难度层。当策略与回答问题所需的来源一致时，即它们不影响模型回答问题所需的工具列表，我们称之为“答案无需更新”——如图 10 所示，除 Granite-4.0-h-Small-32B 外，所有模型在限制访问最相关信息源的策略约束下（即“策略更新了答案”），性能都出现了明显下降。
总体而言，我们发现模型要么违反约束，要么未能检索到足够的信息，有时它们理解了策略但无法正确回答问题，或者表现出之前分析过的某种失败模式。
总的来说，工具使用受策略约束的环境表明，虽然模型能够对工具和来源进行推理，但它们难以将外部约束纳入这种推理中——而这往往是实现可靠现实世界部署的关键要求。
结论
VAKRA 揭示了表面工具能力与稳健的端到端智能体可靠性之间的关键差距。尽管现代模型越来越能够选择 API 并执行孤立的工具调用，但 VAKRA 表明，仅凭这些能力不足以应对现实世界的部署。在实践中，当模型需要在执行约束下进行组合推理时——涉及 API、文档、对话上下文和策略要求——它们往往会崩溃。
试试 VAKRA——你的智能体在哪里崩溃？
认为你的智能体很可靠？那就来测试一下。
在 VAKRA 上运行它，看看它在哪个环节崩溃——是工具选择、多跳推理，还是策略约束。
- ⭐ 提交到排行榜：https://github.com/IBM/vakra?tab=readme-ov-file#submitting-to-the-live-leaderboard
- 📦 探索数据集：https://huggingface.co/datasets/ibm-research/VAKRA
- 🛠️ 查看代码：https://github.com/IBM/vakra
👉 试试看，并告诉我们你的智能体学到了什么
We recently introduced VAKRA, a tool-grounded, executable benchmark for evaluating how well AI agents reason and act in enterprise-like environments.
Unlike traditional benchmarks that test isolated skills, VAKRA measures compositional reasoning across APIs and documents, using full execution traces to assess whether agents can reliably complete multi-step workflows.
VAKRA provides an executable environment where agents interact with over 8,000+ locally hosted APIs backed by real databases spanning 62 domains, along with domain-aligned document collections. Tasks can require 3-7 step reasoning chains that combine structured API interaction with unstructured retrieval under natural-language tool-use constraints.
As can be seen below, models perform poorly on VAKRA - in this blog, we include additional dataset details about the tasks in VAKRA and present an analysis of failure modes we observed on different tasks.
Task Description
As shown below, the VAKRA benchmark comprises of four tasks, each testing a different set of capabilities.
Fig 1: Representative examples of each capability in the VAKRA benchmark
Capability 1: API Chaining using Business Intelligence APIs
This capability includes 2,077 test instances across 54 domains, requiring the use of tools from the SLOT-BIRD and SEL-BIRD collections (Elder et al., 2026). Compared to the setup in Elder et al., the tool universe in SLOT-BIRD and SEL-BIRD is expanded through the inclusion of a larger number of domains. Each domain is restricted to one tool collection, and tasks involve chaining 1–12 tool calls to arrive at the final answer.
{
"query": "Which football team has a build-up play speed of 31, build-up plan dribbling of 53, and build-up play passing of 32?",
"tool_calls":[
{
"name": "get_data",
"arguments":{"tool_universe_id="486ea46224d1-aeb8037c5e78"},
"label": "retrieved_data_1"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"retrieved_data_1","key_name":"play_speed","value":31},
"label": "FILTERED_DF_0"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"FILTERED_DF_0","key_name":"play_dribble","value":53},
"label": "FILTERED_DF_1"
},
{
"name": "select_data_equal_to",
"arguments":{"data_label":"FILTERED_DF_1","key_name":"play_passing","value":32},
"label": "FILTERED_DF_2"
},
{"name":{get_team_name},"arguments":{"data_label":"FILTERED_DF_2","n":1}}}],
"answer": "FC Barcelona"
}
Fig 2: Data sample from SEL-BIRD collection
As shown above, each instance has an associated JSON data source from which the answer must be derived. The MCP servers supporting this task include a special tool, called get_data(tool_universe_id=id), which must be called at the beginning of each instance. This tool initializes the data source, returns a lightweight preview of the data (see below Figure 3), and stores the full dataset server-side to avoid large data transfers. This prevents the inefficient transfer of large data over the MCP protocol. The call also configures the MCP server to expose the appropriate tool set based on the tool_universe_id and aligns the data source with the domain-specific database for the instance.
The SLOT-BIRD collection provides a global set of 7 tools for generic data manipulation (e.g., filtering, sorting), inspired by systems like Tableau and Google Analytics. The SEL-BIRD collection extends this by introducing more specialized tools: some are shared with SLOT-BIRD, while others are derived by flattening categorical arguments into separate functions (e.g., sort_data with argument ascending: bool = False becomes sort_data_ascending and sort_data_descending). Additionally, the generic (retrieve_data) function from SLOT-BIRD is replaced with query-specific getters. Every key in the data for a given instance has an associated get function (get_KEY_NAME) for an average of 4 get functions per instance.
{
"handle": "retrieved_data_1",
"num_records": 2,
"key_details": [
{"name": "team_name", "dtype": "str", "first_3_values": ["FC Barcelona", "Manchester City"]},
{"name": "play_speed", "dtype": "int32", "first_3_values": [31, 40]},
{"name": "play_dribble", "dtype": "int32", "first_3_values": [53, 30]},
{"name": "play_passing", "dtype": "int32", "first_3_values": [32, 16]}
]}
Fig 3: Data preview obtained from get_data function
Capability 2: Tool Selection using Dashboard APIs
This capability includes 1,597 instances across 17 domains, requiring tools from an expanded REST-BIRD collection (Elder et al.). These use endpoint-style interfaces that provide highly specific, query-aligned endpoints that encapsulate most computation. They are served as REST APIs running in a FastAPI server, which is wrapped by the MCP server. This task requires selecting the correct APIs from the domain-specific tool set (as shown in the example in Figure 1). Each domain contains a minimum of 6 to a maximum of 328 tools (with an average of 116 tools). Similar to the previous task, the get_data tool configures the MCP server to expose only the relevant domain-specific APIs.
The OpenAI API Specification restricts the tool list input to a maximum length of 128 tools. This restriction requires an agent builder using this API to manage the length of the tool list directly via a shortlisting mechanism. In the baseline agents in our repository, a simple shortlisting capability handles this challenge.
Capability 3: Multi-Hop Reasoning using Dashboard APIs
The Capability 3 segment of the benchmark has 869 test instances drawn from 38 subject domains. These instances rely again on the REST-BIRD API collection, but add multi-hop reasoning to the challenge (refer to example in Figure 1). Multi-hop questions require multiple pieces of supporting evidence to be extracted and combined to reach an answer. The instances in this section require between one and five logical hops to answer a query. The question types distribution for queries within the test dataset is shown below in Figure 4.
Fig 4: API Hop-Type distribution for Capability 3 (MultiHop) and Hybrid Hop-Type distribution for Capability 4 (MultiHop MultiSource Reasoning)
Capability 4: Multi-Hop, Multi-Source Reasoning and Policy Adherence
Capability 4 includes 644 instances across 41 domains and is also built on the REST-BIRD API collection. Figure 4 above shows a distribution of hybrid hops for test queries without policies. It contains the most complex queries with the following characteristics:
Multi-Source: This segment adds document indices per domain. Queries in this capability could require information from these document indexes as well as API calls. Similar to Capability 3, this task also has Multi-Hop queries. The required information source applies at the per-hop level, so, for example, a question may entail three logical hops with sources: API - RAG (Document Retrieval) - API. To enforce correct reasoning, sources are decontaminated during data generation, i.e. information required for a given hop is available in only one source. For example, if a hop is to be answered using APIs, the document index is built by removing documents that likely contain the information needed to answer the question.
Multi-Turn: This segment of the dataset also adds multi-turn conversations to the setting. Each instance is a dialog with multiple turns. The data is released as context-response pairs, where the context encodes the current dialog history and the agent is only responsible for answering the current turn.
Tool-usage Policies: A subset of these instances includes tool-use policies that the agent is required to follow. These policies take the form of plain-text instructions about the knowledge sources that the agent is allowed to access and under which circumstances. For example:
If a user's query pertains to Technology & Software, which is/are about Topics focusing on codebases,
software platforms, applications, and user interactions in tech, make sure you try answering them by
only using document retrievers. Do not use other types of tools.
The baseline agent in the project repo imposes adherence to these policies through a simple addition to the prompt: "You are a helpful assistant with access to tools.\n Tool Usage Constraint: {additional_instructions}.". Of course, agent builders are free to choose any constraint enforcement mechanism.
Evaluation Framework
VAKRA evaluates agents in tool environments where success depends on both the ability to execute coherent, multi-step workflows and answer correctness. We introduce an execution-centric evaluation framework that assesses not only final outputs but also the full tool-execution trajectory that includes tool calls, inputs, and intermediate results.
Evaluation Metric
The VAKRA Evaluator operates over two key inputs for each sample: a predicted final response and the corresponding tool-call trajectory. The tool calls from the predicted trajectory are executed in the same environment as the ground truth to verify intermediate tool outputs.
The evaluation follows a waterfall-style pipeline (Figure 6), where later stages are conditioned on earlier success:
- For Capability 4 tasks, policy adherence is first verified programmatically (this step is not applied to other capabilities).
- The predicted tool call sequence is then compared against the ground truth sequence.
- Only samples with valid trajectories proceed to final response evaluation.
Fig 6: Waterfall-style Evaluation Pipeline
Tool-Sequence Comparison Due to the presence of an executable environment, agents can explore the environment and sometimes return the answer by invoking a different set of APIs than the ones identified by us. In order to support alternative but valid tool invocations and reasoning paths, correctness is assessed by executing each predicted tool and comparing the set of tool responses against those from the ground truth (rather than enforcing strict step-level matching).
Specifically, we first perform a programmatic check, verifying whether all information present in the ground-truth tool responses is recovered by the predicted tool responses. This check may be inconclusive in cases involving partial matches, semantic equivalence, or differences in representation (e.g., ordering, aggregation, or formatting). In such cases, we apply a secondary LLM-based evaluation, adapted from the CRAG framework Yang et al., 2024, to determine whether the predicted trajectory retrieves all required information despite structural differences. This step uses an adapted prompt to determine whether the predicted trajectory captures all required information, even if obtained through a different sequence of tool calls.
Final Response Evaluation For trajectories that pass the previous check, the final response is evaluated using an LLM-based judge. This step ensures that the response is (i) grounded in the predicted tool outputs, and (ii) factually consistent with the ground truth answer, accounting for potential variations in phrasing or structure.
This design ensures that agents are rewarded not only for producing correct answers, but for obtaining them through valid and complete reasoning processes.
Scoring
Every capability is equally weighted to obtain a final leaderboard score
$$ \text{Leaderboard}\text{Score} = \frac{1}{4} \sum{n = 1}^{4} \text{Capability}_{n} $$
Leaderboard_Score=4 1n=1∑4Capability n
To obtain a capability score, every sample within a capability is equally weighted for capabilities 1 through 3.
$$ \text{Capability}_{1 - 3} = \frac{# \textrm{ } \text{correct}\textrm{ }\text{queries}}{# \textrm{ } \text{total}\textrm{ }\text{queries}} $$
Capability 1−3=#total queries#correct queries
For capability 4, we weight heterogeneous queries higher:
$$ \text{Capability}_{4} = \frac{# \textrm{ } \text{correct}\textrm{ }\text{multi}-\text{source} \times 2 + # \textrm{ } \text{correct}\textrm{ }\text{API}-\text{only}\textrm{ }\text{or}\textrm{ }\text{RAG}-\text{only}}{# \textrm{ } \text{total}\textrm{ }\text{multi}-\text{source} \times 2 + # \textrm{ } \text{total}\textrm{ }\text{API}-\text{only}\textrm{ }\text{or}\textrm{ }\text{RAG}-\text{only}} $$
Capability 4=#total multi-source×2+#total API-only or RAG-only#correct multi-source×2+#correct API-only or RAG-only
Error Analysis
We now present detailed error analysis across the four VAKRA capabilities. To facilitate our analysis, we adopt stage-wise error categorization to assign each failure to the first point of breakdown. Specifically, we evaluate, in order: (i) whether the correct tool(s) were selected, (ii) whether the required arguments were provided without omissions or hallucinations, (iii) whether argument values were correct, and (iv) whether the final response is both accurate and grounded in the tool outputs.
Failure Stage Isolation
Since a single sample may exhibit multiple errors across different steps, we sequentially classify each instance to the earliest failing stage (e.g., tool selection errors take precedence over argument errors). This avoids double-counting and allows error categories to be interpreted as disjoint fractions of the dataset. While more granular metrics (e.g., precision/recall over tool usage) are possible (Elder et al., 2026), we find this formulation provides a simple and interpretable breakdown of agent failures.
The instances in this part of the benchmark required selecting and sequencing multiple tools to solve a single task. We have 2077 samples in this capability. This was challenging for all models, but GPT-OSS-120B performed best on this segment of the benchmark.
- GPT-OSS-120B outperformed the other models by a large margin, mostly from a better understanding of the tool schemas.
- The tools in this part of the benchmark involve a large number of parameters, many of which are optional, and GPT-OSS-120B was especially robust, as compared to the others, at choosing the right parameters to fill.
- Overall, synthesizing a correct answer after making all tool calls correctly was less challenging in this section of the benchmark, most likely because the tool call sequencing made the tool choice problem less amenable to guessing compared with the Dashboard API capability
Fig 7: SEL-BIRD vs SLOT-BIRD Error Types Analysis
The Business Intelligence (BI) API capability contains two sets of APIs, from the SLOT-BIRD and SEL-BIRD tool collections. The SEL part of this benchmark had 600 samples, while the SLOT part of the benchmark had 1477 samples. These two collections are grouped under the BI API capability, but have slightly different characteristics. The SLOT-BIRD collection has a smaller number of generic tools with a large number of parameter values to fill, while the SEL-BIRD collection has a larger set of tools and fewer parameters per tool. This focus is reflected in the relative errors made by models using these two tool collections.
- Using SLOT-BIRD, all models except for GPT-OSS-120b made a substantial number of errors producing correct names for the tool arguments. This is largely the reason that GPT-OSS-120b performed so well overall in this segment of the benchmark.
- With fewer parameters to fill, the same models made very few such errors when using the SEL-BIRD tool collection, but they made many more errors selecting the correct tools, reflecting the increased difficulty of choosing from a larger (and dynamic) tool set.
- As shown above, for the 1597 samples in the tool selection capability, Gemini-3-flash-preview outperforms the other models tested on all error categories.
- As expected, since the dashboard API instances require the models to choose from a large number of tool options, but each tool requires only a small number of parameters, there are a large number of errors in tool selection and parameter value selection.
- There seems to be little problem with hallucinating or skipping required parameters. However, even when all tool calls are made correctly, models (especially Gemini-3-flash-preview and Claude-Sonnet-4-5 still struggle to synthesize a correct answer from the tool responses, as evidenced by the large drop-offs at the far right side of the plot.
Multi-Hop Reasoning: Effect of Hop Depth on Model Performance
Fig 8: Comparison of Accuracy Across Models by Hop Depth
Multi-hop reasoning increases the difficulty of the original task by requiring models to successfully answer multiple implicitly coupled questions, each of which requires selecting and calling the correct API. As expected, all models performed best on the questions with only a single logical hop, and saw performance degradations on 2-hop and again on 3+ hop questions.
Multi-Hop Multi-Source Reasoning: Effect of Hybrid Hops on Model Performance
Fig 9: Model Accuracy Rates by Interaction Type (API, Document-Retriever, Hybrid)
The final segment of the dataset includes document sources in addition to the tool/API sources in the other segments. This leads to instances that require single or multiple API calls, single or multiple document searches, or some combination of API calls and document searches.
- As before, there is a marked difference in performance on instances that require single API calls (1-hop API) as compared to those that require multiple API invocations (2-hop API), and including document retrievers makes the task more challenging (RAG Hops and Hybrid).
- Interestingly, we find that on questions that require a single document retriever call (1-hop RAG), GPT-OSS-120B tries to directly return the answer from parameter knowledge, though when the question appears to require multiple hops, it answers the question. We hypothesize that since the questions for 1-hop RAG are very Wikipedia-entity focussed the model skips the tool call (we don't see this problem on 1-hop API, where back-end database-specific entities/facts might be present more frequently in the question).
- It is also interesting that the performance of Gemini-3-flash-preview shoots up on 2-hop API-RAG as compared to other hybrid hop-patterns. This is likely explained by the relatively strong performance of Gemini-3-flash-preview on the dashboard APIs (Tool Selection Capability), and thus, once the correct intermediate answer is identified using the tool-call, the retrieval query is likely to be more successful.
Effect of Policies on Model Performance
Fig 10: Model Accuracy Rates by Policy Type
Policies introduce an additional layer of difficulty on top of multi-hop, multi-source reasoning. When policies align with the required source for answering i.e. they do not affect the tool list required for models to answer the question, we refer to it as "No Updates to Answer" -- as shown in Figure 10, all models except for Granite-4.0-h-Small-32B experience a clear drop in performance under policy constraints that restrict access to the most relevant information source (i.e. "Policy updates the answer").
In general, we find that models either violate constraints or fail to retrieve sufficient information, where they sometimes understood the policy but could not answer the question correctly, or they exhibit one of the previously analyzed failure modes.
Overall, tool-use policy-constrained settings suggest that while models can reason over tools and sources, they struggle to incorporate external constraints into that reasoning - often a key requirement for reliable real-world deployment.
Conclusion
VAKRA exposes a critical gap between surface-level tool competence and robust, end‑to‑end agent reliability. Although modern models can increasingly select APIs and execute isolated tool calls, VAKRA shows that these abilities alone are insufficient for real‑world deployment. In practice, models often break down when required to perform compositional reasoning under execution constraints—spanning APIs, documents, dialog context, and policy requirements.
Try VAKRA — Where Does Your Agent Break?
Think your agent is solid? Put it to the test.
Run it on VAKRA and see where it falls apart—tool selection, multi-hop reasoning, or policy constraints.
- ⭐ Submit to the leaderboard: https://github.com/IBM/vakra?tab=readme-ov-file#submitting-to-the-live-leaderboard
- 📦 Explore the dataset: https://huggingface.co/datasets/ibm-research/VAKRA
- 🛠️ Check out the code: https://github.com/IBM/vakra
👉 Try it and tell us what your agent learned