# MAPD：多智能体协议蒸馏弥合闭源到开源模型的分布差距

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-07-27 08:00
- AIHOT 分数：48
- AIHOT 链接：https://aihot.virxact.com/items/cms44cyp105y5ro82s73mrlcz
- 原文链接：https://arxiv.org/abs/2607.24280

## AI 摘要

研究者提出多智能体协议蒸馏（MAPD）框架，通过结构化JSON协议为开源学生模型提供密集蒸馏信号，解决异构分词器与隐藏logits导致的分布差距问题。在七个QA基准测试中，MAPD在Qwen3-1.7B和Qwen3-4B上持续优于对比方法，并有效缓解风格漂移与冗长退化。

## 正文

Junlin Liu

Jiangwang Chen

Zixin Song

Shuaiyu Zhou

Chunji Lv

Hank Wu

Kailin Jiang

Bohan Yu

Chenxi Zhou

Abstract

Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of on Qwen3-1.7B and on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration. The code is open sourced in https://github.com/AaronLiu0702/MAPD

1 Introduction

Driven by the rapid advancement of large language models (LLMs), research has shifted from single-turn reasoning to multi-turn agentic tasks, with agentic search emerging as a crucial paradigm for tackling knowledge-intensive challenges (jin2025search; zhang2025web; zheng2025deepresearcher; luo2025globalrag; liang2026search). However, existing agentic search systems are often optimized with outcome-based reinforcement learning from verifiable rewards (RLVR), where final-answer correctness provides only sparse supervision for long interaction trajectories. Recent studies supplement RLVR with online policy distillation (OPD), using token-level distribution alignment to provide denser guidance for intermediate decisions (dong2025agentic; wang2026openclaw; lu2026self).

Although combining RL with OPD provides a more reliable training signal, the overall efficacy of OPD strictly hinges on a high-quality teacher policy. To unlock the potential of open-source models in complex agentic search, leveraging state-of-the-art proprietary models as expert teachers is highly desirable, given their exceptional capabilities in dynamic task decomposition, multi-hop reasoning, and information synthesis. However, distilling knowledge from these proprietary teachers to open-source students presents two major bottlenecks. First, traditional OPD fundamentally relies on token-level KL divergence alignment. Yet, the discrepancy of heterogeneous tokenizers and the absence of accessible teacher logits render this mathematical alignment undefined (agarwal2024policy; li2026rethinking). Second, as a common fallback, natural language trajectory distillation frequently fails in agentic scenarios. By forcing open-source student models to mimic the verbose and idiosyncratic reasoning styles of proprietary experts, this approach exacerbates the distribution gap. It induces severe style drift and hallucinations, failing to transfer the underlying cognitive strategies while degrading performance in agentic tasks (gudibande2023false; zhao2026self). Existing approaches therefore lack an intermediate representation that can convert black-box expert behavior into dense supervision without exposing the student to teacher-specific surface forms.

Figure 1: Distilling from proprietary teacher to open-source student faces two bottlenecks: (1) heterogeneous tokenizers and inaccessible logits preclude direct alignment; (2) free-form natural-language trajectory imitation induces style drift and hallucination.

To address these bottlenecks, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework built around a structured, style-normalized intermediate representation between proprietary experts and open-source students. MAPD separates offline protocol synthesis from online policy alignment. In the offline stage, a proprietary-model-powered MAS decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the exploration trace into a JSON protocol. In the training stage, this protocol is reformatted as privileged information into a conditioned branch of the student policy. Since this privileged branch and the student branch share a same model, their distributions are aligned via a dense self-distillation objective that complements the sparse outcome-based RL objective. By distilling through structured protocols rather than free-form natural-language trajectories, MAPD reduces the student’s exposure to teacher-specific verbosity and formatting artifacts. Crucially, the proprietary model and MAS are used only for offline protocol synthesis, adding zero overhead to inference. Our main contributions are summarized as follows:

•

We propose MAPD, a joint on-policy self-distillation and outcome-based RL framework designed to bridge the distribution gap between proprietary and open-source. By aligning a protocol-conditioned branch with a student branch of the same policy, it provides dense guidance without requiring access to proprietary logits or cross-tokenizer alignment.

•

We introduce a structured, style-normalized JSON protocol as the intermediate representation to decouple core strategies from the linguistic shell. To reliably synthesize protocols, we construct a robust MAS with specialized roles and rigorous automated checks.

•

Experiments with three top-tier proprietary models (Claude-Opus-4.6, GPT-5.5, and Gemini-3.1-Pro) as expert teachers demonstrate that MAPD yields reliable gains and these improvements can transfer across different proprietary models without retuning the pipeline.

2 Problem Formulation & Preliminaries

2.1 Agentic Search

We formulate agentic search as a multi-turn interaction paradigm. Given a question , a language model policy interacts with a retrieval-augmented environment for up to turns. At each turn, the agent generates reasoning steps and optionally issues a retrieval query, to which the environment responds with relevant passages as the next observation. For notational simplicity, we flatten the entire generated trajectory across all turns into a single token sequence

(1)

The interaction terminates either when the agent emits a designated final answer string or reaches the maximum turn limit . Each completed episode receives a sparse, trajectory-level binary reward , which is determined by parsing the final predicted answer from and evaluating it via strict exact-match (EM) against the ground-truth.

2.2 Group Relative Policy Optimization (GRPO)

For a given prompt , GRPO (shao2024deepseekmath) samples a group of rollouts and computes advantages derived from the terminal rewards. The policy is updated by optimizing a clipped surrogate objective alongside a per-token KL penalty against a reference model :

(2)

Where denotes the per-token importance weight. While GRPO effectively drives exploration, it strictly evaluates outcome success and lacks the granular supervision required to optimize complex intermediate reasoning steps.

2.3 Online Policy Self-Distillation (OPSD)

To supply denser token-level supervision and follow the context-conditioned self-distillation formulation of OPSD (zhao2026self), we obtain two next-token distributions from the same current policy parameters. At optimization step , the student branch conditions on , whereas the teacher branch additionally conditions on , where denotes the privileged information (PI) available exclusively during training. By restricting gradient flow solely to the student path during backpropagation, OPSD minimizes the per-token reverse KL divergence:

(3)

This homologous formulation inherently circumvents the vocabulary mismatch issues encountered in proprietary model distillation. However, standard OPSD typically derives the PI from the student’s own correct rollouts, which fundamentally bounds the supervision quality by the student’s existing reasoning limits.

3 Methodology

In this section, we first introduce the Structured JSON Protocol (Section 3.1) that normalizes privileged information and decouples cognitive strategies from proprietary model-specific linguistic patterns. We then detail the Multi-Agent System Generation Pipeline (Section 3.2), which automates the distillation of raw teacher trajectories into our compact protocol through a three-stage collaborative process with rigorous quality control. Finally, we present the Joint Training Objective (Section 3.3) that combines a token-level OPSD distillation signal with a sparse GRPO reinforcement learning signal to effectively align the student model.

Figure 2: Overview of the MAPD framework. (1) A proprietary-model-powered multi-agent system generates high-quality structured protocols. (2) The structured JSON protocol serves as a bridge between proprietary teacher and open-source student. (3) The student model internalizes the teacher’s capability through joint training with protocols as privileged information.

3.1 Structured JSON Protocol

Directly utilizing free-form natural language trajectories as privileged information for OPSD can introduce a substantial representation mismatch or induce severe style drift and hallucinations, because they contain model-specific verbosity, formatting conventions, and linguistic patterns. To address this, we introduce a style-normalized Structured JSON Protocol as the fundamental unit of distillation, and this design reduces the student’s direct exposure to teacher-specific language and provides traceable evidence for the privileged teacher branch. Specifically, rather than extracting raw text, this protocol establishes a purified, traceable format (e.g., Example 3.1). Formally, the protocol encapsulates five essential dimensions of task:

•

Task Type: Determined by the proprietary expert, this field categorizes the query into single_hop, multi_hop, comparison, others, thereby dictating the overarching cognitive strategy.

•

Reasoning Plan: An ordered array of actionable sub-goals that strictly decomposes the complex agentic task into tractable search and inference steps.

•

Grounding Facts: A curated set of evidence extracted from the environment, which enforces faithful reasoning and actively mitigates parametric hallucinations.

•

Partial Findings: An optional field recording intermediate discoveries when multiple search rounds fail to yield the correct answer.

•

Answer Verification: The final extracted answer paired with a boolean flag (answer_grounded), ensuring the conclusion is rigorously supported by the retrieved facts.

By formatting the protocol via a descriptive template such that the PI is defined as , we effectively decouple the core cognitive strategies from the proprietary model’s surface linguistic patterns. This mechanism not only facilitates the transfer of high-quality reasoning capabilities into OPSD, but also effectively preserves the student’s native token distribution.

3.2 Multi-Agent System Generation Pipeline

While the structured JSON protocol mitigates style drift, reliably extracting such knowledge from a proprietary teacher remains challenging. We automate this via a three-stage MAS pipeline that distills the teacher’s raw problem-solving into our compact protocol. Crucially, to prevent data leakage, ground-truth answers are strictly confined to offline synthesis and the privileged teacher context, remaining completely invisible to the student.

Stage A: Multi-Agent Collaborative Search.

Given a question , the Orchestrator agent decomposes into sub-tasks with explicit dependency annotations. Subsequently, these sub-tasks are dispatched in parallel to independent Searcher agents. In response, each Searcher issues up to queries to a local Wikipedia corpus and compresses the retrieved passages into a concise finding. Once all findings within a topological level are collected, the Orchestrator synthesizes them; if the accumulated evidence remains insufficient, it formulates an additional round of sub-tasks (up to rounds). After the initial exploration pass, an exact match (EM) check evaluates whether the candidate answer matches the ground truth answer. If the check fails, a Repair agent leverages the ground-truth answer as diagnostic guidance to trace back the reasoning chain (The GT answer is never propagated into the exploration log and retrieval queries). It diagnoses the failure as either an expression issue (correct evidence retrieved but answer poorly formatted) or a search issue. It then triggers targeted re-decomposition and additional search operations accordingly, for up to repair iterations. Ultimately, the output of Stage A comprises an original question, a candidate answer, a binary success label, and a comprehensive exploration log documenting every sub-task, query, retrieved passage, and finding.

Stage B: Protocol Generation.

To transform the exhaustive yet unstructured exploration log from Stage A into our structured JSON protocol target, a Protocolizer agent is deployed. Conditioned on the success label, the protocolizer compiles this raw information into one of two structured JSON variants. For succeeded trajectories, it generates a Succeeded Protocol that includes the fully verified answer. Conversely, if the search failed to converge, it generates an Evidence Protocol; here, a definitive answer is omitted and replaced with a neutral summary of partial findings (e.g., “Confirmed , but reliable evidence for is missing”). To guarantee generation quality, two integrity constraints are strictly enforced: (i) No Hindsight: the reasoning_plan must be formulated step-by-step, strictly prohibiting any mention of the final outcome or subsequent search results to prevent data leakage; and (ii) Extractive Grounding: the grounding_facts must be strictly extractive, appearing as verbatim substrings within the retrieved passages.

Stage C: Quality Gate.

Before a generated protocol is admitted into the final training phase, it must pass a rigorous quality gate comprising four automated checks: (i) Schema Validation, ensuring a well-formed and valid JSON structure; (ii) EM Consistency (for succeeded protocol), confirming the extracted answer strictly matches the ground truth; (iii) Grounding Verification, enforcing extractive substring matching for all provided facts; and (iv) Leak Detection, guaranteeing no oracle knowledge is injected into the reasoning steps. Protocols failing any of these criteria are discarded, and the corresponding training falls back to a self-rollout baseline. Finally, manual review confirms the robustness of this pipeline, demonstrating a accuracy in yielding high-quality, strictly compliant structured JSON protocols (total instances across three representative models).

3.3 Joint Training Objective

The overall training objective integrates a sparse, outcome-driven reinforcement learning signal with a dense distillation signal to achieve high-quality alignment:

(4)

where provides a token-level distillation signal derived from the structured protocol, which facilitates efficient step-by-step credit assignment. Conversely, provides a sparse RL signal via clipped policy gradients, optimizing the model toward the final environment reward. The hyperparameter dictates the relative strength of this distillation guidance (extensively ablated in Section 4.2).

4 Experiments and Analysis

4.1 Experimental Setup

Data Splits and Benchmarks.

Following Search-R1 (jin2025search), our training uses only the NQ and HotpotQA train splits. Evaluation spans seven knowledge-intensive QA benchmarks, encompassing single-hop (NQ, TriviaQA, and PopQA) and multi-hop (HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle) reasoning tasks (kwiatkowski2019natural; min2019discrete; mallen2023not; yang2018hotpotqa; ho2020constructing; trivedi2022musique; press2023measuring). Five of the seven evaluation sets remain entirely absent from training, serving as out-of-distribution tests; for the two shared benchmarks, we ensure no question-level overlap between train and test splits. Performance is reported as success rate (%) via exact match against ground-truth answers.

Model Method Single-hop QA Multi-hop QA

NQ TriviaQA PopQA HotpotQA 2Wiki MuSiQue Bamboogle Avg.

Qwen3-1.7B Vanilla 29.90.3 46.40.2 39.10.1 23.90.2 18.80.1 4.90.3 16.81.1 25.70.1

OPSD 4.20.1 8.30.1 4.60.1 6.60.1 15.30.1 0.70.1 1.60.5 5.90.1

GRPO 36.90.4 54.60.3 43.10.1 29.80.3 27.50.2 6.80.4 22.41.6 31.60.2

GRPO+OPSD 37.00.3 54.60.2 41.40.1 29.90.3 23.30.2 6.50.3 20.81.6 30.50.2

SDAR 43.40.4 58.10.3 47.60.2 37.00.3 34.30.3 10.60.5 32.01.6 37.60.3

MAPD(Ours) 45.10.2 58.60.2 48.60.1 39.00.2 36.20.2 11.20.3 36.81.1 39.40.1

Improvement (%) 3.9% 0.9% 2.1% 5.4% 5.5% 5.7% 15.0% 4.8%

Qwen3-4B Vanilla 29.30.3 48.00.2 34.90.1 26.50.2 30.90.2 6.10.3 25.61.6 28.80.2

OPSD 16.20.2 36.70.1 23.10.1 20.50.2 24.60.1 5.70.2 19.21.1 20.90.1

GRPO 40.40.3 61.00.2 43.50.1 36.70.3 37.30.2 10.30.4 32.81.6 37.40.2

GRPO+OPSD 36.30.3 57.90.2 43.10.1 37.70.3 36.70.2 12.70.3 44.01.6 38.30.2

SDAR 46.10.4 62.40.2 48.50.2 43.30.3 43.10.3 13.10.5 44.81.6 43.00.3

MAPD(Ours) 47.70.2 62.50.2 49.40.1 44.30.2 45.70.2 14.00.3 47.21.1 44.40.1

Improvement (%) 3.5% 0.2% 1.9% 2.3% 6.0% 6.9% 5.4% 3.3%

Table 1: Results are reported as success rate (%) (mean std across five independent evaluation seeds from a single training run). Avg. denotes the macro-averaged success rate. Improvement (%) is the relative gain of MAPD over the SDAR baseline. Bold: best performance; underline: second best.

Baselines.

To rigorously assess the effectiveness of our approach, we compare against a spectrum of baselines, ranging from pure alignment strategies to advanced hybrid methods: (1) Vanilla: The pretrained base model evaluated zero-shot without any further alignment; (2) OPSD (zhao2026self): Pure online policy self-distillation, which utilizes the student’s own successful rollouts as privileged information; (3) GRPO (shao2024deepseekmath): Pure outcome-based reinforcement learning, relying solely on sparse environment rewards without any dense distillation guidance; (4) GRPO+OPSD: A naive hybrid approach combining RL with online policy self-distillation, using successful rollouts as privileged information; (5) SDAR (lu2026self): An advanced hybrid baseline combining RL with token-level gated distillation, utilizing skill-generated content as privileged information; (6) MAPD (Ours): Our proposed Multi-Agent Protocol Distillation framework, integrating RL with dense, token-level distillation guided by the structured MAS protocol without gating mechanisms.

Implementation Details.

For the student policy, we train two open-source models, Qwen3-1.7B and Qwen3-4B (yang2025qwen3), both initialized from their respective pretrained base checkpoints. During joint training, we utilize a group size of rollouts per prompt, a batch size of , a maximum prompt length of , and a maximum response length of tokens. The learning rate is set to with a linear warmup. Training runs for steps across 8 GPUs. The agentic environment allows up to interaction turns, utilizing a local Wikipedia corpus (wiki-18) as the retrieval backend, returning the top-3 passages per query.

MAS Pipeline Configuration.

For protocol synthesis, the MAS agents (Orchestrator, Searcher, Repair, and Protocolizer) utilize Claude-Opus-4.6 as the default backbone LLM. We also explore GPT-5.5 and Gemini-3.1-Pro as alternative teacher sources (ablated in Section 4.2). The MAS parameters are constrained as follows: the Orchestrator decomposes each query into a maximum of sub-questions over at most rounds; each Searcher issues up to retrieval queries; and the Repair agent is permitted up to diagnostic rounds. To ensure training efficiency, all protocols are pre-synthesized offline and cached, entirely decoupling the teacher’s generation latency from the student’s RL loop.

4.2 Results and Analysis

Main Result.

As shown in Table 1, our proposed MAPD framework consistently outperforms all baseline methods across the seven benchmarks, achieving average success rates of on Qwen3-1.7B and on Qwen3-4B. Compared to the strongest baseline SDAR, MAPD yields relative average improvements of and , respectively. Crucially, these performance gains are more pronounced on complex multi-hop QA tasks. For Qwen3-1.7B, the average relative gain on multi-hop benchmarks reaches 7.9%, substantially exceeding the 2.3% on single-hop benchmarks. A similar pattern holds for Qwen3-4B (5.1% vs. 1.8%). These larger relative gains on multi-hop benchmarks are consistent with the intended role of multi-agent decomposition and evidence aggregation, as the MAS pipeline in MAPD systematically breaks down complex queries, synthesizes evidence across multiple retrieval steps, and distills these strategies into the student policy via our structured protocol, thereby raising its upper bound for complex reasoning. Furthermore, we observe a catastrophic performance collapse when applying pure OPSD to the models under our agentic training configuration (dropping to 5.9% on Qwen3-1.7B and 20.9% on Qwen3-4B). This indicates that dense self-distillation is insufficient for agentic search and motivates the use of external PI. Lacking external information gain, self-rollouts merely drive the model toward pathologically long outputs——as evidenced by the length clip ratio surging from to approximately during training, a collapse that underscores the necessity of our MAS protocol.

媒体内容 · 前往原文查看

Method PM SP MAS Avg.

Qwen3-1.7B Qwen3-4B

GRPO+OPSD ✗ ✗ ✗ 30.5 38.3

SDAR ✗ ✗ ✗ 37.6 43.0

Raw trajectory (single PM) ✓ ✗ ✗ 30.1 37.3

Raw trajectory (MAS) ✓ ✗ ✓ 29.2 36.1

Protocol (single PM) ✓ ✓ ✗ 37.1 42.9

MAPD (Ours) ✓ ✓ ✓ 39.4 44.4

Table 2: Component ablation study of MAPD. PM = Proprietary Model; SP = Structured Protocol; MAS = Multi-Agent System.

Ablation Study.

We systematically ablate the Proprietary Model (PM), Structured Protocol (SP), and Multi-Agent System (MAS) to assess the contribution of each component, results are reported in Table 2.

(1) Structured protocol is essential for cross-model distillation. The most striking finding is that directly distilling from a proprietary model’s raw natural-language trajectories (w/o SP&MAS) performs worse than baselines using no proprietary model at all (GRPO+OPSD). By comparing w/o SP&MAS with w/o MAS, we isolate the impact of the representation format under a single-teacher setting: replacing raw text with our structured protocol drives an observed gain of 7.0 and 5.6 points, respectively. This is consistent with the style drift and distribution gap bottlenecks discussed in Section 1: without structured decoupling, the student imitates the teacher’s surface patterns (e.g., verbose reasoning chains, idiosyncratic phrasing), which conflicts with its own learned distribution and actively degrades performance.

(2) MAS cannot replace structured protocol. Adding the MAS pipeline without the structured protocol (w/o SP: 29.2%/36.1%) does not improve performance and is slightly worse, because the MAS produces richer but also noisier natural-language trajectories that exacerbate the style-transfer problem. Conversely, applying the structured protocol generated by single proprietary agent (w/o MAS: 37.1%/42.9%) recovers a relative gain, showing that the JSON protocol alone already alleviates style contamination. However, this single-agent protocol still underperforms SDAR, indicating that a single proprietary model call cannot match the retrieval depth and error recovery of the full MAS pipeline.

(3) MAS pipeline improves the quality of protocol generation. The full MAPD configuration performs best, suggesting that the structured representation and the MAS pipeline provide complementary benefits in the evaluated setting. Specifically, SP ensures that the distilled signal largely mitigates style drift and remains learnable, while MAS ensures that the signal is factually accurate, reasoning-complete and traceable. Compared with the single-agent protocol without MAS, this full protocol delivers a significantly higher-quality training signal that bridges the gap between proprietary and open-source models. Moreover, our results demonstrate that the primary bottleneck in distillation lies in PI quality rather than training mechanisms: with sufficiently clean, style-decoupled protocols, a simple token-level KL objective achieves performance comparable to gated approaches (SDAR).

Proprietary Teacher Model Single-hop QA Multi-hop QA

NQ TriviaQA PopQA HotpotQA 2Wiki MuSiQue Bamboogle Avg.

Claude-Opus-4.6 Qwen3-1.7B 0.01 44.1 58.0 47.5 37.8 33.5 10.9 28.0 37.1

0.05 45.1 58.6 48.6 39.0 36.2 11.2 36.8 39.4

0.10 43.7 58.2 49.1 38.2 36.6 10.2 36.0 38.9

Qwen3-4B 0.01 46.3 63.0 48.3 42.5 37.0 13.9 43.2 42.0

0.05 47.7 62.5 49.4 44.3 45.7 14.0 47.2 44.4

0.10 47.4 63.7 50.1 42.3 38.4 11.1 42.4 42.2

GPT-5.5 Qwen3-1.7B 0.01 43.5 58.6 49.1 38.4 35.5 11.2 36.0 38.9

0.05 43.5 59.7 48.6 38.7 38.0 9.3 35.2 39.0

0.10 43.3 58.4 48.6 36.4 31.9 9.2 32.8 37.2

Qwen3-4B 0.01 47.4 62.2 48.7 43.5 46.0 14.3 46.4 44.1

0.05 48.3 62.9 50.6 44.2 44.6 14.6 47.2 44.6

0.10 47.4 63.7 50.3 43.4 43.1 12.6 44.0 43.5

Gemini-3.1-Pro Qwen3-1.7B 0.01 42.8 58.2 48.9 37.4 32.4 9.9 33.6 37.6

0.05 43.2 58.9 47.5 37.7 36.3 9.1 32.8 37.9

0.10 42.5 58.3 47.9 37.7 32.6 10.5 35.2 37.8

Qwen3-4B 0.01 48.8 63.7 47.9 44.8 45.2 13.0 43.2 43.8

0.05 48.9 62.8 50.8 43.8 39.1 14.0 48.8 44.0

0.10 47.4 62.5 49.3 43.0 42.3 12.8 43.2 42.9

Table 3: Effect of three proprietary models as privileged information source and OPSD loss weight . Results are reported as Success Rate (%). Bold: best performance.

Effect of the OPSD Loss Weight.

Table 3 and Figure 4 summarizes the sensitivity of our framework to the OPSD loss weight in the joint objective. Across both model scales, consistently achieves the best among the tested values (39.4% on 1.7B and 44.4% on 4B), revealing a sweet-spot associated with two observed patterns at the extremes. Below we analyze these in detail.

(1) Weak distillation signal (). A small OPSD weight provides only a weak distillation signal relative to the dominant GRPO policy-gradient objective. Training logs on Claude-Opus-4.6 reveal that the teacher-student gap collapses prematurely by step 100, indicating that the student quickly exhausts the available supervision. However, the KL divergence from the reference policy remains low (0.24 on 1.7B), suggesting that the model does not simply revert to the reference behavior. Despite the early disappearance of the distillation signal, this configuration still achieves a clear improvement over GRPO alone (37.1% vs. 31.6% on 1.7B), implying that even a short-lived OPSD term can provide beneficial guidance during the initial phase of training.

(2) Excessive distillation pressure (). A large OPSD weight pushes the student aggressively toward the teacher’s distribution, which inflates the KL divergence from the reference policy (1.06 on 4B, nearly that of ). More importantly, we observe a behavioral degeneration specific to agentic search: on 4B the mean response length collapses from 135 tokens to roughly 42 tokens while the number of tool calls saturates at the maximum of 3.0 per episode. This pattern suggests that excessive distillation pressure drives the model toward a “retrieve-without-reasoning” shortcut–issuing retrieval calls with minimal intermediate reasoning. As a result, single-hop accuracy slightly improves (e.g., PopQA rises from 49.4% to 50.1%, TriviaQA from 62.5% to 63.7% on 4B), but performance on multi-hop tasks drops sharply because coherent chain-of-thought across hops is essential (2WikiMultihopQA falls from 45.7% to 38.4%, MuSiQue from 14.0% to 11.1%). This trade-off reveals that over-regularization erodes the model’s capacity for multi-step reasoning, ultimately harming the very capability that agentic search relies on.

Taken together, these results highlight a fundamental tension in agentic distillation: too little distillation provides insufficient guidance, while too much forces the student into shallow, retrieval-heavy behaviors. strikes a practical balance, maintaining a sustained distillation signal throughout training without destabilizing the policy, thereby fostering a robust balance between efficient retrieval and rigorous reasoning.

Figure 3: The relationship between the OPSD loss weight and Avg success rate (%). balances distillation strength and policy stability, achieving the best Avg on both model scales.

Figure 4: Performance with three top-tier proprietary teacher backbones. All configurations use the same MAPD architecture and training settings without retuning the MAS pipeline.

Transfer Across Proprietary Teacher Backbones.

A key advantage of MAPD is transferability across different proprietary models. Because the structured JSON protocol successfully decouples the teacher’s core reasoning strategies from its surface stylistic artifacts, any sufficiently capable model can serve as the teacher backbone. To validate this flexibility, we evaluate whether the framework can seamlessly utilize protocols generated by three top-tier commercial models without requiring any architectural modifications or teacher-specific hyperparameter tuning—Claude-Opus-4.6, GPT-5.5, and Gemini-3.1-Pro. Results are shown in Table 3 and Figure 4. All three configurations produce positive observed gains over SDAR on both student scales: Claude-Opus-4.6 (39.4%/44.4%), GPT-5.5 (39.0%/44.6%), and Gemini-3.1-Pro (37.9%/44.0%). This narrow variance across teachers (within 2 points) confirms that our protocol successfully abstracts away teacher-specific idiosyncrasies, isolating semantic reasoning from the source model. Consequently, practitioners can seamlessly swap APIs to optimize cost-latency tradeoffs without retuning the pipeline.

Cost Analysis.

To quantify the overhead of protocol synthesis, we report the generation cost using Gemini-3.1-Pro as an illustrative example. The MAS pipeline processes 25,600 training instances, producing 25,584 valid protocols that pass the quality gate (99.94% yield). On average, the pipeline invokes the teacher LLM times per instance, consuming approximately 12.5K tokens per instance. The amortized cost is $0.057 per instance, yielding a total one-time generation cost of approximately $1,454 for the entire training set. Since all protocols are pre-synthesized and cached, this cost is incurred only once.

5 Related Work

LLM Reasoning and Agentic Search.

The paradigm of LLM reasoning has recently shifted from static, single-turn tasks toward dynamic, multi-turn agentic tasks that require sustained interaction with external environments (liu2026amo; liu2026general365; dong2025agentic2; mai2025agent; lv2026physagent; jiang2026beyond; jiang2025kore; jiang2026mined). As a critical extension of this reasoning paradigm, agentic search interleaves multi-step logical deduction with environment-augmented retrieval to tackle knowledge-intensive tasks. Recent frameworks have formalized this as an RL optimization problem. For instance, WebRL (qi2025webrl) employs self-evolving RL with process-level rewards, Search-R1 (jin2025search) leverages classic RL algorithms for multi-turn retrieval, and RAGEN (wang2025ragen) optimizes retrieval-augmented reasoning chains via trajectory-based rewards. However, these policies remain predominantly driven by sparse, outcome-only signals (e.g., terminal answer accuracy), which fundamentally struggle to provide granular credit assignment for intermediate planning, searching, and diagnostic steps. To resolve this optimization bottleneck, our MAPD framework introduces a joint training paradigm that integrates outcome-driven RL signal with a dense distillation signal.

Knowledge Distillation from Proprietary Models.

Transferring capabilities from proprietary LLMs to smaller open-source models has become a dominant paradigm. Classical knowledge distillation (hinton2015distilling) relies on matching output probability distributions, a mechanism fundamentally precluded by the opaque nature of black-box APIs and the mismatch between heterogeneous tokenizers. Consequently, contemporary approaches resort to text-level imitation, distilling the natural language reasoning traces or Chain-of-Thought (CoT) generated by proprietary teachers (hsieh2023distilling; mukherjee2023orca; xu2025magpie). However, as highlighted by recent studies (gudibande2023false), directly mimicking unconstrained teacher trajectories forces the student to overfit to the teacher’s verbose linguistic patterns and authoritative tone. This capacity mismatch induces severe style drift and exacerbates hallucinations. Our MAPD framework extracts the teacher’s semantic strategies into a style-normalized structured JSON protocol to decouple the core reasoning logic from the proprietary linguistic shell, enabling safe online self-distillation.

Multi-Agent System for Complex Reasoning.

LLM-based MAS have achieved remarkable success in complex reasoning and open-ended problem-solving (wu2023autogen; hong2024metagpt; lei2024macm). In retrieval-augmented scenarios, multi-agent collaboration can significantly enhance recall on multi-step questions through parallel sub-query orchestration and iterative debate (du2023improving; talebirad2023multi; wang2026mad). However, deploying these collaborative swarms at inference time incurs prohibitive computational overhead, severe latency, and massive API costs, rendering them highly impractical for real-time applications. Instead of relying on MAS during test-time interactions, MAPD utilizes the agent swarm exclusively as an offline training-time knowledge synthesizer, distilling the swarm’s collaborative problem-solving strategies into a single student model via our structured protocol.

6 Conclusion

In this paper, we introduce Multi-Agent Protocol Distillation (MAPD), a novel joint distillation and RL framework designed to bridge the heterogeneous distribution gap between proprietary teachers and open-source students in agentic search. To overcome the severe style drift and verbosity collapse inherent in raw trajectory imitation, MAPD elegantly decouples core cognitive strategies from idiosyncratic linguistic shells via a style-normalized, structured JSON protocol. Synthesized exclusively offline by a robust multi-agent system, this protocol serves as privileged information to provide dense guidance that seamlessly complements sparse outcome-driven RL without incurring any inference overhead. Extensive evaluations across seven knowledge-intensive QA benchmarks demonstrate that MAPD not only consistently outperforms all evaluated baselines, but also generalizes seamlessly across different proprietary teachers without retuning. By isolating semantic reasoning from idiosyncratic stylistic artifacts, MAPD unlocks a safer and more efficient pathway for heterogenes models alignment. We expect that the structural decoupling pioneered by MAPD will serve as a catalyst for future research, inspiring the community to push beyond the reliance on proprietary models and toward the development of fully autonomous, self-improving agentic frameworks.

References

Appendix A Structured Protocol Example

Example A, Example A and Example A show three complete structured protocols produced by our MAS pipeline with different teacher backbones (Claude-Opus-4.6, GPT-5.5 and Gemini-3.1-Pro, respectively).

Appendix B Training Dynamics

To provide a comprehensive overview of the optimization process, Figures 5 and 6 illustrate the detailed training dynamics of MAPD. These visualizations track the learning trajectories across different model scales, while systematically highlighting how varying the distillation weight, , influences the overall convergence and performance evolution(jiang2026large; jia2026benchmarking; jiang2025mmke; qi2025context; Fu2026CanML; Fu2026MMKUBenchAM).

Claude

GPT

Gemini

Figure 5: Training success rate of Qwen3-1.7B under varying distillation weight with different proprietary teacher models.

Claude

GPT

Gemini

Figure 6: Training success rate of Qwen3-4B under varying distillation weight with different proprietary teacher models.