摘要
我们提出 Darwin Family 框架，这是一种通过无梯度权重空间重组实现大语言模型免训练进化式合并的方法。我们探究的是：能否在不进行额外训练的情况下，通过重组已有检查点中编码的潜在能力，来提升前沿水平的推理性能。
Darwin 引入了三个关键思想：(i) 一个 14 维自适应合并基因组，支持细粒度的组件级和模块级重组；(ii) MRI-Trust Fusion，通过一个可学习的信任参数，自适应地平衡诊断性层重要性信号与进化搜索；(iii) 一个架构映射器，能够实现异构模型族之间的跨架构杂交。
实验表明，旗舰模型 Darwin-27B-Opus 在 GPQA Diamond 上达到 86.9% 的准确率，在 1252 个评估模型中排名第 6，并且在不进行任何基于梯度的训练的情况下，超越了其经过完整训练的基座模型。在 4B 到 35B 参数的不同规模上，Darwin 模型始终优于其父代模型，支持递归的多代进化，并实现了结合 Transformer 和 Mamba 组件的免训练进化式合并。总而言之，Darwin Family 证明了，对于以推理为中心的语言模型，诊断引导的进化式合并是昂贵后训练流程的一种实用且可复现的替代方案。
1 引言
近期的大语言模型展现了强大的推理性能，但实现这种能力在很大程度上依赖于昂贵的后训练流程，包括指令微调、强化学习和大规模知识蒸馏。虽然这些方法有效，但它们需要大量的计算资源，并且通常难以在不同场景下复现或适配。越来越多的证据表明，推理能力并非由后训练过程统一塑造。
多项研究表明，监督学习和指令微调能够提升任务层面的准确率，但同时会降低推理的忠实度、鲁棒性或迁移能力，尤其是在思维链场景下（wei2022cot；kojima2022zeroshot；wang2023selfconsistency）。关于基于提示词的推理的相关工作进一步指出，通常无需修改模型参数即可激发推理能力，这表明核心推理机制主要是在预训练阶段形成的（wei2022cot；zhou2023least）。对内部表征层面的分析为这一观点提供了佐证。逐层探针分析和结构诊断一致表明，不同的语言和推理功能在模型深度上分布不均，与推理相关的关键计算集中在预训练阶段形成的少数层中，并且在后续训练或微调后保持相对稳定（tenney2019bert；ethayarajh2019contextual；hewitt2019structural）。
近期更深入的诊断和因果分析进一步证实，神经网络中的功能重要性既具有局部性，又受到结构约束，这促使人们倾向于选择性干预而非统一的参数修改（bau2020neurons；geiger2021causal）。综合来看，这些发现表明，后续训练主要重组的是表层行为，而非重塑底层的推理回路。这些观察结果引出了一个根本性问题：能否在不进行进一步训练的情况下，通过重组预训练检查点中已编码的潜在能力来提升推理性能？
模型融合提供了一种有前景的免训练替代方案，它直接在权重空间中组合专用模型。早期方法依赖于静态启发式策略，例如权重平均或固定线性组合，并因其简单性而被广泛使用（wortsman2022soups; ilharco2023task）。然而，这些方法常常受到任务干扰的影响，因为它们将所有参数视为可统一合并，而忽略了专用模型之间存在的显著表征差异（yadav2023ties）。最近的研究通过选择性参数组合和稀疏化推进了免训练模型融合，证明了有原则的约束可以在不依赖基于梯度的训练的情况下显著提升融合后的性能（xu2024trainingfree）。
进化方法进一步自动化了有效融合配置的发现过程，实现了在融合空间上的无梯度优化（akiba2024evolutionary; akiba2025nature）。尽管如此，现有的大多数方法在诊断方面仍是盲目的，这促使了需要开发诊断引导的、自适应的免训练融合策略。
2 相关工作
2.1 大语言模型中的知识与推理
近期研究日益表明，在大语言模型中，知识获取与推理能力是部分解耦的。虽然指令微调和对齐过程通常能提升最终答案的准确性，但它们并不能可靠地提升多步推理的保真度，反而可能在结构化推理场景（尤其是在思维链场景）中降低鲁棒性或迁移能力（wei2022cot; kojima2022zeroshot; wang2023selfconsistency）。相比之下，基于提示词的方法，如思维链、从少到多提示和自一致性，表明推理通常可以在推理阶段通过提示词激发，而无需修改模型参数，这暗示核心推理机制主要是在预训练期间形成的（wei2022cot; zhou2023least）。这一观点推动了重新组织或重组现有表征的方法，而非依赖额外的训练。
2.2 诊断探针与功能分析
一系列长期的探针研究表明，Transformer 模型的不同层编码了不同的语言和推理相关功能。早期工作显示，预训练语言模型在不同层上恢复了一个经典的 NLP 处理流程，其中句法、语义和上下文抽象在不同深度涌现。后续研究揭示，功能重要性分布不均，这促使人们采用分层感知和组件特定的诊断方法，而非统一的参数启发式策略。更近期的研究通过识别局部的因果区域和神经元（对这些区域和神经元的操控会显著影响模型行为）来扩展这一视角，强化了神经网络中功能相关性既具有局部性又受结构约束的观点。多语言探针研究进一步表明，这种结构特化在不同语言间具有普遍性，支持将诊断探针作为指导模型重组的原则性先验知识。
2.3 免训练与静态模型融合
静态模型融合使用固定系数（例如权重平均或任务算术）来组合预训练或微调后的模型。虽然这种方法对紧密对齐的模型有效，但在融合异构专家模型时，由于表征不兼容和干扰，常常会降低性能。最近的进展通过引入带有结构化稀疏化、选择性参数对齐或双空间约束的免训练融合方法来解决这些限制，证明了原则性的参数选择可以在无需基于梯度的训练的情况下显著提升融合后的性能。这些工作将免训练模型融合确立为昂贵多任务训练流程的可行替代方案，同时强调了结构和表征方面考虑的重要性。
2.4 进化式模型融合
进化优化为在黑箱、无梯度环境下探索模型合并配置提供了一种自然框架。神经进化领域的经典研究表明，进化策略能够在无需梯度信息的情况下有效优化高维神经架构，这推动了其在大型预训练模型中的应用。近期研究进一步显示，进化搜索可以自动发现性能优于人工设计启发式方法的高效模型合并方案，从而验证了该方法在模型合并领域的适用性（akiba2024evolutionary；akiba2025nature）。然而，现有大多数方法在诊断层面仍存在盲区，这凸显了对诊断引导型、自适应且免训练的合并策略的需求。
2.5 跨架构与混合模型
近期架构发展探索了将基于注意力的 Transformer 架构与状态空间模型等替代序列建模机制相结合的混合模型，以提升效率与长上下文性能。这类混合架构表明，互补的归纳偏置能够成功整合于单一模型之中，这激发了超越传统微调的跨架构重组思路。此类进展为免训练的跨架构合并提供了架构层面的先例，支持了在配备适当对齐与选择机制的前提下重组异构模型组件的可行性。
3 Darwin 框架
图 1 展示了 Darwin 框架的高层概览，其核心设计原则是将诊断引导与进化探索解耦，并通过显式融合机制将二者协调统一。Darwin 不执行基于梯度的训练，而是完全在权重空间中运作，通过结构感知的合并决策对冻结的父模型检查点进行重组。
从高层次来看，Darwin 的流程如下。首先，模型层响应重要性（MRI）利用静态统计数据和基于轻量级探测器的响应来估计单个参数张量的功能相关性；同时，一个低维基因组编码了通过进化搜索探索的候选合并配置。这些信号通过 MRI-信任融合（MRI-Trust Fusion）进行组合，以确定它们的相对影响，从而产生逐张量的合并比率，并由一个免训练的合并内核应用这些比率来构建最终的合并模型。我们现在对这一过程进行形式化描述，从问题表述和参数分解开始。
3.1 问题表述
设两个父模型共享一个共同的预训练基座模型。它们的参数被分解为
| (1) |
其中，和分别代表由任务专业化或知识蒸馏引入的模型特定偏差。我们的目标是构建一个合并模型，该模型能够在不进行任何基于梯度的训练的情况下，仅通过在权重空间中重新组合和来提升推理性能。Darwin 并非对所有参数一视同仁，而是分配逐张量的合并比率，并通过一个诊断引导的进化过程来优化这些比率。
3.2 合并内核与参数重组
每个表示一个标量混合系数，该系数在张量的所有元素间共享。Darwin 构建合并张量的方式为
| (2) |
其中表示共享的预训练基座。这种公式化方法使得无需任何基于梯度的优化即可实现父参数的选择性重组。
3.3 模型层响应重要性（MRI）
Darwin 引入了模型层响应重要性（MRI）作为一种诊断先验，用于估计单个参数张量在推理行为方面的功能相关性。对于一个张量，MRI 结合了静态张量统计数据和基于探测器的功能响应：
| (3) | ||||
| (4) |
静态项聚合了归一化熵、方差和带上限的范数统计量，而探针项则测量由小型校准集引发的、基于推理条件的激活与通用激活之间的余弦距离。权重参数控制静态诊断与基于探针的诊断的相对贡献，并在所有实验中固定不变。MRI 导出的比率作为软先验而非固定合并规则，随后通过 MRI-Trust Fusion 与基因组导出的比率融合。
3.4 架构感知的张量对齐
对于异构的父架构，Darwin 应用了一个架构映射器，该映射器在进行数值重组之前建立张量级别的对应关系。对于一对候选张量，映射器计算一个兼容性分数
| (5) |
其中 表示功能角色对应关系， 衡量维度一致性， 捕捉参数形状相似性。系数 、 和 是固定的启发式权重。层对应关系通过受最小兼容性阈值约束的贪心匹配建立，从而无需重新训练即可实现有限的跨架构重组。
3.5 MRI-Trust Fusion 与基于基因组的控制
一个关键的设计问题是，合并应该在多大程度上依赖诊断方法，又在多大程度上依赖进化探索。Darwin 通过一个标量参数 来解决这个问题，该参数控制 MRI 信任度。最终的逐张量合并比率定义为
的中间值允许进化优化在保留结构化先验的同时，纠正诊断噪声。
3.6 基因组与进化优化。Darwin 中的每个合并策略都由一个 14 维的基因组表示
该基因组控制全局合并平衡、组件级混合比率、稀疏化密度、块级专业化系数、MRI 信任度以及合并核的插值行为。评估一个候选基因组需要实例化一个合并后的模型并测量其推理性能，这使得直接的进化搜索代价高昂。为应对这一挑战，Darwin 采用了一种两阶段优化策略，将结构筛选与经验评估分离开来。
4 实验与分析
4.1 实验设置
我们将 Darwin 评估为一种免训练推理增强框架，主要重点放在旗舰模型 Darwin-27B-Opus 上，并辅以实验来评估其在规模、生成方式和架构上的泛化能力。父模型尽可能选择共享相同预训练基座的模型，遵循同源模型合并的标准做法。
我们的主要评测基准是 GPQA Diamond，这是一个研究生级别的多项选择基准，专注于在标准化推理设置下评估稳健的科学推理能力。为了评估更广泛的推理泛化能力，我们还额外在 ARC-Challenge（侧重于多步符号推理和常识推理）和 MMLU（衡量跨多个学术领域的大规模多任务语言理解能力）上进行评测。
我们对比的对象包括：(i) 各个父模型；(ii) 静态免训练合并基线方法，例如均匀平均合并和 TIES 风格合并；(iii) 无诊断指导的进化合并方法。所有结果均在相同的推理设置下，对多次随机解码运行的结果取平均值，以确保公平比较。
4.2 主要结果：Darwin‑27B‑Opus（核心证据）
这一旗舰结果为 Darwin 的核心主张提供了主要验证。表 1 报告了 Darwin-27B-Opus 在 GPQA Diamond 和 ARC-Challenge 上的主要推理结果，同时列出了其父模型和代表性基线方法的结果。
Darwin-27B-Opus 在 GPQA Diamond 上达到了 86.9% 的准确率，在 1,252 个已评估模型中排名第 6（截至 2026-04-22），并且在不依赖任何基于梯度的训练的情况下，超越了其最强的父模型。值得注意的是，Darwin 在仅需少量 GPU 小时进行进化搜索的条件下，超越了多个规模更大、经过完整训练的模型。这些结果表明，仅通过权重空间重组，即可恢复甚至提升前沿水平的推理性能。
与静态合并方法相比，Darwin 始终表现出更高的准确率和更低的方差，表明其对表征干扰具有更强的鲁棒性。与无诊断机制的进化合并方法（real2019regularized；such2017deep；akiba2024evolutionary；akiba2025nature）相比，Darwin 实现了更高的峰值性能和更可靠的收敛，这表明诊断引导在有效探索合并空间中起着关键作用。
我们进一步分析了不同合并核的影响。线性插值带来了适度的改进，但容易受到任务干扰的影响。SLERP 在早期探索阶段提供了更平滑的插值，但始终获得较低的峰值准确率。相比之下，DARE-TIES 在所有配置下都实现了卓越的性能。其丢弃与重新缩放机制有效减轻了父模型之间的破坏性干扰，验证了其被选为 Darwin 框架中主要合并核的合理性。
| 基准测试 | 父模型（基础版） | 母模型 | ||
|---|---|---|---|---|
| （推理蒸馏版） | 简单平均 | |||
| / SLERP | Darwin-27B-Opus | |||
| GPQA-Diamond | 0.855 | 0.862 | 0.861 | 0.869 |
| ARC-Challenge | 0.710 | 0.740 | 0.750 | 0.779 |
| CommonsenseQA | 0.770 | 0.776 | 0.778 | 0.783 |
| TruthfulQA | 0.772 | 0.775 | 0.776 | 0.778 |
| HellaSwag | 0.858 | 0.864 | 0.866 | 0.870 |
| RACE | 0.821 | 0.825 | 0.828 | 0.831 |
| MMLU | 0.754 | 0.782 | 0.768 | 0.776 |
| Natural Questions | 0.748 | 0.753 | 0.756 | 0.760 |
| TriviaQA | 0.711 | 0.718 | 0.719 | 0.722 |
| 总体平均 | 0.767 | 0.776 | 0.775 | 0.786 0.040 |
4.3 学习到的基因组与合并动态分析
接下来，我们分析 Darwin 性能提升背后的机制，重点关注 MRI-Trust 融合、合并核选择以及基因组结构。首先，学习到的信任参数始终收敛到中间值（跨不同规模），这表明仅靠纯诊断规则或不受约束的进化搜索都不足以达到最佳效果。相反，Darwin 受益于一种自适应平衡，其中诊断先验引导搜索方向，而进化优化则弥补诊断噪声和层间交互带来的不足。
其次，我们比较了多种合并内核，发现 DARE-TIES 始终优于线性插值和 SLERP。虽然 SLERP 在搜索早期能提供更平滑的探索，但其峰值准确率较低。DARE-TIES 通过丢弃与缩放机制，有效缓解了父模型之间的破坏性干扰，使其特别适用于异构或高度专业化的父模型。
最后，对进化基因组的分析揭示了稳定的结构模式，包括注意力模块的选择性保留以及前馈组件中更强的重组。这些模式在独立运行和不同模型规模中反复出现，表明 Darwin 发现的是架构上的规律性，而非利用单一模型独有的特性。
4.4 消融实验
为了分离 MRI-Trust 机制的贡献，我们在 Darwin-27B-Opus 配置上进行了三路消融实验，仅改变融合方式，同时保持所有其他基因组参数不变。
消融实验揭示了两项关键发现。不同设置下的消融实验结果汇总于表 2，该表比较了仅基因组合并、基于静态 MRI 的合并、固定变体以及完整的自适应 Darwin 配置。首先，MRI 作为信号带来了明显的性能提升：使用基于静态 MRI 的合并相比仅基因组合并，GPQA 准确率提升了 pp。其次，自适应学习信任参数进一步提升了性能：进化出的变体相比固定设置额外获得了 pp 的提升。总体而言，完整的自适应变体在 GPQA 上相比无 MRI 基线提升了 pp，表明 MRI-Trust Fusion 是所观察到的推理能力提升的主要贡献者。
| 配置 | 设置 | GPQA Diamond | CLIcK | 与完整版对比 |
|---|---|---|---|---|
| 无 MRI（仅基因组） | （固定） | 84.4 | 69.2 | / |
| 仅 MRI（静态合并启发式） | （固定） | 85.6 | 72.4 | / |
| 固定 0.7（如 V5 中） | （固定） | 86.0 | 73.7 | / |
| 完整版 Darwin V6（自适应） | 86.9 | 75.3 | 基线 |
4.5 旗舰模型之外的泛化能力
尽管 Darwin-27B-Opus 为该框架提供了主要的实证验证，但我们观察到，相同的进化原理能够跨模型规模、代际以及父代构成进行泛化。在所有测试的规模（4B–35B）中，独立进化的 Darwin 模型始终收敛于中等程度的 MRI 信任值，并展现出非对称的重组模式，即注意力组件得到更强的保留，而前馈层则发生更激进的重组。
这些结构规律在独立进化的模型中保持稳定，包括递归的第二代合并模型以及混合架构变体，这表明 Darwin 发现的是尺度不变的合并原理，而非利用单一模型配置所独有的特性。详细的逐模型结果和基因组值见附录 B.2 和表 B.1，完整的模型族概览见附录 B.6。该框架还支持跨架构重组。
该框架还支持跨架构重组。Darwin-4B-Genesis 成功地将基于 Transformer 的注意力机制与 Mamba 风格的状态空间前馈组件进行了合并，且无需任何重新训练，在定向推理基准测试上超越了其两个父模型。这个案例表明，Darwin 能够跨异构架构重组互补的归纳偏置，其应用范围超越了同一模型族的微调变体。总的来说，这些模型并非确立 Darwin 有效性的必要条件（该有效性已由 Darwin-27B-Opus 单独验证）。相反，它们提供了支持性证据，表明相同的、由诊断引导的进化原理能够超越单一的旗舰实例，在模型规模、进化深度和架构多样性上实现泛化。我们强调，跨架构的结果是作为可扩展性的支持性证据，而非主要的性能驱动因素，旗舰验证由同源合并承担。详细的逐模型结果和模型族级别比较见附录 B.6。
5 局限性与未来工作
对父模型能力的依赖。Darwin 通过重组预训练期间获得的潜在能力来改进其父模型，但它并非凭空创造新能力。如果两个父模型都缺乏某项特定技能或知识领域，仅靠进化融合无法恢复该能力。
架构与对齐约束。目前，高性能的 Darwin 模型要求其父模型共享相同的预训练基础。虽然通过架构感知对齐可以实现有限的跨架构重组，但大规模、通用的跨基础模型融合仍然是一个待解决的挑战。
搜索成本与验证范围。尽管比训练或微调便宜得多，但 Darwin 的进化搜索并非零成本，需要运行一组精简的评估。此外，虽然中等规模的模型已在公开排行榜上得到独立验证，但最大规模变体的验证工作仍在进行中。
未来工作。有前景的方向包括：利用分片评估将 Darwin 扩展到 100B 参数规模，改进跨基础模型的对齐机制，以及将 Darwin 与互补的测试时或推理时干预手段相结合。
6 结论
我们提出了 Darwin 框架以及 Darwin 系列——八个通过进化融合得到的语言模型，参数规模从 4B 到 35B 不等。我们的主要贡献包括：14 维自适应基因组（第 3.6 节）、带有可学习参数的 MRI-Trust 融合公式（第 3.5 节），以及实现跨架构繁殖的架构映射器（第 3.4 节）。主要案例研究 Darwin-27B-Opus 在 GPQA Diamond 排行榜上正式排名第 6，在性能上超越了其父模型 Qwen3.5-27B 以及其他前沿模型。
Darwin 系列证明了免训练的进化融合并非一种用于模型集成平均的偏门技术，而是一条实用且可复现的路径，能够以比传统预训练低三到六个数量级的计算成本，实现前沿水平的推理能力。通过以 Apache 2.0 许可证发布所有模型、V6 代码库以及 MRI 工具，我们希望促成广泛的独立验证，并推动一个关于原则性、诊断引导的权重空间优化的新研究方向。
参考文献
- [1] Jason Wei、Xuezhi Wang、Dale Schuurmans 等人。思维链提示激发大语言模型的推理能力。发表于《神经信息处理系统进展》，2022 年。
- [2] Takeshi Kojima、Shixiang Gu、M. Reid 等人。大语言模型是零样本推理者。发表于《神经信息处理系统》，2022 年。
- [3] Xuezhi Wang、Jason Wei、Dale Schuurmans 等人。自一致性改进了语言模型中的思维链推理。发表于《国际学习表征会议》，2023 年。
- [4] Denny Zhou、Natalie Schärli、Le Hou 等人。从少到多提示使大语言模型能够进行复杂推理。发表于《国际学习表征会议》，2023 年。
- [5] Ian Tenney、Dipanjan Das 和 Ellie Pavlick。BERT 重新发现了经典 NLP 流水线。发表于《计算语言学协会》，2019 年。
- [6] Kawin Ethayarajh。上下文词表示到底有多上下文相关？发表于 EMNLP-IJCNLP，2019 年。
- [7] John Hewitt 和 Christopher D. Manning。一种用于在词表示中发现句法结构的结构探针。发表于 NAACL，2019 年。
- [8] David Bau、Jun-Yan Zhu、Hendrik Strobelt 等人。识别并控制神经网络中的重要神经元。发表于《国际学习表征会议》，2020 年。
- [9] Atticus Geiger、Zhiwei Wu、David Lu 等人。神经网络的因果抽象。发表于《神经信息处理系统》，2021 年。
- [10] Mitchell Wortsman、Gabriel Ilharco、Samir Y. Gadre 等人。模型汤：对多个微调模型的权重进行平均。发表于《国际机器学习会议》，2022 年。
- [11] Gabriel Ilharco、Marco Tulio Ribeiro、Mitchell Wortsman 等人。通过任务算术编辑模型。发表于《国际学习表征会议》，2023 年。
- [12] Pratyush Yadav、Derek Tam、Leshem Choshen 等人。TIES-Merging：解决模型合并时的干扰问题。发表于《神经信息处理系统》，2023 年。
- [13] Zhen Xu、Kai Yuan、Hao Wang 等人。无需训练的预训练模型合并。发表于《IEEE 计算机视觉与模式识别会议》，2024 年。
- [14] Takuya Akiba、Makoto Shing、Yu Tang、Qi Sun 和 David Ha。模型合并配方的进化优化。arXiv 预印本 arXiv:2403.13187，2024 年。
- [15] Takuya Akiba、Makoto Shing、Yu Tang、Qi Sun 与 David Ha。模型合并配方的进化优化。《自然·机器智能》，2025年。
- [16] Anna Rogers、Olga Kovaleva 与 Anna Rumshisky。BERT学入门。《ACL汇刊》，2020年。
- [17] Dehua Li、Haoyan Zhao、Qing Zeng 与 Mengnan Du。大语言模型中的多语言探针研究。arXiv预印本 arXiv:2409.14459，2024年。
- [18] David Rein、Benjamin L. Hou、Asa Cooper Stickland 等。GPQA：一项研究生级别的防谷歌问答基准。arXiv预印本 arXiv:2311.12022，2023年。
- [19] Peter Clark、Brian Cowhey、Oren Etzioni 等。你以为问答问题已经解决了？试试ARC。arXiv预印本 arXiv:1803.05457，2018年。
- [20] Dan Hendrycks、Collin Burns、Steven Basart 等。衡量大规模多任务语言理解能力。国际学习表征会议，2021年。
- [21] Esteban Real、Alok Aggarwal、Yanping Huang 与 Quoc V. Le。面向图像分类器架构搜索的正则化进化。AAAI人工智能会议，2019年。
- [22] Felipe Petroski Such 等。深度神经进化：遗传算法是训练深度神经网络的一种有竞争力的替代方案。arXiv预印本 arXiv:1712.06567，2017年。
- [23] Lei Yu、Bowen Yu、Hongyi Yu、Fei Huang 与 Yongbin Li。语言模型是超级马里奥：将同源模型的能力吸收为免费午餐。国际机器学习会议，2024年。
- [24] Michael Matena 与 Colin Raffel。基于Fisher信息的模型合并。arXiv预印本 arXiv:2210.07289，2022年。
- [25] Mohammad Reza Davari 与 Eugene Belilovsky。模型面包屑：利用稀疏掩码扩展多任务模型合并。欧洲计算机视觉会议，2024年。
- [26] Yuki Kuroki、Yu Zhang 与 Risto Miikkulainen。CycleQD：通过循环进化动力学实现质量-多样性优化。国际学习表征会议，2025年。
- [27] Nuno Abrantes、Miguel Lourenço 与 João Monteiro。M2N2：具有自适应网络组合的模块化神经进化。遗传与进化计算会议，2025年。
- [28] 徐震、袁凯、王浩等。基于双空间约束的无训练模型融合。IEEE 计算机视觉与模式识别会议，2024年。
- [29] 李文静、高浩、高明桥等。面向多目标域自适应的无训练模型融合。欧洲计算机视觉会议，2024年。
- [30] Kenneth O. Stanley 与 Risto Miikkulainen。通过增强拓扑结构进化神经网络。《进化计算》，2002年。
- [31] 杨毅、沈力、郭光远等。大语言模型、多模态大语言模型及更广范围内的模型融合：方法、理论与应用。《ACM 计算调查》，2026年。
附录 A 可复现性
A.1 数据与可复现性站点
模型集合：huggingface.co/collections/FINAL-Bench/darwin-family。从 Claude Opus 4.6 进行知识蒸馏，是指使用通过公开 Claude Opus 4.6 API 生成的推理轨迹，对开放权重的基座模型进行监督式微调。未使用或分发任何专有的 Claude 模型权重。本工作中使用的所有供体模型均为公开可用的社区发布版本，且该蒸馏流程本身在能够访问 Claude API 的前提下完全可复现。
交互式演示 / 进化工作室：huggingface.co/spaces/VIDraft/DARWIN-Evolution。
Darwin V6 代码库：约 13,771 行代码，分布于 15 个 Python 文件中，包括 mri_extractor.py、mergekit_integration.py、parent_attribution.py、calibration_data.py、benchmarks.py、live_engine.py 和 live_blend.py。所有代码均以 Apache 2.0 许可证发布。
GPQA Diamond 验证：huggingface.co/datasets/Idavidrein/gpqa（截至 2026-04-22，Darwin-27B-Opus 排名第 6，Darwin-31B-Opus 排名第 11）。
社区量化版本：bartowski/Darwin-27B-Opus-GGUF、bartowski/Darwin-31B-Opus-GGUF、bartowski/Darwin-35B-A3B-Opus-GGUF、mradermacher/Darwin-27B-Opus-i1-GGUF 以及其他众多版本。
A.2 超参数与硬件
所有实验均在 NVIDIA A100 或 H100 GPU 上进行。运行时间大致随模型规模线性增长，4B 模型约需 1 小时，35B 模型约需 5 小时。
CMA-ES 种群规模：50。
进化代数（第一阶段）：20。
进化代数（第二阶段）：5–10（取决于模型规模）。
变异标准差（初始值）：0.01。
每代突变衰减率：0.95。
精英保留：每代前5名。
交叉：基因组空间中的SLERP。
随机种子：每个问题固定，MD5()取模。
评估运行：每个候选方案。
采样温度：1.0（David使用maj@8）。
采样top-：0.95。
采样top-：64。
硬件（每个模型）：
Darwin-4B-Opus：A100-80GB，约1小时。
Darwin-4B-David：H100-80GB，约1小时。
Darwin-4B-Genesis：H100-80GB，155分钟。
Darwin-9B-Opus：H100-80GB，约90分钟。
Darwin-27B-Opus：H100-80GB，约5小时。
Darwin-31B-Opus：H100-80GB，约134分钟。
Darwin-35B-A3B-Opus：H100-80GB，约5小时。
A.3 校准探针集
MRI校准探针集包含123个样本，涵盖六个类别，韩语与英语比例约为50:50，遵循先前的探针与诊断分析实践[5, 6, 7, 16, 8, 9, 17]。所有探针样本均可在公开的Darwin V6代码库中以calibration_data.py文件形式获取。通过前向钩子在每个Transformer层输出处计算探针条件隐藏状态；GENERIC类别作为基线锚点，用于第3.3.2节所述的余弦距离重要性测量。
| 类别 | 样本数 | 目的 |
|---|---|---|
| 推理 | 28 | 多步骤思维链（算术、逻辑演绎、条件推理） |
| 代码 | 22 | 编程任务（Python、算法综合、代码理解） |
| 逻辑 | 18 | 形式化演绎（三段论、范畴/假设推理） |
| 多语言_韩语 | 20 | 韩语理解、文化知识（源自CLIcK风格） |
| 多语言_英语 | 20 | 用于韩语差异的英语基线 |
| 通用 | 15 | 日常对话查询（余弦距离锚点的基线） |
| 总计 | 123 | 韩语:英语按字符计为50:50 |
附录B 基因组设计与进化优化
B.1 基于基因组的合并策略表示。
为清晰阐述概念，我们首先描述六个核心组件级参数，它们构成了第3.6节定义的完整14维基因组的一个子集。Darwin将合并策略编码为一个紧凑的六维基因组向量，在表达能力与计算可行性之间取得平衡：
— 全局权重比例，控制每个父模型的整体贡献程度；
— 注意力模块的层级特定权重；
— 前馈网络的层级特定权重；
— token 嵌入与解嵌入矩阵的权重；
— 伯努利稀疏率，控制每个父模型参数差值的丢弃与重新缩放力度。
不同的基因组图谱对应性质各异的合并策略。平衡型基因组（即各组件比例均匀）有利于通用融合；非对称型基因组（例如 ）适用于以推理为核心的任务，其中注意力模块承载了不成比例的信号；稀疏型基因组（低密度值）则能在任务干扰下提升稳定性。通过将 MRI 导出的层级敏感度映射到基因组初始化中，Darwin 确保初始搜索种群反映的是功能性的层级特化，而非任意启发式规则。
核心组件之一是使用基于 MRI 的层级敏感度探针来引导基因组初始化。通过分析各层上供体模型之间的参数差异，MRI 提供了结构化的信号，用于指导注意力模块、前馈网络、嵌入层及全局组件的初始合并比例。这种热启动策略确保进化搜索从基因组空间中一个有前景的区域开始，相比随机初始化，能加速收敛并提升合并质量。这六个参数构成了完整 14 维基因组的核心子集，其正式定义见附录 B.2。
B.2 完整 14 维基因组定义
每个 Darwin 合并操作由一个 14 维基因组控制，该基因组由组件级比率、稀疏化密度、块级系数和融合参数组成。表 4 总结了每个基因组参数的代表性值以及在不同模型规模演化过程中得到的经验稳定范围。
| 参数 | 4B-Opus | 27B-Opus | 31B-Opus | 典型范围 |
| global_ratio () | 0.5204 | 0.4893 | 0.4712 | |
| attn_ratio () | 0.3195 | 0.1463 | 0.1890 | |
| ffn_ratio () | 0.8421 | 0.8768 | 0.9204 | |
| embed_ratio () | 0.3508 | 0.3021 | 0.2894 | |
| density_a | 0.8934 | 0.8507 | 0.8625 | |
| density_b | 0.9011 | 0.9413 | 0.9228 | |
| mri_trust | 0.4907 | 0.5557 | 0.3631 | |
| merge_method_weight | 0.3124 | 0.2783 | 0.3502 |
B.3 参数。
Darwin 将合并策略编码为一个由三组参数组成的 14 维基因组向量。
核心参数（6 个）：核心组件级参数控制全局和模块特定的混合行为：（整体合并平衡，）；（注意力组件权重，）；（前馈组件权重，）；（token 和位置嵌入权重，）；以及 ，它们分别控制应用于父本和母本参数增量的伯努利稀疏化比率。
MRI 导出的块参数（6 个）：块级参数 到 为 MRI 识别出的六个连续层块分配独立的合并比率，从而捕捉网络深度上的粗粒度主导模式。
元演化参数（2 个）：参数 控制基于诊断的 MRI 比率与基因组驱动比率之间的插值，如第 3.5 节所定义。参数 ，在第 3.6 节中记为 ，控制 DARE-TIES 和 SLERP 合并内核之间的插值。
B.4 尺度不变与非对称基因组模式
在参数量从 4B 到 35B 不等的模型规模中，进化后的达尔文基因组展现出稳定且重复出现的参数范围，尤其是在注意力保留比率、前馈重组比率和 MRI 信任值方面。如表 B.1 所示，这些参数在独立训练的模型中集中于狭窄的区间内，表明进化过程发现的是尺度不变的结构规律，而非特定规模的产物。
一个显著的模式是注意力组件与前馈组件之间的系统性不对称。在所有测试的规模下，达尔文始终从基础（父本）模型中保留大部分注意力参数，同时激进地重组来自专门化（母本）模型的前馈层。这种不对称性与先前的探测和分析研究结果一致，这些研究表明注意力层主要负责信息路由和聚焦，而前馈网络则编码任务特定的计算与变换 [5, 6]。值得注意的是，这一模式在多次进化运行中一致出现，无法通过人工设计或均匀比率合并轻易预判。相反，它反映了由诊断引导的进化搜索所发现的架构规律，进一步印证了达尔文重组的是潜在的推理结构，而非引入临时的参数配置。
B.5 进化优化过程
达尔文采用两阶段进化优化策略，在限制完整模型实例化计算成本的同时，高效搜索合并基因组空间。
图 3 提供了这一进化过程的概览示意图，展示了适应度评估、选择、交叉和自适应突变的迭代循环，该循环推动收敛至一组紧凑的高质量基因组。
在第一阶段，候选合并基因组通过标准进化优化循环进行演化。每一代使用轻量级代理适应度分数评估候选基因组，随后通过交叉和自适应变异进行亲本选择和变异。该阶段旨在快速过滤结构上不合理或低质量的合并配置，而无需构建完整的合并模型。
在第二阶段，将第一阶段识别出的一小组高质量基因组实例化为合并模型，并直接在推理基准上进行评估。最终选择基于固定推理设置下的实证性能。
B.6 Darwin 系列概览：跨规模与世代
该表格总结了所有已发布的 Darwin 模型，并强调了中间 MRI-trust 值以及跨规模的非对称注意力/FFN 重组模式的重复出现。表 5 提供了 Darwin 系列在模型规模、进化世代和亲本组成方面的概览，报告了每个已发布变体的代表性基准性能及显著特性。有关详细的模型级配置和比较，请参阅表 5。
| 模型 | 世代 | 亲本（父本 母本） | GPQA% | 显著特性 |
|---|---|---|---|---|
| Darwin-4B-Opus | 1 | gemma-4-E4B Deckard | — | , 14-D 基因组 |
| Darwin-4B-David | 2 | Darwin-4B-Opus DECKARD-24B-D | 85.0 | 首次递归进化（+26.4pp maj@8） |
| Darwin-4B-Genesis | 3 | Darwin-4B-David Qwen3.5-4B | 60 | 首次跨架构 FFN（CLIcK 92%） |
| Darwin-9B-Opus | 1 | qwen3_5-10B base | — | 紧凑型 Qwen 变体 |
| Darwin-27B-Opus | 1 | Qwen3.5-27B Claude Opus 4.6 风格推理蒸馏变体（开放权重基础） | 86.9 | GPQA 官方第 6 名，主要案例 |
| Darwin-31B-Opus | 1 | gemma-4-31B TeichAI-distill | 85.9 | GPQA 官方第 11 名， |
| Darwin-35B-A3B-Opus | 1 | Qwen3.5-35B-A3B MoE Jackrong | 90.0∗ | 旗舰级 MoE，262K 上下文窗口，201 种语言 |
附录 C 架构映射器与合并内核（扩展版）
本附录提供了 Darwin 中用于在架构对齐后重新组合对齐参数张量的合并内核的扩展细节。架构映射器（Architecture Mapper）用于确定父模型之间的张量级对应关系，在第 3.4 节中介绍，并被视为一个结构性的预处理步骤。相比之下，附录 C 侧重于内核级操作，这些操作指定了在建立对应关系后如何组合对齐的张量。
C.1 角色分离：映射器与内核
为清晰起见，我们区分了跨模型重组中涉及的两个组件。架构映射器（第 3.4 节）通过在结构层面操作，基于类型、维度和位置兼容性建立张量对应关系，从而确定两个父模型中的哪些参数张量可以进行有意义的重组。映射器不修改参数值，也不执行任何数值组合。给定此对齐步骤识别出的匹配张量对后，合并内核（本附录中描述）指定如何组合它们的参数值。合并内核在参数层面操作，应用张量级的混合比率、稀疏化规则或插值方案来生成合并后的权重。这种分离使得 Darwin 能够清晰地将结构对齐与数值重组解耦。
C.2 进化优化过程（上下文）
Darwin 采用两阶段进化优化策略来搜索合并配置，无需基于梯度的更新。在第一阶段，使用轻量级代理目标筛选候选基因组，以排除退化或结构上不可行的配置。在第二阶段，将一小批有前景的基因组实例化为合并模型，并直接在推理基准上进行评估。下面描述的合并内核仅在架构映射器完成张量对齐以及通过 MRI-Trust 融合（第 3.5 节）完成比率选择之后才被调用。
C.3 DARE-TIES 合并内核
DARE-TIES（带任务区间消除的丢弃与重新缩放）是 Darwin 中用于最终模型构建的主要合并核。给定对齐的父张量和基因组指定的混合系数，该核通过计算相对于共享基模型的参数增量，并应用伯努利稀疏掩码来选择性地保留信息性组件来运行。
具体来说，对于共享共同基模型的父模型，Darwin 计算 和 ，应用基因组控制的伯努利掩码 和 作用于每个增量，重新缩放存活的条目以保留期望的幅度，然后执行加权重组，如下所示：
| (6) |
其中 表示基因组为组件 指定的混合权重。
这种丢弃与重新缩放的过程减轻了父模型之间的破坏性干扰，同时保留了互补的推理行为，并且经验观察表明，其性能比均匀平均或线性插值更稳定。因此，Darwin 优先使用 DARE-TIES 进行基准驱动的适应度评估。
DARE-TIES 合并过程的概述如图 4 所示，该图直观地总结了在重组之前应用于对齐父参数增量的丢弃与重新缩放操作。
C.4 SLERP 核（探索阶段）
SLERP（球面线性插值）在早期进化探索阶段被用作一种轻量级替代核。通过沿超球面路径对张量进行插值，SLERP 能够以较低的计算开销平滑地探索合并配置。然而，在第二阶段评估中，SLERP 的表现始终不如 DARE-TIES，因此不用于最终模型选择。
C.5 总结
总之，架构映射器（第 3.4 节）负责确定异构父模型之间的张量对应关系，而此处描述的合并内核则定义了组合这些张量的数值规则。因此，附录 C 通过详细说明对齐参数如何被合并（而非对齐如何建立），对正文进行了补充。
附录 D 与先前模型合并方法的比较
D.1 比较概述
本附录将 Darwin 置于更广泛的模型合并方法图景中，特别关注无需训练的方法。我们聚焦于假设、优化结构、诊断用途和可扩展性方面的差异，而非原始性能（原始性能已在正文中报告）。表 6 提供了 Darwin 与先前模型合并方法在关键维度上的比较概览，包括基因组维度、诊断指导、跨架构支持和多代能力。
Darwin 是目前调研过的先前方法中，唯一同时满足以下条件的方法：(a) 在两位数维度的基因组中运行，(b) 通过可学习参数将功能重要性诊断信号整合到合并内核中，(c) 支持跨架构繁殖，以及 (d) 已在多个进化世代中得到验证，并具有可遗传的增益。
| 方法 | 基因组维度 | 合并前诊断 | 跨架构繁殖 | 多代 |
|---|---|---|---|---|
| TIES-Merging [12] | — | 无 | 否 | 否 |
| DARE [23] | — | 稀疏化 | 否 | 否 |
| Model Soups [10] | — | 无 | 否 | 否 |
| Task Arithmetic [11] | — | 无 | 否 | 否 |
| Fisher Merging [24] | — | Fisher 信息 | 否 | 否 |
| Model Breadcrumbs [25] | — | 稀疏掩码 | 否 | 否 |
| Sakana EvoMerge [14, 15] | 每层 2 个 | 无 | 部分（DFS） | 否 |
| CycleQD [26] | MAP-Elites | 无 | 否 | 否 |
| M2N2 [27] | 可进化分割 | 无 | 否 | 否 |
| Darwin V5（2026 年，我们的） | 2（均匀） | MRI（前身） | 否 | 否 |
| Darwin V6（我们的） | 14 个自适应 | MRI 6 探针 | 通过映射器 | 是 |
| Darwin-4B-Genesis（V6+42D） | 每层 42 个 | MRI 6 探针 | 是（TF Mamba） | 第 3 代（遗传） |
D.2 静态与基于启发式的模型合并
早期的模型融合方法依赖于静态、低维的启发式规则，通过固定系数来组合预训练或微调后的模型。代表性例子包括均匀权重平均（Model Soups）[10] 和权重空间中的线性向量算术（Task Arithmetic）[11]。这些方法因其简单性和低计算成本而具有吸引力，并且在父模型在功能和训练历程上高度对齐时表现良好。
然而，静态启发式规则隐含地假设所有参数都是同等可融合的。正如后续研究所证明的，这一假设对于异构的专家模型往往不成立，会导致表征干扰和性能下降 [12]。TIES-merging 通过选择性修剪和重新缩放参数增量部分地解决了这个问题，但其选择规则仍然是人工设计的且与任务无关 [12]。因此，这些方法缺乏对父模型特定结构的适应性，并且难以泛化到关系不够紧密的模型之外。
D.3 带参数选择的结构化免训练融合
近期的工作通过引入结构化的参数选择和对齐约束，改进了免训练融合方法。Training-Free Pretrained Model Merging [13] 和 Dual-Space Constraint Merging [28] 显式地对权重空间和激活空间之间的一致性进行建模，证明了选择性对齐能够在无需梯度更新的情况下显著提升融合性能。关于多目标领域适应的相关工作进一步表明，当父模型共享一个共同的预训练主干网络时 [29]，原则性的免训练融合方法能够与基于数据共享的基线方法相媲美。
尽管这些方法相比静态启发式规则代表了显著的进步，但它们通常依赖于固定的选择规则或优化目标，这些规则或目标无法适应下游的推理行为。此外，它们通常局限于单次融合，并且无法自然地扩展到迭代或进化式的组合。
D.4 进化式模型融合
进化优化提供了一种互补的视角，它将模型合并视为一个黑盒搜索问题，在无需梯度信息的情况下优化合并配置。神经进化领域的经典研究表明，进化策略能够有效搜索高维神经参数空间[30, 21, 22]。在此基础之上，近期研究显示，进化搜索可以自动发现性能优异的模型合并方案，其效果优于人工设计的启发式方法[14, 15]。
然而，现有的进化合并方法在很大程度上存在诊断盲区。它们通常作用于低维或统一的合并参数，并将所有组件视为对称可变的，这导致了探索效率低下，且进化所得方案的可解释性有限。
D.5 Darwin 中的诊断引导式进化合并
Darwin 整合了结构化免训练合并与进化优化两方面的优势，同时弥补了各自的不足。与静态或基于规则的方法[10, 11, 12, 13, 28, 29]不同，Darwin 用显式的自适应基因组取代了固定的启发式方法，该基因组在多个结构层级上对合并行为进行参数化。与先前的进化方法[30, 21, 22, 14, 15]不同，Darwin 引入了用于评估功能相关性的诊断先验知识，从而使进化搜索能够聚焦于对推理能力至关重要的组件。
一个关键区别在于 MRI-Trust Fusion（磁共振成像信任融合），它通过一个可学习的信任参数自适应地平衡诊断引导与进化探索。这种设计使 Darwin 能够在启发式驱动合并与无约束搜索之间进行插值，而非固守任一极端。因此，Darwin 支持多代进化、跨架构合并以及稳健的推理能力提升，且无需基于梯度的训练。
D.6 总结性对比
总而言之，现有的模型合并方法在简洁性、结构性和灵活性之间需要权衡取舍。静态启发式方法简单但脆弱 [10, 11, 12]；结构化的免训练方法有理论依据但不够灵活 [13, 28, 29]；进化方法灵活但在缺乏引导时效率低下 [30, 21, 22, 14, 15]。Darwin 在该领域中占据了一个独特的位置，它在一个统一的框架内结合了免训练操作、诊断性选择能力和进化适应性。关于更广泛的模型合并领域的全面综述可参见 [31]。
附录 E 失败模式与负面结果
在将 Darwin 与先前的免训练模型合并方法（附录 D）进行对比定位之后，我们现在分析失败案例，以阐明诊断引导的进化合并方法的操作边界。
对未产生改进的父模型对的分析揭示了若干反复出现的失败模式，这些模式是结构性的而非偶然的。重要的是，这些案例并不与 Darwin 的有效性相矛盾，而是阐明了诊断引导的进化合并方法预期能够成功的条件。
缺乏互补性专长。当两个父模型展现出高度相似的能力和错误模式时，进化合并带来的益处有限。当两个父模型都没有贡献出独特或主导性的能力时，重组主要是在重新分配冗余结构，而非组合互补功能，从而导致改进微乎其微或毫无改进。
严重的表征对齐问题。一些未产生改进的合并涉及内部表征对齐较差的父模型，即使它们名义上源自相同的基础架构。在这种情况下，权重空间重组可能比进化优化能够恢复的速度更快地破坏推理关键路径，导致收益过早饱和。
诊断信号中的模糊性。Darwin 将 MRI 作为软性诊断先验而非真实标签指标。当诊断信号较弱、存在噪声或在各层之间不一致时——例如，当与推理相关的激活是分散而非局部的——MRI 的指导作用就会减弱。进化搜索可以部分补偿此类噪声，但由此带来的收益通常较小且不稳定。
搜索空间饱和。最后，某些父代组合已经接近目标推理基准的局部最优。在这些情况下，Darwin 的进化搜索会快速收敛，但进一步的改进受到缺乏潜在互补结构的制约，而非搜索效率低下。
综合来看，这些失败模式表明，Darwin 在应用于具有部分互补推理结构的异质但兼容的父模型时最为有效。因此，失败案例并非反例，而是作为边界条件，阐明了诊断引导进化合并的操作范围。
附录 F 资源与社区采用情况
所有 Darwin 系列模型、代码和 MRI 工具均以 Apache 2.0 许可证发布。作为社区采用的一个指标，已发布的 Darwin 模型在官方和社区维护的分发渠道中已积累了可观的下载量。表 7 总结了截至 2026-04-22 的累计下载次数，涵盖官方检查点以及流行的 GGUF 和第三方发布版本。关于按模型变体和分发渠道的详细分类，请读者参阅表 7。截至 2026 年 4 月，官方和量化分发渠道的社区下载量已超过 96,000 次。所有分发渠道的合并下载量超过 96,000 次，与主要实验室发布的开源推理模型的采用水平相当。大量的社区量化活动（bartowski, mradermacher）进一步表明，Darwin 系列模型不仅用于基准测试，而且正在被积极部署。
| 模型 | 官方 | bartowski GGUF | mradermacher + 其他 |
|---|---|---|---|
| Darwin-27B-Opus | 14,000 | 22,000 | 8,500 |
| Darwin-35B-A3B-Opus | 9,000 | 14,500 | 6,000 |
| Darwin-31B-Opus | 6,500 | 8,000 | 3,000 |
| Darwin-4B-David、-4B-Opus、-4B-Genesis、-9B-Opus（合并统计） | 3,500 | 1,200 | 400 |
| 系列总计 | 33,000 | 45,700 | 17,900 |
Abstract
We present Darwin Family, a framework for training-free evolutionary merging of large language models via gradient-free weight-space recombination. We ask whether frontier-level reasoning performance can be improved without additional training, by reorganizing latent capabilities already encoded in existing checkpoints.
Darwin introduces three key ideas: (i) a 14-dimensional adaptive merge genome enabling fine-grained component- and block-level recombination; (ii) MRI-Trust Fusion, which adaptively balances diagnostic layer-importance signals with evolutionary search through a learnable trust parameter; and (iii) an Architecture Mapper that enables cross-architecture breeding between heterogeneous model families.
Empirically, the flagship Darwin-27B-Opus achieves 86.9% on GPQA Diamond, ranking #6 among 1,252 evaluated models, and outperforming its fully trained foundation model without any gradient-based training. Across scales from 4B to 35B parameters, Darwin models consistently improve over their parents, support recursive multi-generation evolution, and enable a training-free evolutionary merge that combines Transformer- and Mamba-based components. Together, the Darwin Family demonstrates that diagnostic-guided evolutionary merging is a practical and reproducible alternative to costly post-training pipelines for reasoning-centric language models.
1 Introduction
Recent large language models (LLMs) demonstrate strong reasoning performance, but achieving such capability has largely depended on expensive post-training pipelines, including instruction tuning, reinforcement learning, and large-scale distillation. While effective, these procedures require substantial compute and are often difficult to reproduce or adapt across settings. A growing body of evidence suggests that reasoning ability is not uniformly shaped by post-training.
Multiple studies show that supervised and instruction tuning can improve task-level accuracy while degrading reasoning faithfulness, robustness, or transfer, particularly in chain-of-thought settings wei2022cot ; kojima2022zeroshot ; wang2023selfconsistency . Related work on prompting-based reasoning further indicates that reasoning can often be elicited without modifying model parameters, suggesting that core reasoning mechanisms are largely formed during pretraining wei2022cot ; zhou2023least . Analysis at the level of internal representations provides converging support for this view. Layer-wise probing and structural diagnostics consistently show that different linguistic and reasoning functions are unevenly distributed across depth, with reasoning-critical computation localized to a subset of layers established during pretraining and relatively invariant under post-training or fine-tuning tenney2019bert ; ethayarajh2019contextual ; hewitt2019structural .
More recent diagnostic and causal analyses reinforce the view that functional importance in neural networks is both localized and structurally constrained, motivating selective interventions over uniform parameter modification bau2020neurons ; geiger2021causal . Together, these findings suggest that post-training primarily reorganizes surface behavior rather than reshaping the underlying reasoning circuitry. These observations raise a fundamental question: can reasoning performance be improved without further training, by reorganizing latent capabilities already encoded in pretrained checkpoints?
Model merging offers a promising training-free alternative by directly combining specialized models in weight space. Early approaches rely on static heuristics such as weight averaging or fixed linear combinations and are widely used for their simplicity wortsman2022soups ; ilharco2023task . However, these methods often suffer from task interference, as they treat all parameters as uniformly mergeable despite substantial representational divergence between specialized models yadav2023ties . Recent work advances training-free model merging through selective parameter combination and sparsification, demonstrating that principled constraints can significantly improve merged performance without gradient-based training xu2024trainingfree .
Evolutionary approaches further automate the discovery of effective merge configurations, enabling gradient-free optimization over the merge space akiba2024evolutionary ; akiba2025nature . Nevertheless, most existing methods remain diagnostically blind, motivating the need for diagnostic-guided, adaptive training-free merging strategies.
2 Related Work
2.1 Knowledge versus Reasoning in LLMs
Recent studies increasingly indicate that knowledge acquisition and reasoning ability are partially decoupled in large language models. While instruction tuning and alignment procedures often improve final answer accuracy, they do not reliably improve multi-step reasoning fidelity and may degrade robustness or transfer in structured reasoning settings, particularly in chain-of-thought settings wei2022cot ; kojima2022zeroshot ; wang2023selfconsistency . In contrast, prompting-based approaches such as chain-of-thought, least-to-most prompting, and self-consistency demonstrate that reasoning can often be elicited at inference time without modifying model parameters, suggesting that core reasoning mechanisms are largely formed during pretraining wei2022cot ; zhou2023least . This perspective motivates approaches that reorganize or recombine existing representations rather than relying on additional training.
2.2 Diagnostic Probing and Functional Analysis
A long line of probing studies demonstrates that different layers of transformer models encode distinct linguistic and reasoning-related functions. Early work shows that pretrained language models recover a classical NLP processing pipeline across layers, with syntactic, semantic, and contextual abstractions emerging at different depths tenney2019bert ; ethayarajh2019contextual ; hewitt2019structural ; rogers2020bertology . Subsequent studies reveal that functional importance is unevenly distributed, motivating layer-aware and component-specific diagnostics rather than uniform parameter heuristics ethayarajh2019contextual ; hewitt2019structural ; rogers2020bertology . More recent work extends this perspective by identifying localized causal regions and neurons whose manipulation significantly affects model behavior, reinforcing the view that functional relevance in neural networks is both localized and structurally constrained bau2020neurons ; geiger2021causal . Multilingual probing studies further show that such structural specialization generalizes across languages, supporting the use of diagnostic probes as a principled prior for guiding model reorganization li2024multilingual .
2.3 Training-Free and Static Model Merging
Static model merging combines pretrained or fine-tuned models using fixed coefficients, such as weight averaging or task arithmetic. While effective for closely aligned models, these approaches often degrade performance when merging heterogeneous specialists due to representational incompatibility and interference wortsman2022soups ; ilharco2023task ; yadav2023ties . Recent advances address these limitations by introducing training-free merging methods with structured sparsification, selective parameter alignment, or dual-space constraints, demonstrating that principled parameter selection can substantially improve merged performance without gradient-based training xu2024trainingfree . These works establish training-free model merging as a viable alternative to expensive multi-task training pipelines, while highlighting the importance of structural and representational considerations.
2.4 Evolutionary Model Merging
Evolutionary optimization provides a natural framework for exploring merge configurations in a black-box, gradient-free setting. Classic work in neuroevolution demonstrates that evolutionary strategies can effectively optimize high-dimensional neural architectures without gradient information, motivating their application to large pretrained models. More recent work shows that evolutionary search can automatically discover high-performing model merging recipes that outperform manually designed heuristics, validating its applicability to model merging akiba2024evolutionary ; akiba2025nature . Nevertheless, most existing methods remain diagnostically blind, motivating the need for diagnostic-guided, adaptive training-free merging strategies.
2.5 Cross-Architecture and Hybrid Models
Recent architectural developments explore hybrid models that combine attention-based transformers with alternative sequence modeling mechanisms, such as state-space models, to improve efficiency and long-context performance. These hybrid architectures demonstrate that complementary inductive biases can be successfully combined within a single model, motivating cross-architecture recombination beyond traditional fine-tuning. Such advances provide architectural precedent for training-free cross-architecture merging, supporting the feasibility of recombining heterogeneous model components when equipped with appropriate alignment and selection mechanisms.
3 The Darwin Framework
Figure 1 provides a high-level overview of the Darwin framework, whose core design principle is to decouple diagnostic guidance from evolutionary exploration and reconcile them through an explicit fusion mechanism. Rather than performing gradient-based training, Darwin operates entirely in weight space, recombining frozen parent checkpoints through structurally informed merge decisions.
At a high level, Darwin proceeds as follows. Model-layer Response Importance (MRI) first estimates the functional relevance of individual parameter tensors using static statistics and lightweight probe-based responses, while a low-dimensional genome encodes candidate merge configurations explored via evolutionary search. These signals are combined through MRI-Trust Fusion to determine their relative influence, producing tensor-wise merge ratios that are applied by a training-free merge kernel to construct the final merged model. We now formalize this process, beginning with the problem formulation and parameter decomposition.
3.1 Problem Formulation
Let two parent models and share a common pretrained base model . Their parameters are decomposed as
| (1) |
where and represent model-specific deviations introduced by task specialization or distillation. Our objective is to construct a merged model that improves reasoning performance without any gradient-based training, solely by recombining and in weight space wortsman2022soups ; ilharco2023task ; yadav2023ties ; xu2024trainingfree . Rather than treating all parameters uniformly, Darwin assigns tensor-specific merge ratios and optimizes them through a diagnostic-guided evolutionary process.
3.2 Merge Kernel and Parameter Recombination.
Each denotes a scalar mixing coefficient shared across all elements of tensor . Darwin constructs the merged tensor as
| (2) |
where denotes the shared pretrained base. This formulation enables selective recombination of parent parameters without any gradient-based optimization.
3.3 Model-layer Response Importance (MRI)
Darwin introduces Model-layer Response Importance (MRI) as a diagnostic prior estimating the functional relevance of individual parameter tensors for reasoning behavior tenney2019bert ; ethayarajh2019contextual ; hewitt2019structural ; rogers2020bertology ; bau2020neurons ; geiger2021causal ; li2024multilingual . For a tensor , MRI combines static tensor statistics and probe-based functional responses:
| (3) | ||||
| (4) |
The static term aggregates normalized entropy, variance, and capped -norm statistics, while the probe term measures cosine distance between reasoning-conditioned and generic activations induced by a small calibration set. The weighting parameter controls the relative contribution of static and probe-based diagnostics and is fixed to in all experiments. MRI-derived ratios serve as a soft prior rather than a fixed merge rule and are subsequently fused with genome-derived ratios through MRI-Trust Fusion.
3.4 Architecture-Aware Tensor Alignment
For heterogeneous parent architectures, Darwin applies an Architecture Mapper that establishes tensor-level correspondences prior to numerical recombination. For a candidate pair of tensors , the mapper computes a compatibility score
| (5) |
where indicates functional role correspondence, measures dimensional consistency, and captures parameter-shape similarity. The coefficients , , and are fixed heuristic weights. Layer correspondences are established via constrained greedy matching under a minimum compatibility threshold, enabling limited cross-architecture recombination without retraining.
3.5 MRI-Trust Fusion and Genome-Based Control
A key design question is how much the merge should rely on diagnostics versus evolutionary exploration. Darwin resolves this using a single scalar parameter , which controls MRI trust. The final tensor-wise merge ratio is defined as
Intermediate values of allow evolutionary optimization to correct diagnostic noise while retaining structured priors.
3.6 Genome and Evolutionary Optimization. Each merge strategy in Darwin is represented by a 14-dimensional genome
which controls global merge balance, component-level mixing ratios, sparsification densities, block-level specialization coefficients, MRI trust, and merge-kernel interpolation behavior. Evaluating a candidate genome requires instantiating a merged model and measuring its reasoning performance, making direct evolutionary search expensive. To address this challenge, Darwin employs a two-phase optimization strategy that separates structural screening from empirical evaluation.
4 Experiments and Analysis
4.1 Experimental Setup
We evaluate Darwin as a training-free reasoning enhancement framework, with primary emphasis on the flagship Darwin-27B-Opus and auxiliary experiments assessing generalization across scale, generation, and architecture. Parent models are selected to share a common pretrained base whenever possible, following standard practice in homologous model merging.
Our primary benchmark is GPQA Diamond, a graduate-level multiple-choice benchmark targeting robust scientific reasoning under standardized inference settings rein2023gpqa . To assess broader reasoning generalization, we additionally evaluate on ARC-Challenge, which emphasizes multi-step symbolic and commonsense reasoning, and MMLU, which measures massive multitask language understanding across diverse academic subjects clark2018arc ; hendrycks2021mmlu .
We compare against (i) individual parent models, (ii) static training-free merging baselines such as uniform averaging and TIES-style merging wortsman2022soups ; ilharco2023task ; yadav2023ties , and (iii) evolutionary merging without diagnostic guidance real2019regularized ; such2017deep ; akiba2024evolutionary ; akiba2025nature . All results are averaged over multiple stochastic decoding runs using identical inference settings to ensure fair comparison.
4.2 Main Results: Darwin‑27B‑Opus (Primary Evidence)
This flagship result provides primary validation of the core claims of Darwin. Table 1 reports the main reasoning results for Darwin-27B-Opus on GPQA Diamond and ARC-Challenge, together with its parent models and representative baselines.
Darwin-27B-Opus achieves 86.9% on GPQA Diamond, ranking #6 among 1,252 evaluated models (as of 2026-04-22), and outperforms its strongest parent without any gradient-based training. Notably, Darwin surpasses several substantially larger, fully trained models while requiring only a small number of GPU hours for evolutionary search. These results demonstrate that frontier-level reasoning performance can be recovered, and even improved, through weight-space reorganization alone.
Compared to static merging methods, Darwin shows consistently higher accuracy and reduced variance, indicating greater robustness to representational interference. Compared to evolutionary merging without diagnostics real2019regularized ; such2017deep ; akiba2024evolutionary ; akiba2025nature , Darwin achieves higher peak performance and more reliable convergence, suggesting that diagnostic guidance plays a critical role in navigating the merge space effectively.
We further analyze the impact of different merge kernels. Linear interpolation yields modest improvements but is susceptible to task interference. SLERP provides smoother interpolation during early exploration but consistently attains lower peak accuracy. In contrast, DARE-TIES achieves superior performance across all configurations. Its drop-and-rescale mechanism effectively mitigates destructive interference between parent models, validating its selection as the primary merge kernel in the Darwin framework.
| Benchmark | Father (Base) | Mother | ||
|---|---|---|---|---|
| (Reasoning-distilled) | Simple Avg | |||
| / SLERP | Darwin-27B-Opus | |||
| GPQA-Diamond | 0.855 | 0.862 | 0.861 | 0.869 |
| ARC-Challenge | 0.710 | 0.740 | 0.750 | 0.779 |
| CommonsenseQA | 0.770 | 0.776 | 0.778 | 0.783 |
| TruthfulQA | 0.772 | 0.775 | 0.776 | 0.778 |
| HellaSwag | 0.858 | 0.864 | 0.866 | 0.870 |
| RACE | 0.821 | 0.825 | 0.828 | 0.831 |
| MMLU | 0.754 | 0.782 | 0.768 | 0.776 |
| Natural Questions | 0.748 | 0.753 | 0.756 | 0.760 |
| TriviaQA | 0.711 | 0.718 | 0.719 | 0.722 |
| Overall Average | 0.767 | 0.776 | 0.775 | 0.786 0.040 |
4.3 Analysis of Learned Genome and Merge Dynamics
We next analyze the mechanisms underlying Darwin’s performance gains, focusing on MRI-Trust Fusion, merge kernel selection, and genome structure. First, the learned trust parameter consistently converges to intermediate values (– across scales), indicating that neither pure diagnostic rules nor unconstrained evolutionary search is sufficient. Instead, Darwin benefits from an adaptive balance in which diagnostic priors guide search while evolutionary optimization compensates for diagnostic noise and inter-layer interactions.
Second, we compare merge kernels and find that DARE-TIES consistently outperforms linear interpolation and SLERP. While SLERP provides smoother exploration during early search, it suffers from lower peak accuracy. DARE-TIES effectively mitigates destructive interference between parent models through drop-and-rescale behavior, making it particularly well-suited for heterogeneous or highly specialized parents.
Finally, analysis of evolved genomes reveals stable structural patterns, including selective preservation of attention modules and stronger recombination in feed-forward components. These patterns recur across independent runs and model scales, suggesting that Darwin discovers architectural regularities, rather than exploiting properties unique to a single model.
4.4 Ablation Studies
To isolate the contribution of the MRI-Trust mechanism, we conduct a three-way ablation on the Darwin-27B-Opus configuration, varying only the fusion while holding all other genome parameters constant.
The ablation reveals two key findings. A summary of the ablation results across different settings is reported in Table 2, which compares genome-only merging, static MRI-based merging, fixed- variants, and the full adaptive Darwin configuration. First, MRI as a signal provides a clear performance benefit: using static MRI-based merging () improves GPQA accuracy by pp relative to genome-only merging (). Second, adaptively learning the trust parameter further improves performance: the evolved variant achieves an additional pp gain over a fixed setting. Overall, the full adaptive variant yields a pp improvement over the no-MRI baseline on GPQA, indicating that MRI-Trust Fusion is a primary contributor to the observed reasoning gains.
| Configuration | setting | GPQA Diamond | CLIcK | vs. full |
|---|---|---|---|---|
| No-MRI (genome only) | (fixed) | 84.4 | 69.2 | / |
| MRI-only (static merge heuristic) | (fixed) | 85.6 | 72.4 | / |
| Fixed- 0.7 (as in V5) | (fixed) | 86.0 | 73.7 | / |
| Full Darwin V6 (adaptive ) | 86.9 | 75.3 | baseline |
4.5 Generalization Beyond the Flagship Model
While Darwin-27B-Opus provides the primary empirical validation of the framework, we observe that the same evolutionary principles generalize across model scale, generation, and parent composition. Across all tested sizes (4B–35B), independently evolved Darwin models consistently converge to intermediate MRI-trust values and exhibit asymmetric recombination patterns, with stronger preservation of attention components and more aggressive recombination in feed-forward layers.
These structural regularities remain stable across independently evolved models, including recursive second-generation merges and mixed-architecture variants, suggesting that Darwin discovers scale-invariant merging principles rather than exploiting properties unique to a single model configuration. Detailed model-wise results and genome values are reported in Appendix B.2 and Table B.1, and full family overview is provided in and the full family overview is provided in Appendix B.6. The framework also supports cross-architecture recombination.
The framework also supports cross-architecture recombination. Darwin-4B-Genesis successfully merges Transformer-based attention with Mamba-style state-space feed-forward components without any retraining, outperforming both parents on targeted reasoning benchmarks. This case illustrates that Darwin can recombine complementary inductive biases across heterogeneous architectures, beyond fine-tuning variants of the same model family. Collectively, these models are not required to establish the effectiveness of Darwin, which is validated by Darwin-27B-Opus alone. Instead, they provide supporting evidence that the same diagnostic-guided evolutionary principles extend beyond a single flagship instance, generalizing across model scale, evolutionary depth, and architectural diversity. We emphasize that cross-architecture results are included as supporting evidence of extensibility rather than as a primary performance driver, with flagship validation carried by homologous merging. Detailed model-wise results and family-level comparisons are provided in Appendix B.6.
5 Limitations and Future Work
Dependency on parent capabilities. Darwin improves upon its parent models by reorganizing latent capabilities acquired during pretraining, but it does not create new capabilities ex nihilo. If both parents lack a specific skill or knowledge domain, evolutionary merging alone cannot recover it.
Architectural and alignment constraints. At present, high-performing Darwin models require parents that share a common pretrained base. While limited cross-architecture recombination is possible through architecture-aware alignment, general cross-base merging at scale remains an open challenge.
Search cost and verification scope. Although substantially cheaper than training or fine-tuning, Darwin’s evolutionary search is not free and requires running a compact set of evaluations. In addition, while mid-scale models have been independently verified on public leaderboards, verification of the largest variants is ongoing.
Future work. Promising directions include extending Darwin to the 100B regime using sharded evaluation, improving cross-base alignment mechanisms, and combining Darwin with complementary test-time or inference-time interventions.
6 Conclusion
We presented the Darwin framework and the Darwin Family of eight evolutionarily-merged language models spanning 4B to 35B parameters. Our primary contributions are the 14-dimensional adaptive genome (§3.6), the MRI-Trust Fusion formula with learnable (§3.5), and the Architecture Mapper enabling cross-architecture breeding (§3.4). The primary case study, Darwin-27B-Opus, is officially ranked #6 on the GPQA Diamond Leaderboard, outperforming its own Father Qwen3.5-27B by pp and other frontier models.
The Darwin Family establishes training-free evolutionary merging not as a niche technique for model ensemble averaging, but as a practical and reproducible pathway to frontier-scale reasoning capability at three to six orders of magnitude lower compute cost than conventional pretraining. By releasing all models, the V6 codebase, and the MRI tooling under Apache 2.0, we hope to enable broad independent verification and to catalyze a new research program on principled, diagnostic-guided weight-space optimization.
References
- [1] Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
- [2] Takeshi Kojima, Shixiang Gu, M. Reid, et al. Large language models are zero-shot reasoners. In Neural Information Processing Systems, 2022.
- [3] Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. Self-consistency improves chain-of-thought reasoning in language models. In International Conference on Learning Representations, 2023.
- [4] Denny Zhou, Natalie Schärli, Le Hou, et al. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations, 2023.
- [5] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Association for Computational Linguistics, 2019.
- [6] Kawin Ethayarajh. How contextual are contextualized word representations? In EMNLP-IJCNLP, 2019.
- [7] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In NAACL, 2019.
- [8] David Bau, Jun-Yan Zhu, Hendrik Strobelt, et al. Identifying and controlling important neurons in neural networks. In International Conference on Learning Representations, 2020.
- [9] Atticus Geiger, Zhiwei Wu, David Lu, et al. Causal abstractions of neural networks. In Neural Information Processing Systems, 2021.
- [10] Mitchell Wortsman, Gabriel Ilharco, Samir Y. Gadre, et al. Model soups: Averaging weights of multiple fine-tuned models. In International Conference on Machine Learning, 2022.
- [11] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, et al. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
- [12] Pratyush Yadav, Derek Tam, Leshem Choshen, et al. Ties-merging: Resolving interference when merging models. In Neural Information Processing Systems, 2023.
- [13] Zhen Xu, Kai Yuan, Hao Wang, et al. Training-free pretrained model merging. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
- [14] Takuya Akiba, Makoto Shing, Yu Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187, 2024.
- [15] Takuya Akiba, Makoto Shing, Yu Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025.
- [16] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology. Transactions of the ACL, 2020.
- [17] Dehua Li, Haoyan Zhao, Qing Zeng, and Mengnan Du. Exploring multilingual probing in large language models. arXiv preprint arXiv:2409.14459, 2024.
- [18] David Rein, Benjamin L. Hou, Asa Cooper Stickland, et al. Gpqa: A graduate-level google-proof question answering benchmark. arXiv preprint arXiv:2311.12022, 2023.
- [19] Peter Clark, Brian Cowhey, Oren Etzioni, et al. Think you have solved question answering? try arc. arXiv preprint arXiv:1803.05457, 2018.
- [20] Dan Hendrycks, Collin Burns, Steven Basart, et al. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
- [21] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, 2019.
- [22] Felipe Petroski Such et al. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks. arXiv preprint arXiv:1712.06567, 2017.
- [23] Lei Yu, Bowen Yu, Hongyi Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning, 2024.
- [24] Michael Matena and Colin Raffel. Merging models with fisher information. arXiv preprint arXiv:2210.07289, 2022.
- [25] Mohammad Reza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision, 2024.
- [26] Yuki Kuroki, Yu Zhang, and Risto Miikkulainen. Cycleqd: Quality-diversity optimization through cyclic evolutionary dynamics. In International Conference on Learning Representations, 2025.
- [27] Nuno Abrantes, Miguel Lourenço, and João Monteiro. M2n2: Modular neuroevolution with adaptive network composition. In Genetic and Evolutionary Computation Conference, 2025.
- [28] Zhen Xu, Kai Yuan, Hao Wang, et al. Training-free model merging under dual-space constraints. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
- [29] Wenjing Li, Hao Gao, Mingqiao Gao, et al. Training-free model merging for multi-target domain adaptation. In European Conference on Computer Vision, 2024.
- [30] Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 2002.
- [31] Eric Yang, Li Shen, Guangyuan Guo, et al. Model merging in llms, mllms, and beyond: Methods, theories, and applications. ACM Computing Surveys, 2026.
Appendix A Reproducibility
A.1 Data and reprodctibility site
Model collection: huggingface.co/collections/FINAL-Bench/darwin-family. Distillation from Claude Opus 4.6 refers to supervised fine-tuning of an open-weight base model using reasoning traces generated via the public Claude Opus 4.6 API. No proprietary Claude model weights are used or distributed. All donor models employed in this work are publicly available community releases, and the distillation pipeline itself is fully reproducible given access to the Claude API.
Interactive demo / evolution studio: huggingface.co/spaces/VIDraft/DARWIN-Evolution.
Darwin V6 codebase: approximately 13,771 lines across 15 Python files including mri_extractor.py, mergekit_integration.py, parent_attribution.py, calibration_data.py, benchmarks.py, live_engine.py, and live_blend.py. All code is released under the Apache 2.0 license.
GPQA Diamond verification: huggingface.co/datasets/Idavidrein/gpqa (Darwin-27B-Opus at #6, Darwin-31B-Opus at #11 as of 2026-04-22).
Community quantizations: bartowski/Darwin-27B-Opus-GGUF, bartowski/Darwin-31B-Opus-GGUF, bartowski/Darwin-35B-A3B-Opus-GGUF, mradermacher/Darwin-27B-Opus-i1-GGUF, and numerous others.
A.2 Hyperparameters and Hardware
All experiments were conducted on NVIDIA A100 or H100 GPUs. Runtime scales approximately linearly with model size, from approximately 1 hour for 4B models to approximately 5 hours for 35B models.
CMA-ES population size: 50.
Generations (Phase 1): 20.
Generations (Phase 2): 5–10 (model-size dependent).
Mutation standard deviation (initial): 0.01.
Mutation decay per generation: 0.95.
Elite preservation: top-5 per generation.
Crossover: SLERP in genome space.
Random seed: fixed per question, MD5() mod .
Evaluation runs: per candidate.
Sampling temperature: 1.0 (maj@8 for David).
Sampling top-: 0.95.
Sampling top-: 64.
Hardware (per model):
Darwin-4B-Opus: A100-80GB, approximately 1 hour.
Darwin-4B-David: H100-80GB, approximately 1 hour.
Darwin-4B-Genesis: H100-80GB, 155 minutes.
Darwin-9B-Opus: H100-80GB, approximately 90 minutes.
Darwin-27B-Opus: H100-80GB, approximately 5 hours.
Darwin-31B-Opus: H100-80GB, approximately 134 minutes.
Darwin-35B-A3B-Opus: H100-80GB, approximately 5 hours.
A.3 Calibration Probe Set
The MRI calibration probe set comprises 123 samples across six categories with an approximate Korean–English balance of 50:50, following prior probing and diagnostic analysis practices [5, 6, 7, 16, 8, 9, 17]. All probe samples are available in the public Darwin V6 repository as calibration_data.py. Probe-conditional hidden states are computed via forward hooks at each transformer layer output; the GENERIC category serves as the baseline anchor for cosine-distance importance measurement described in Section 3.3.2.
| Category | Samples | Purpose |
|---|---|---|
| REASONING | 28 | Multi-step chain-of-thought (arithmetic, logical deduction, conditional inference) |
| CODE | 22 | Programming tasks (Python, algorithm synthesis, code understanding) |
| LOGIC | 18 | Formal deduction (syllogism, categorical/hypothetical reasoning) |
| MULTILINGUAL_KO | 20 | Korean language comprehension, cultural knowledge (from CLIcK-style) |
| MULTILINGUAL_EN | 20 | English baseline for Korean delta |
| GENERIC | 15 | Everyday conversational queries (baseline for cosine-distance anchor) |
| Total | 123 | Korean:English 50:50 by character |
Appendix B Genome Design and Evolutionary Optimization
B.1 Genome-Based Representation of Merge Strategies.
For conceptual clarity, we first describe the six core component-level parameters, which form a subset of the full 14-dimensional genome defined in Section 3.6. Darwin encodes merge strategies as a compact six-dimensional genome vector that balances expressiveness with computational tractability:
— global weight ratio controlling the overall contribution of each parent;
— layer-specific weight for attention blocks;
— layer-specific weight for feed-forward networks;
— weight for token embedding and unembedding matrices;
— Bernoulli sparsity rates controlling how aggressively parameter deltas are dropped and rescaled for each parent.
Different genome profiles correspond to qualitatively distinct merge strategies. Balanced genomes (, uniform component ratios) favor general-purpose fusion; asymmetric genomes (e.g., ) are suited to reasoning-focused tasks where attention blocks carry disproportionate signal; and sparse genomes (low density values) promote stability under task interference. By mapping MRI-derived layer sensitivities into genome initialization, Darwin ensures that the initial search population reflects functional layer specialization rather than arbitrary heuristics.
A central component is the use of MRI-based layer sensitivity probes to guide genome initialization. By analyzing parameter differences between donor models at each layer, MRI provides structured signals that inform initial merge ratios for attention, feed-forward, embedding, and global components. This warm-start strategy ensures that evolutionary search begins from a promising region of the genome space, accelerating convergence and improving merge quality relative to random initialization. These six parameters constitute the core subset of the full 14-dimensional genome, which is formally defined in Appendix B.2.
B.2 Full 14-Dimensional Genome Definition.
Each Darwin merge is controlled by a 14-dimensional genome consisting of component-level ratios, sparsification densities, block-level coefficients, and fusion parameters. Representative values and empirically stable ranges for each genome parameter, as evolved across model scales, are summarized in Table 4.
| Parameter | 4B-Opus | 27B-Opus | 31B-Opus | Typical range |
| global_ratio () | 0.5204 | 0.4893 | 0.4712 | |
| attn_ratio () | 0.3195 | 0.1463 | 0.1890 | |
| ffn_ratio () | 0.8421 | 0.8768 | 0.9204 | |
| embed_ratio () | 0.3508 | 0.3021 | 0.2894 | |
| density_a | 0.8934 | 0.8507 | 0.8625 | |
| density_b | 0.9011 | 0.9413 | 0.9228 | |
| mri_trust | 0.4907 | 0.5557 | 0.3631 | |
| merge_method_weight | 0.3124 | 0.2783 | 0.3502 |
B.3 Parameters.
Darwin encodes merge strategies as a 14-dimensional genome vector composed of three groups of parameters.
Core parameters (6): The core component-level parameters control global and module-specific mixing behavior: (overall merge balance, ); (attention component weight, ); (feed-forward component weight, ); (token and position embedding weight, ); and , which control Bernoulli sparsification rates applied to the Father and Mother parameter deltas, respectively.
MRI-derived block parameters (6): The block-level parameters through assign independent merge ratios to six contiguous layer blocks identified by MRI, capturing coarse-grained dominance patterns across network depth.
Meta-evolution parameters (2): The parameter controls the interpolation between diagnostic MRI-based ratios and genome-driven ratios, as defined in Section 3.5. The parameter , denoted by in Section 3.6, controls interpolation between the DARE-TIES and SLERP merge kernels.
B.4 Scale-Invariant and Asymmetric Genome Patterns
Across model sizes ranging from 4B to 35B parameters, evolved Darwin genomes exhibit stable and recurring parameter ranges, particularly for attention preservation ratios, feed-forward recombination ratios, and MRI-trust values. As shown in Table B.1, these parameters concentrate within narrow intervals across independently trained models, indicating that the evolutionary process discovers scale-invariant structural regularities rather than size-specific artifacts.
A salient pattern is the systematic asymmetry between attention and feed-forward components. Across all tested scales, Darwin consistently preserves a large fraction of attention parameters from the base (Father) model while aggressively recombining feed-forward layers from the specialized (Mother) model. This asymmetry aligns with prior probing and analysis studies suggesting that attention layers primarily mediate information routing and focus, whereas feed-forward networks encode task-specific computation and transformation [5, 6]. Notably, this pattern emerges consistently across evolutionary runs and cannot be readily anticipated through manual design or uniform-ratio merging. Instead, it reflects an architectural regularity discovered by diagnostic-guided evolutionary search, reinforcing the view that Darwin reorganizes latent reasoning structure rather than introducing ad hoc parameter configurations.
B.5 Evolutionary Optimization Procedure
Darwin employs a two-phase evolutionary optimization strategy to efficiently search the merge-genome space while limiting the computational cost of full model instantiation.
Figure 3 provides a schematic overview of this evolutionary process, illustrating the iterative cycle of fitness evaluation, selection, crossover, and adaptive mutation that drives convergence toward a compact set of high-quality genomes.
In Phase 1, candidate merge genomes are evolved using a standard evolutionary optimization loop. Each generation evaluates candidate genomes using a lightweight proxy fitness score, followed by parent selection and variation through crossover and adaptive mutation. This phase is designed to rapidly filter structurally implausible or low-quality merge configurations without constructing full merged models.
In Phase 2, a small set of high-quality genomes identified in Phase 1 is instantiated as merged models and evaluated directly on reasoning benchmarks. Final selection is based on empirical performance under fixed inference settings.
B.6 Darwin Family Overview Across Scale and Generation
The table summarizes all released Darwin models and highlights the recurrence of intermediate MRI-trust values and asymmetric attention/FFN recombination patterns across scales. Table 5 provides an overview of the Darwin Family across model scale, evolutionary generation, and parent composition, reporting representative benchmark performance and notable properties for each released variant. For detailed model-level configurations and comparisons, we refer the reader to Table 5.
| Model | Gen | Parents (Father Mother) | GPQA% | Notable Property |
|---|---|---|---|---|
| Darwin-4B-Opus | 1 | gemma-4-E4B Deckard | — | , 14-D genome |
| Darwin-4B-David | 2 | Darwin-4B-Opus DECKARD-24B-D | 85.0 | First recursive evolution (+26.4pp maj@8) |
| Darwin-4B-Genesis | 3 | Darwin-4B-David Qwen3.5-4B | 60 | First cross-arch FFN (CLIcK 92%) |
| Darwin-9B-Opus | 1 | qwen3_5-10B base | — | Compact Qwen variant |
| Darwin-27B-Opus | 1 | Qwen3.5-27B Claude Opus 4.6–style reasoning-distilled variant (open-weight base) | 86.9 | GPQA official #6, primary case |
| Darwin-31B-Opus | 1 | gemma-4-31B TeichAI-distill | 85.9 | GPQA official #11, |
| Darwin-35B-A3B-Opus | 1 | Qwen3.5-35B-A3B MoE Jackrong | 90.0∗ | Flagship MoE, 262K ctx, 201 langs |
Appendix C Architecture Mapper and Merge Kernels (Extended)
This appendix provides extended details on the merge kernels used in Darwin to recombine aligned parameter tensors after architectural alignment. The Architecture Mapper, which determines tensor-level correspondences between parent models, is introduced in Section 3.4 and is treated as a structural preprocessing step. In contrast, Appendix C focuses on the kernel-level operations that specify how aligned tensors are combined once correspondence has been established.
C.1 Role Separation: Mapper vs. Kernel
For clarity, we distinguish two components involved in cross-model recombination. The Architecture Mapper (Section 3.4) determines which parameter tensors from the two parent models can be meaningfully recombined by operating at the structural level, where it establishes tensor correspondences based on type, dimensionality, and positional compatibility. The mapper does not modify parameter values and performs no numerical combination. Given the matched tensor pairs identified by this alignment step, the merge kernel (described in this appendix) specifies how their parameter values are combined. Merge kernels operate at the parameter level and apply tensor-wise mixing ratios, sparsification rules, or interpolation schemes to produce the merged weights. This separation allows Darwin to cleanly decouple structural alignment from numerical recombination.
C.2 Evolutionary Optimization Procedure (Context)
Darwin employs a two-phase evolutionary optimization strategy to search over merge configurations without gradient-based updates. In Phase 1, candidate genomes are screened using a lightweight proxy objective to eliminate degenerate or structurally implausible configurations. In Phase 2, a small set of promising genomes is instantiated into merged models and evaluated directly on reasoning benchmarks. The merge kernels described below are invoked only after tensor alignment by the Architecture Mapper and ratio selection via MRI-Trust Fusion (Section 3.5).
C.3 DARE-TIES Merge Kernel
DARE-TIES (Drop-And-Rescale with Task-Interval Elimination) is the primary merge kernel used for final model construction in Darwin. Given aligned parent tensors and genome-specified mixing coefficients, this kernel operates by computing parameter deltas relative to a shared base model and applying Bernoulli sparsity masks to selectively retain informative components.
Specifically, for parent models and sharing a common base , Darwin computes and , applies genome-controlled Bernoulli masks and to each delta, rescales the surviving entries to preserve expected magnitude, and then performs weighted recombination as
| (6) |
where denotes the genome-specified mixing weight for component .
This drop-and-rescale procedure mitigates destructive interference between parent models while preserving complementary reasoning behaviors, and has been empirically observed to yield more stable performance than uniform averaging or linear interpolation. For this reason, Darwin prioritizes DARE-TIES for benchmark-driven fitness evaluation.
An overview of the DARE-TIES merge procedure is illustrated in Figure 4, which visually summarizes the drop-and-rescale operation applied to aligned parent parameter deltas before recombination.
C.4 SLERP Kernel (Exploration Phase)
SLERP (Spherical Linear Interpolation) is used as a lightweight alternative kernel during early evolutionary exploration. By interpolating tensors along a hyperspherical path, SLERP enables smooth exploration of merge configurations with lower computational overhead. However, in Phase 2 evaluation, SLERP consistently underperforms DARE-TIES and is therefore not used for final model selection.
C.5 Summary
In summary, the Architecture Mapper (Section 3.4) determines tensor correspondences across heterogeneous parent models, while the merge kernels described here define the numerical rules for combining those tensors. Appendix C therefore complements the main text by detailing how aligned parameters are merged, not how alignment is established.
Appendix D Comparison with Prior Model Merging Methods
D.1 Overview of Comparison
This appendix situates Darwin within the broader landscape of model merging approaches, with particular emphasis on training-free methods. We focus on differences in assumptions, optimization structure, diagnostic usage, and extensibility, rather than raw performance, which is reported in the main text. Table 6 provides a comparative overview of Darwin and prior model merging methods along key dimensions, including genome dimensionality, diagnostic guidance, cross-architecture support, and multi-generation capability.
Darwin is the only prior-art-surveyed method that simultaneously (a) operates in a double-digit-dimensional genome, (b) integrates a functional-importance diagnostic signal into the merge kernel via a learnable parameter, (c) supports cross-architecture breeding, and (d) has been demonstrated across multiple evolutionary generations with heritable gains.
| Method | Genome dim. | Pre-merge diagnosis | Cross-arch breeding | Multi-generation |
|---|---|---|---|---|
| TIES-Merging [12] | — | none | no | no |
| DARE [23] | — | sparsification | no | no |
| Model Soups [10] | — | none | no | no |
| Task Arithmetic [11] | — | none | no | no |
| Fisher Merging [24] | — | Fisher info. | no | no |
| Model Breadcrumbs [25] | — | sparse mask | no | no |
| Sakana EvoMerge [14, 15] | 2 per layer | none | partial (DFS) | no |
| CycleQD [26] | MAP-Elites | none | no | no |
| M2N2 [27] | evolvable splits | none | no | no |
| Darwin V5 (2026, ours) | 2 (uniform) | MRI (precursor) | no | no |
| Darwin V6 (ours) | 14 adaptive | MRI 6-probe | via mapper | yes |
| Darwin-4B-Genesis (V6+42D) | 42 per-layer | MRI 6-probe | yes (TF Mamba) | Gen-3 (heritage) |
D.2 Static and Heuristic-Based Model Merging
Early model merging approaches rely on static, low-dimensional heuristics that combine pretrained or fine-tuned models using fixed coefficients. Representative examples include uniform weight averaging (Model Soups) [10] and linear vector arithmetic in weight space (Task Arithmetic) [11]. These methods are attractive due to their simplicity and low computational cost, and they perform well when parent models are closely aligned in function and training history.
However, static heuristics implicitly assume that all parameters are equally mergeable. As later work demonstrates, this assumption often fails for heterogeneous specialist models, leading to representational interference and degraded performance [12]. TIES-merging partially addresses this issue by selectively trimming and rescaling parameter deltas, but the selection rules remain hand-designed and task-agnostic [12]. As a result, these approaches lack adaptivity to parent-specific structure and cannot easily generalize beyond closely related models.
D.3 Structured Training-Free Merging with Parameter Selection
Recent work improves training-free merging by introducing structured parameter selection and alignment constraints. Training-Free Pretrained Model Merging [13] and Dual-Space Constraint Merging [28] explicitly model consistency between weight space and activation space, demonstrating that selective alignment significantly improves merged performance without gradient updates. Related work on multi-target domain adaptation further shows that principled training-free merging can rival data-sharing baselines when parent models share a common pretrained backbone [29].
While these methods represent a significant advance over static heuristics, they typically rely on fixed selection rules or optimization objectives that are not adaptive to downstream reasoning behavior. Moreover, they are usually limited to single-generation merging and do not naturally extend to iterative or evolutionary composition.
D.4 Evolutionary Model Merging
Evolutionary optimization offers a complementary perspective by treating model merging as a black-box search problem, optimizing merge configurations without gradient information. Classical work in neuroevolution demonstrates that evolutionary strategies can effectively search high-dimensional neural parameter spaces [30, 21, 22]. Building on this foundation, recent work shows that evolutionary search can automatically discover high-performing model merging recipes that outperform human-designed heuristics [14, 15].
However, existing evolutionary merging approaches are largely diagnostically blind. They typically operate over low-dimensional or uniform merge parameters and treat all components as symmetrically mutable, resulting in inefficient exploration and limited interpretability of evolved solutions.
D.5 Diagnostic-Guided Evolutionary Merging in Darwin
Darwin integrates the strengths of structured training-free merging and evolutionary optimization while addressing their limitations. Unlike static or rule-based methods [10, 11, 12, 13, 28, 29], Darwin replaces fixed heuristics with an explicit adaptive genome that parameterizes merge behavior at multiple structural levels. Unlike prior evolutionary approaches [30, 21, 22, 14, 15], Darwin incorporates diagnostic priors that estimate functional relevance, allowing evolutionary search to focus on reasoning-critical components.
A key distinction is MRI-Trust Fusion, which adaptively balances diagnostic guidance and evolutionary exploration via a learnable trust parameter. This design enables Darwin to interpolate between heuristic-driven merging and unconstrained search, rather than committing to either extreme. As a result, Darwin supports multi-generation evolution, cross-architecture merging, and robust reasoning improvements without gradient-based training.
D.6 Summary Comparison
In summary, existing model merging approaches trade off simplicity, structure, and flexibility. Static heuristics are simple but fragile [10, 11, 12]; structured training-free methods are principled but inflexible [13, 28, 29]; evolutionary methods are flexible but inefficient without guidance [30, 21, 22, 14, 15]. Darwin occupies a distinct point in this space by combining training-free operation, diagnostic selectivity, and evolutionary adaptability within a unified framework. A comprehensive survey of the broader model merging landscape can be found in [31].
Appendix E Failure Modes and Negative Results
Having situated Darwin relative to prior training-free model merging methods (Appendix D), we now analyze failure cases to clarify the operational boundaries of diagnostic-guided evolutionary merging.
Analysis of non-improving parent pairs reveals several recurring failure modes that are structural rather than incidental. Importantly, these cases do not contradict the effectiveness of Darwin, but instead clarify the conditions under which diagnostic-guided evolutionary merging is expected to succeed.
Lack of complementary specialization. In cases where both parent models exhibit highly similar capabilities and error patterns, evolutionary merging provides limited benefit. When neither parent contributes a distinct or dominant capability, recombination primarily redistributes redundant structure rather than composing complementary functions, resulting in negligible or no improvement.
Severe representational misalignment. Some non-improving merges involve parent models whose internal representations are poorly aligned, even when nominally derived from the same base architecture. In such cases, weight-space recombination may disrupt reasoning-critical pathways faster than evolutionary optimization can recover them, leading to early saturation of gains.
Ambiguity in diagnostic signals. Darwin relies on MRI as a soft diagnostic prior rather than a ground-truth indicator. When diagnostic signals are weak, noisy, or inconsistent across layers—for example, when reasoning-relevant activations are diffused rather than localized—MRI guidance becomes less informative. Evolutionary search can partially compensate for such noise, but the resulting gains are typically smaller and less stable.
Search space saturation. Finally, some parent pairs already approach a local optimum for the targeted reasoning benchmarks. In these regimes, Darwin’s evolutionary search converges quickly, but further improvement is constrained by the lack of latent complementary structure rather than by search inefficiency.
Together, these failure modes indicate that Darwin is most effective when applied to heterogeneous but compatible parent models with partially complementary reasoning structure. Failure cases therefore serve not as counter-examples, but as boundary conditions that clarify the operational scope of diagnostic-guided evolutionary merging.
Appendix F Resources and Community Adoption
All Darwin Family models, code, and MRI tooling are released under the Apache 2.0 license. As an indicator of community adoption, released Darwin models have accumulated substantial downloads across official and community-maintained distributions. Table 7 summarizes cumulative download counts as of 2026-04-22, covering official checkpoints as well as popular GGUF and third-party releases. For a detailed breakdown by model variant and distribution channel, we refer the reader to Table 7. As of April 2026, community downloads exceed 96,000 across official and quantized distributions. The combined download count exceeds 96,000 across all distribution channels, comparable to the adoption level of released open-source reasoning models from major labs. The substantial community-quantization activity (bartowski, mradermacher) further indicates that Darwin Family models are not merely benchmarked but actively deployed.
| Model | Official | bartowski GGUF | mradermacher + others |
|---|---|---|---|
| Darwin-27B-Opus | 14,000 | 22,000 | 8,500 |
| Darwin-35B-A3B-Opus | 9,000 | 14,500 | 6,000 |
| Darwin-31B-Opus | 6,500 | 8,000 | 3,000 |
| Darwin-4B-David, -4B-Opus, -4B-Genesis, -9B-Opus (combined) | 3,500 | 1,200 | 400 |
| Family total | 33,000 | 45,700 | 17,900 |