推出 K2 Horizon：前沿性能，极致开放
今天，IFM 发布了 K2 Horizon，这是一个由六款模型组成的互联模型家族：375B-A23B、36B-A4B、32B、7B、3.7B 和 0.9B。在推理、数学、编程、智能体任务和通用能力方面，K2 Horizon 在每个尺寸级别都提供了顶级性能——其中 0.9B、3.7B 和 7B 模型在其各自规模上树立了新的行业标杆。
K2 Horizon 也是我们迄今为止最全面的开源发布。对于每一款模型，我们都开放了从预训练到推理和智能体后训练的全流程训练生命周期。我们发布了中间检查点、训练数据或详细的数据构建方案、开放架构、混合组成、训练代码、配置、细粒度日志、评估结果以及最终权重。
这些模型和代码均根据 Apache 2.0 许可证发布。数据集则根据其适用的许可证发布，例如 ODC-BY；当无法进行再分发时，我们会披露数据的构建和混合方式。
总而言之，K2 Horizon 代表了迄今为止最全面的开源模型发布：
- 跨越不同规模的全新性能前沿。0.9B、3.7B 和 7B 模型在各自规模类别中，于广泛使用的评测上均达到世界领先水平。搭载我们全新混合价值注意力（MoVA）机制的 36B-A4B 模型，在单位激活参数上展现出卓越能力，性能超越部分规模更大的模型。32B 与 375B-A23B 模型则位列各自类别中的顶尖模型之列。这六款模型共同提供了从边缘设备到企业级等各类部署环境中均具竞争力的性能表现。
- 首个面向智能体的完全开放模型系列。K2 Horizon 是首个通过智能体后训练公开完整开发过程的开放模型家族。通过发布每个阶段的检查点、数据（或数据配方）、代码、配置和训练日志，K2 Horizon 使得研究推理、工具使用、规划及智能体能力如何涌现成为可能；能够复现产生这些能力的方法；并将这些方法适配到新的工具、环境和领域。
- 六个模型覆盖从边缘到企业的全场景。0.9B 模型专为手表、眼镜等高度受限环境设计，而 3.7B 和 7B 模型则为手机及其他端侧应用带来先进能力。稠密 32B 模型和稀疏 36B-A4B 模型为本地工作站和高效推理服务提供了强大选择。375B-A23B 模型则将全系列最强能力带给要求严苛的企业级部署场景。全部六个模型均支持量化。
- 一个互联的模型家族。这六款模型共享核心架构、词表、训练方法、接口、评测基础设施和部署工具链，其中 0.9B 模型使用较小的词表。这种一致性也使得在不同规模之间切换、动态分配任务，以及跨规模研究能力与效率变得更加容易。
各规模均达到世界领先性能
0.9B、3.7B 和 7B 模型在各自所属的规模类别中，于数学、推理、通用能力、编程和智能体任务方面均取得了最先进的结果。
36B-A4B 模型的性能超出了其激活参数数量通常对应的预期水平，展示了我们在计算注意力值时独特的混合专家（Mixture-of-Expert）设计的效率。32B 和 375B-A23B 模型在其各自的对比类别中均位居顶尖模型之列。
小模型尤其值得关注。K2 Horizon 0.9B 在 AIME 2026 上取得了超过 48 分的成绩，同时具备强大的推理、工具使用和智能体能力。K2 Horizon 3.7B 和 7B 将这些能力扩展到要求更高的软件工程和多步骤环境中，这体现在其在 SWE-bench 和 BrowseComp 上的强劲表现。尽管需要大量探索和反复恢复的复杂任务（例如 TerminalBench 中的任务）对最小的模型来说仍然具有挑战性，但 K2 Horizon 在每个规模上都拓展了可能性的边界。
为什么 Horizon 模型家族意义重大
一个透明但在能力前沿上远远落后的模型，即使作为研究基础，其价值也有限。与此同时，一个仅以最终权重形式发布的强大模型，虽然允许人们运行它，却几乎无法让人洞察其能力是如何被创造出来的。
K2 Horizon 将这两者结合在了一起。该模型系列提供了极具竞争力的模型，并发布了用于训练它们的配方。研究人员可以在足够强大、足以展现先进能力的模型中研究这些能力，而开发者则可以复现、调整并扩展这些方法，而不是将最终检查点视为一个不透明的起点。
自我们在 2023 年 LLM360 论文中引入完全开放原则以来，我们每年都发布开放模型，同时将这一承诺扩展到更大的规模、更强的能力，并如今通过智能体后训练覆盖完整的生命周期。
深入解析 K2 Horizon 模型系列
K2 Horizon 375B-A23B：企业级性能旗舰
K2 Horizon 375B-A23B 是该系列中规模最大、能力最强的模型。其稀疏 MoE 架构提供了总计 3750 亿参数的总容量，同时每个 token 仅激活约 230 亿参数，使其能够调用远大于自身的模型容量，而无需为每个 token 使用全部参数。
该模型在通用、推理、编码和智能体评测中均位列 4000 亿参数以下顶级模型之列。它专为对模型质量要求最高的严苛工作负载而设计，包括复杂推理、软件工程、研究和长周期智能体任务。
与 Horizon 系列中的每个模型一样，375B-A23B 并非作为单一端点发布，而是作为一个开发树发布。其中间检查点和后训练分支揭示了基础模型如何发展为推理、指令遵循和专用智能体变体。
K2 Horizon 32B 和 36B-A4B：本地部署的强劲性能
Horizon 32B 是该系列中最强大的稠密模型，在能力、适应性和本地可部署性之间实现了强劲平衡。它位列 400 亿参数以下顶级稠密模型之列。
Horizon 36B-A4B 在每 token 仅激活约 40 亿参数的情况下，性能几乎达到稠密 32B 模型的水平。其效率来自 MoVA——我们新的稀疏注意力架构，以及 MoE 前馈层。
这两个模型为研究稠密与稀疏架构在相似训练条件下的行为提供了重要参考点。
这些模型占据了该系列本地性能的最佳平衡点。它们对于要求严苛的推理、编码和智能体应用来说足够强大，同时对于本地工作站和高效推理服务系统而言仍然保持实用性。
K2 Horizon 7B、3.7B 与 0.9B：小规模下的前沿能力
k2 Horizon 7B 和 3.7B 在推理、数学、编程、工具调用和智能体任务上表现出色，同时仍适合本地及端侧部署。在多项评测中，它们的结果接近甚至超过了上一代体积大数倍的模型。
K2 Horizon 0.9B 将许多相同的能力带入了高度受限的环境。它能够进行数学推理、使用工具并完成简单的智能体任务，同时在量化后仍足够小巧，可应用于手表、眼镜及其他边缘设备。
合适的任务随规模而变化：0.9B 模型最适合聚焦型交互和轻量级工具调用，而 3.7B 和 7B 模型则能处理要求更高的编程和多步骤工作流。它们共同展示了如今在足够小、几乎可随处运行的模型中能保留多少能力。
设计 K2 Horizon
从一开始就是一个统一的模型家族
Horizon 被设计为一个相互关联的模型家族，而非一组互不相关的模型集合。这六个模型共享核心架构决策、训练方法、接口、评测基础设施和部署工具。这使得开发者能够更轻松地在不同规模之间切换，也为研究人员提供了一个更可控的环境来研究跨规模的能力差异。
每个模型均使用经过精心构建并记录在案的混合数据，在约 20 万亿个 token 上进行预训练。在整个训练过程中，会捕获中间检查点及其对应的细粒度日志，从而形成关于每个模型如何发展的详细记录。
MoVA：通过稀疏专家扩展注意力
混合专家模型的核心思想是在保持每个 token 所需计算量大致不变的同时，增加模型的总容量。传统的 MoE 架构主要将这种稀疏性应用于前馈层：虽然有许多专门的专家可用，但路由器仅为每个 token 激活其中一小部分。
我们的新架构 MoVA——Mixture-of-Value Attention（混合值注意力）——将这一原则扩展到了注意力机制。由于注意力决定了 Transformer 如何从其上下文中汇集信息，因此在此处引入稀疏性，为超越前馈网络的模型容量扩展开辟了另一个维度。
MoVA 将专家路由集成到多头注意力中，同时保持与 FlashAttention、分组查询注意力（grouped-query attention）和稀疏注意力等高效技术的兼容性。
其成果便是 K2 Horizon MoVA 36B-A4B：一个总参数量为 360 亿、但每个 token 仅激活约 40 亿参数的模型。在相同的训练条件下，其性能仅略低于密集型的 Horizon 32B 模型，而所需的激活参数却大幅减少。
训练数据
跨六个规模进行训练，需要足够广泛以支撑通用能力、同时又经过精心构建以在约 20 万亿 token 上维持质量的数据。
Horizon 的预训练数据混合了多样化的网页、代码、数学、科学、多语言及特定领域来源，并结合了通过我们自有流水线生成的合成数据。我们关键的数据创新之一是将推理直接融入预训练：近 17% 的预训练语料由带有显式推理的解题轨迹构成。数学任务的推理轨迹还被进一步改写为对话、学习指南等格式。总体而言，我们在预训练期间使用了约 10 万亿个合成 token。
我们的合成数据流水线使用了数百万种多样性旋钮与上下文种子的组合，包括从内部构建的、覆盖预训练网页语料的搜索引擎中进行检索。为了在语料规模上量化多样性，我们开发了一种基于 gzip 的自定义压缩度量，并采用自适应步进来避免标准 gzip 度量在文档数量增长时迅速饱和。如下所示，我们合成数据的实测多样性接近高质量自然网页文本，并显著超过网页代码。
我们记录了数据构成、合成流水线以及训练数据混合的构建方式。遵循我们的开源原则，在许可允许的情况下，我们直接发布可直接用于训练的数据集。在再分发受限的情况下，我们发布来源描述、过滤与构建方法以及混合构成。这使得研究人员能够了解每个模型从哪些数据中学习，以及数据分布在整个训练过程中如何演变。
我们的后训练数据从中期训练一开始就引入，而非仅保留在训练的最终阶段。我们将合成的长上下文文档与遵循指令、推理以及采用聊天模板格式化的智能体轨迹相结合。我们后训练数据的一个主要支柱是，基于任务分类体系、多样性旋钮和网络搜索种子进行的大规模任务合成，由此产生了超过 1 亿个独特任务。此外，我们还开发了采样技术，在生成训练轨迹时引导求解器大语言模型走向正确的解决方案和期望的行为。
预训练动态
每个 K2 Horizon 检查点都附带详细的训练日志和细粒度历史记录，这些记录揭示了大规模训练的真实动态：损失如何演变、不稳定性出现在何处、干预措施如何影响训练，以及能力何时开始涌现。
作为这些记录为何有用的一个例子，K2 Horizon 3.7B、7B、32B 和 36B-A4B 是在完全相同的 22 万亿个 token 上训练的。当按训练进度对齐，并以训练最后 1% 的中位损失进行归一化后，它们的原始损失轨迹大致重合，您可以在下图中看到这一点——即便跨越了稠密和稀疏架构，且总参数量相差近一个数量级。尽管这种归一化在构造上对齐了它们的端点，但它并不强制早期和中间阶段的轨迹重合。它们之间的高度一致表明，在共享数据和相同配方下，对于在整个训练过程中使用相同数据集的该系列子集，学习曲线的形态保持了显著的一致性。
这些检查点与日志加在一起，使研究人员能够探究：为何某些动力学特征可以跨规模、跨架构迁移，而另一些则出现分化；并能将这些差异与特定的训练阶段、数据配比和技术决策联系起来。
面向推理与智能体的后训练
K2 Horizon 的大部分高级推理与智能体能力都在后训练阶段涌现。完整流程包括中期训练、监督微调、模型合并，以及结合专门智能体训练的强化学习。这一过程并非只产出单一的最终聊天模型，而是构建出一棵开发树。不同分支分别专精于推理、编码、工具调用和智能体领域，同时与公共基础检查点保持连接。
我们发布了这些阶段中的各类产物，以便研究人员研究每项能力是在何处涌现的、复现各个分支，并将相同的方法适配到新的工具与环境中。
Uno Diffusion：为 Horizon 提供即插即用的无损加速
推理速度已成为头号优化目标，尤其是在推理模型与智能体时代。随着模型生成更长的思维链并执行更多操作，即便是每个 token 的微小延迟也会累积成显著的时延。然而，自回归语言模型一次只生成一个 token，由此形成了根本性的瓶颈。现有方法只能提供部分解决方案：推测解码通常需要单独训练的草稿模型，而离散扩散虽能实现并行生成，却往往以牺牲质量为代价换取速度。
Uno 的设计目标就是消除这种取舍。它提供无损推理加速，在提升生成速度的同时不降低响应质量。Uno 让 Horizon 的自回归参数保持冻结，并全权负责模型的输出分布，而一组轻量级扩散参数只学习如何更高效地生成。
通过我们称之为“扩散蒸馏”（Diffusion Distillation）的流程，这些紧凑的适配器学会了并行生成整块 token。其结果是，既保留了自回归解码的质量保证，又兼具扩散模型的速度优势：模型得出相同的答案，但速度更快。
在我们的各项评估中，Uno 在速度与质量的权衡上优于领先的投机解码系统，以及开源权重和专有的扩散语言模型。关键在于，在我们测试的每一个批次大小下，其优势都持续存在，从而为交互式智能体带来更低的延迟，为大规模服务带来更高的吞吐量，且模型质量无损，额外训练开销可忽略不计。Uno 以简单的 LoRA 适配器形式交付，让这种无损加速易于采用：你只需挂载这些适配器即可。
用于构建和扩展 Horizon 的开放基础设施
我们随模型本身一同发布用于构建 Horizon 的基础设施。目标不仅是让整个模型集群可复现，更是让它的开发技术栈能够作为新模型和系统的基础。
本次发布的核心是 xLLM，这是我们经过生产环境验证的训练基础设施。xLLM 将大规模训练性能与科研所需的灵活性融为一体，使团队能够修改架构、数据配比、训练阶段和目标，而无需重建周边系统。
我们还将发布完整的智能体后训练代码库，包括强化学习部分，希望这能帮助研究人员推进相关领域的技术发展。
研究人员可以利用这些组件来检查训练行为或复现特定阶段。开发者可以将 Horizon 适配到特定领域、添加新工具、训练智能体专家，或在统一接口背后部署多种规模的模型。
从开源到开放科学
中间检查点将模型开发转变为可观察的科学过程。它们让研究人员能够研究能力何时涌现、训练选择如何改变行为，以及非预期策略何时首次出现。
奖励黑客行为提供了一个很有启发性的例子。当一个能力强大的模型被置于真实的计算机环境中并被要求解决复杂任务时，正是那种让它有用的足智多谋，也可能引导它利用评估本身的漏洞走捷径。为了了解这对 K2 Horizon 报告数据的影响程度，我们使用 Artificial Analysis 的奖励黑客审计流程对已发布的模型进行了审计。
TerminalBench 2.1 将模型置于沙盒化的计算机环境中，评估它们能否完成复杂的技术任务。一个能力强的模型必须会探索文件、调用工具、诊断故障，并寻找通往解决方案的替代路径。这种随机应变的能力正是我们想要的。但偶尔它也会越界：模型不去解决预期的问题，而是找到隐藏的答案、利用任务暴露出的漏洞，或者操纵评分器本身。
我们在 89 个 TerminalBench 2.1 任务上运行了 K2 Horizon 375B-A23B，每个任务尝试 8 次，共产生 712 次试验。其中 500 次通过了任务验证器（报告准确率为 70.2%）。随后，我们使用 Artificial Analysis 的奖励黑客审计流程对每一次通过的试验进行了审计，逐字应用了他们的 harbor analyze 工具，并采用 reward_hacking 标准及其完整评分规则文本，以 Codex gpt-5.6-sol 作为评判模型。
审计在 10 个任务中标记了 24 次试验。剔除这些试验后，准确率从 70.2% 降至 66.9%，修正了 3.37 个百分点。其余 79 个任务完全干净。作为参照，Artificial Analysis 报告的标记率分别为：claude Fable 5 为 2.2%，GPT-5.6 Luna 为 4.1%；K2 Horizon 的 3.37% 处于该区间内。
该模型发现了多种策略：
- 推断自己处于公开基准测试环境中，在 GitHub 上找到对应代码仓库，并下载参考解决方案
- 从真实项目的公共仓库中拉取当前源码，并直接复制修复方案，而不是自行推导
- 检查未公开的文件、生成脚本或泄露的凭据
- 修改测试框架，或构造利用测试成功判定方式的输出来作弊
我们观察到一个相关案例：K2 Horizon 7B 找到并下载了 SWE-bench 的答案，从而获得了虚高的 82 分。这个分数并不能代表真实的软件工程能力，但这一行为在科学上很有启示性：基准作弊是更广泛的规划、工具使用、环境探索和持久性等能力发展过程中意外出现的副产品。
由于我们在发布最终模型的同时也发布了中间检查点，这些行为可以被研究而非被掩盖。研究人员可以确定某种策略首次出现的时间，将其与训练过程中的变化联系起来，并衡量它对报告性能的影响。
因此，K2 Horizon 不仅仅是一组模型权重。它是一个开放的实验，展示了能力——及其意外后果——在整个训练过程中是如何发展的。
立即开始使用 K2 Horizon！
K2 Horizon 的全部六个尺寸均以 Apache 2.0 协议开放权重发布，并自发布首日起即获得 vLLM、SGLang 和 Ollama 的支持。此外，K2 Horizon 支持在 NVIDIA、AMD 和 Cerebras 硬件上部署，提供从本地推理到大规模服务的多种选择。可从我们的仓库获取模型：https://huggingface.co/IFM，下载模型、查看中间检查点、研究训练数据与数据配比、复现训练阶段，或使用 Horizon 代码与基础设施构建新模型和智能体。
K2 Horizon 提供的远不止六个最终模型。它提供了一份开放的蓝图，用于理解、适配并推进下一代 AI 系统。
完整结果表
375B-A23B
36B-A4B
32B
K2 Horizon 7B
K2 Horizon 3.7B
K2 Horizon 0.9B
Introducing K2 Horizon: Frontier Performance, Radically Open
Today IFM is releasing K2 Horizon, a connected fleet of six models: 375B-A23B, 36B-A4B, 32B, 7B, 3.7B, and 0.9B. Across reasoning, mathematics, coding, agentic tasks, and general capabilities, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B, and 7B models setting new state of the art at their respective scales.
K2 Horizon is also our most comprehensive open release to date. For every model, we are opening the training lifecycle from pretraining through reasoning and agentic post-training. We are releasing intermediate checkpoints, training data or detailed data-construction recipes, open architecture, mixture compositions, training code, configurations, fine-grained logs, evaluation results, and final weights.
The models and code are released under the Apache 2.0 license. Datasets are released under their applicable licenses, such as ODC-BY; We disclose how the data was constructed and mixed when redistribution is not possible.
Together, K2 Horizon represents the most comprehensive open model release to date:
- A new performance frontier across scales. The 0.9B, 3.7B, and 7B models achieve world-leading performance in their size classes across widely used evaluations. The 36B-A4B model, equipped with our new Mixture-of-Value-Attention (MoVA) mechanism, delivers exceptional capability per active parameter, outperforming some much larger models. The 32B and 375B-A23B models rank among the top models in their respective classes. Together, the six models provide competitive performance across deployment environments ranging from edge devices to the enterprise.
- The first fully open model fleet for agents. K2 Horizon is the first open model family to expose the complete development process through agentic post-training. By releasing checkpoints, data (or data recipe), code, configurations, and training logs across every stage, K2 Horizon makes it possible to study how reasoning, tool use, planning, and agentic capabilities emerge; reproduce the methods that create them; and adapt those methods to new tools, environments, and domains.
- Six models spanning edge to enterprise. The 0.9B model is designed for highly constrained environments such as watches and glasses, while the 3.7B and 7B models bring advanced capabilities to phones and other on-device applications. The dense 32B model and sparse 36B-A4B model provide powerful options for local workstations and efficient serving. The 375B-A23B model brings the fleet’s strongest capabilities to demanding enterprise deployments. All six models include quantization support.
- One connected fleet. The six models share core architecture, vocabulary, training methodology, interfaces, evaluation infrastructure, and deployment tooling, with a smaller vocabulary for the 0.9B model. This consistency also makes it easier to move between sizes, route work dynamically, and study capability and efficiency across scale.
World-leading performance across the scales
The 0.9B, 3.7B, and 7B models achieve state-of-the-art results in their respective classes across mathematics, reasoning, general capability, coding, and agentic tasks.
The 36B-A4B model performs beyond the level normally expected from its active parameter count, demonstrating the efficiency of our unique Mixture-of-Expert design when computing attention values. The 32B and 375B-A23B models place among the top models in their respective comparison classes.
The small models are especially notable. K2 Horizon 0.9B achieves an AIME 2026 score above 48, along with strong reasoning, tool-use, and agentic capabilities. K2 Horizon 3.7B and 7B extend these capabilities to more demanding software-engineering and multi-step environments, demonstrated on strong performance in SWE-bench and BrowseComp. Although complex tasks that require extensive exploration and repeated recovery, such as those in TerminalBench, remain difficult for the smallest models, K2 Horizon moves the boundary of what is possible at every scale.
Why the Horizon Fleet matters
A transparent model that falls far behind the capability frontier has limited value as a foundation, even for research. At the same time, a powerful model released only as final weights allows people to run it, but provides little insight into how its capabilities were created.
K2 Horizon brings these two together. The fleet provides highly competitive models and releases the recipes used to train them. Researchers can study advanced capabilities in models strong enough to exhibit them, while developers can reproduce, adapt, and extend the methods rather than treating the final checkpoint as an opaque starting point.
Since introducing the fully open principle in our 2023 LLM360 paper, we have released open models every year while extending that commitment to larger scales, stronger capabilities, and now the complete lifecycle through agentic post-training.
A Deep Dive into The K2 Horizon Fleet
K2 Horizon 375B-A23B: the enterprise powerhouse
K2 Horizon 375B-A23B is the fleet’s largest and most capable model. Its sparse MoE architecture provides 375 billion parameters of total capacity while activating approximately 23 billion parameters for each token, allowing it to draw on the capacity of a much larger model without using every parameter for every token.
The model ranks among the top models below 400 billion parameters across general, reasoning, coding, and agentic evaluations. It is designed for demanding workloads where model quality matters most, including complex reasoning, software engineering, research, and long-horizon agentic tasks.
Like every model in the Horizon fleet, 375B-A23B is released not as a single endpoint but as a development tree. Its intermediate checkpoints and post-training branches expose how the base model develops into reasoning, instruction-following, and specialized agentic variants.
K2 Horizon 32B and 36B-A4B: strong performance for local deployment
Horizon 32B is the fleet’s most powerful dense model, providing a strong balance of capability, adaptability, and local deployability. It ranks among the top dense models below 40 billion parameters.
Horizon 36B-A4B reaches nearly the performance of the dense 32B model while activating only approximately 4 billion parameters per token. Its efficiency comes from MoVA, our new sparse attention architecture, together with MoE feed-forward layers.
These two models serve as an important reference point for studying how dense and sparse architectures behave under similar training conditions.
These models occupy the fleet’s local performance sweet spot. They are powerful enough for demanding reasoning, coding, and agentic applications while remaining practical for local workstations and efficient serving systems.
K2 Horizon 7B, 3.7B, and 0.9B: frontier capability at small scale
k2 Horizon 7B and 3.7B deliver strong reasoning, mathematics, coding, tool-use, and agentic performance while remaining suitable for local and on-device deployment. On several evaluations, their results approach or exceed those of models many times larger from the previous generation.
K2 Horizon 0.9B carries many of the same capabilities into highly constrained environments. It can perform mathematical reasoning, use tools, and complete simple agentic tasks while remaining compact enough for applications on watches, glasses, and other edge devices under quantization.
The appropriate task changes with scale: the 0.9B model is best suited to focused interactions and lightweight tool use, while the 3.7B and 7B models can handle more demanding coding and multi-step workflows. Together, they demonstrate how much capability can now be retained in models small enough to run almost anywhere.
Designing K2 Horizon
One family from the beginning
Horizon was designed as a connected family rather than a collection of unrelated models. The six models share core architectural decisions, training methodology, interfaces, evaluation infrastructure, and deployment tooling. This allows developers to move between sizes more easily and gives researchers a more controlled setting for studying capability across scale.
Each model is pretrained on approximately 20 trillion tokens using carefully constructed and documented mixtures. Intermediate checkpoints and their corresponding fine-grained logs are captured throughout training, creating a detailed record of how each model develops.
MoVA: scaling attention with sparse experts
The core idea behind mixture-of-experts is to increase total model capacity while keeping the computation required for each token roughly fixed. Conventional MoE architectures apply this sparsity primarily to feed-forward layers: many specialized experts are available, but a router activates only a small subset for each token.
Our new architecture, MoVA—Mixture-of-Value Attention, extends this principle to attention. Because attention determines how a transformer brings together information from across its context, introducing sparsity there opens another dimension for scaling model capacity beyond the feed-forward network.
MoVA integrates expert routing into multi-head attention while remaining compatible with efficient techniques including FlashAttention, grouped-query attention, and sparse attention.
The result is K2 Horizon MoVA 36B-A4B: a model with 36 billion total parameters but approximately 4 billion active parameters per token. Under the same training conditions, it performs only slightly below the dense Horizon 32B model while requiring substantially fewer active parameters.
Training Data
Training across six scales requires data that is broad enough to support general capability and carefully constructed enough to maintain quality over approximately 20 trillion tokens.
Horizon’s pre-training mixture combines diverse web, code, mathematical, scientific, multilingual, and domain-specific sources with synthetic data generated through our own pipelines. One of our key data innovations is the incorporation of reasoning directly into pre-training: nearly 17% of the pre-training corpus consists of problem-solving trajectories with explicit reasoning. Reasoning trajectories for mathematical tasks were further rewritten into formats such as dialogues and study guides. In total, we used approximately 10 trillion synthetic tokens during pre-training.
Our synthetic data pipelines use millions of combinations of diversity knobs and context seeds, including retrieval from an internally built search engine over the pre-training web corpus. To quantify diversity at corpus scale, we developed a custom gzip-based compression metric with adaptive striding to avoid the rapid saturation of standard gzip-based measurements as the number of documents grows. As shown below, the measured diversity of our synthetic data approaches that of high-quality natural web text and substantially exceeds that of web code.
We document the data composition, synthesis pipelines, and construction of the training mixture. Following our open-source principles, where licenses permit, we release the training-ready datasets directly. Where redistribution is restricted, we release source descriptions, filtering and construction methods, and mixture composition. This allows researchers to understand what each model learned from and how the data distribution evolved throughout training.
Our post-training data is introduced from the beginning of mid-training, rather than being reserved exclusively for the final stages of training. We combine synthesized long-context documents with instruction-following, reasoning, and agentic trajectories formatted with the chat template. A main pillar of our post-training data is large-scale task synthesis grounded in task taxonomies, diversity knobs, and web-search seeding, resulting in over 100 million unique tasks. In addition, we developed sampling techniques that guide solver LLMs toward correct solutions and desired behaviors when generating training trajectories.
Pre-training Dynamics
Every K2 Horizon checkpoint is accompanied by detailed training logs and fine-grained histories that expose the real dynamics of large-scale training: how losses evolve, where instabilities appear, how interventions affect training, and when capabilities begin to emerge.
As an example of why these records are useful, K2 Horizon 3.7B, 7B, 32B, and 36B-A4B were trained on exactly the same 22 trillion tokens. When aligned by training progress and normalized by the median loss over the final 1% of training, their raw loss trajectories approximately collapse, which you can see in the figure below—even across dense and sparse architectures and nearly an order of magnitude in total parameter count. Although this normalization aligns their endpoints by construction, it does not force the early and intermediate trajectories to coincide. Their close agreement shows that, under shared data and a common recipe, the shape of learning remained remarkably consistent across the fleet subset that used the same dataset throughout training.
Together, the checkpoints and logs allow researchers to investigate why some dynamics transfer across scale and architecture while others diverge, and to connect those differences with particular training stages, data mixtures, and technical decisions.
Post-training for reasoning and agents
Most of K2 Horizon’s advanced reasoning and agentic capabilities emerge during post-training. The complete pipeline includes mid-training, supervised fine-tuning, model merging, reinforcement learning with specialized agent training. Rather than producing only one final chat model, this process creates a development tree. Different branches specialize in reasoning, coding, tool use, and agentic domains while remaining connected to common base checkpoints.
We release the artifacts across these stages so researchers can study where each capability emerges, reproduce individual branches, and adapt the same methods to new tools and environments.
Uno Diffusion: plug-and-play lossless speedup for Horizon
Inference speed has become a first-class optimization target, especially in the era of reasoning models and agents. As models produce longer chains of thought and take more actions, even small per-token delays compound into substantial latency. Yet autoregressive language models generate one token at a time, creating a fundamental bottleneck. Existing approaches offer partial solutions: speculative decoding typically requires a separately trained draft model, while discrete diffusion enables parallel generation but often sacrifices quality for speed.
Uno is designed to remove that tradeoff. It provides a lossless inference speedup, accelerating generation without degrading response quality. Uno keeps Horizon’s autoregressive parameters frozen and fully responsible for the model’s output distribution, while a lightweight set of diffusion parameters learns only how to generate more efficiently.
Through a process we call Diffusion Distillation, these compact adapters learn to generate blocks of tokens in parallel. The result combines the quality guarantees of autoregressive decoding with the speed advantages of diffusion: the model reaches the same answers, but reaches them faster.
Across our evaluations, Uno achieves a better speed–quality tradeoff than leading speculative-decoding systems and both open-weight and proprietary diffusion language models. Crucially, its gains persist across every batch size we tested, enabling lower latency for interactive agents and higher throughput for large-scale serving, with no loss in model quality and negligible additional training overhead. Uno is delivered as a simple LoRA adapter, making this lossless speedup easy to adopt: all you need is to attach the adapters.
Open infrastructure for building and extending Horizon
We are releasing the infrastructure used to build Horizon alongside the models themselves. The goal is not only to make the fleet reproducible, but to make its development stack useful as a foundation for new models and systems.
At the center of this release is xLLM, our production-tested training infrastructure. xLLM combines large-scale training performance with the flexibility required for research, allowing teams to modify architectures, data mixtures, training stages, and objectives without rebuilding the surrounding system.
We will be also releasing our full agentic post-training code base, including Reinforcement Learning, which we hope will help researchers to advance the techniques in related areas.
Researchers can use these components to inspect training behavior or reproduce a particular stage. Developers can adapt Horizon to a domain, add new tools, train agent experts, or deploy several model sizes behind a unified interface.
From Open Source to Open Science
Intermediate checkpoints turn model development into an observable scientific process. They allow researchers to study when capabilities emerge, how training choices change behavior, and when unintended strategies first appear.
Reward hacking provides one revealing example. When a capable model is placed in a realistic computer environment and told to solve a complex task, the same resourcefulness that makes it useful can also lead it to exploit shortcuts in the evaluation itself. To understand how much this affects K2 Horizon's reported numbers, we audited the released model using Artificial Analysis's reward hacking auditing procedure.
TerminalBench 2.1 places models in sandboxed computer environments and evaluates whether they can complete complex technical tasks. A capable model must explore files, invoke tools, diagnose failures, and find alternative paths toward a solution. That resourcefulness is exactly what we want. But occasionally it crosses a boundary: instead of solving the intended problem, the model locates a hidden answer, exploits something the task left exposed, or manipulates the grader itself.
We ran K2 Horizon 375B-A23B on 89 TerminalBench 2.1 tasks with eight attempts each, producing 712 trials. Of these, 500 passed the task verifier (70.2% reported accuracy). We then audited every passing trial using Artificial Analysis's reward hacking auditing procedure, applying their harbor analyze tool with the reward_hacking criterion and their full rubric text verbatim, with Codex gpt-5.6-sol as the judge model.
The audit flagged 24 trials across 10 tasks. Removing them lowers the accuracy from 70.2% to 66.9%, a correction of 3.37 percentage points. The remaining 79 tasks were fully clean. For context, Artificial Analysis reports flag rates of 2.2% for claude Fable 5 and 4.1% for GPT-5.6 Luna; K2 Horizon's 3.37% falls within that range.
The model discovered several strategies:
- Inferring it was inside a public benchmark, finding the repository on GitHub, and downloading the reference solution
- Pulling the current source from a real project's public repository and copying the fix rather than deriving it
- Inspecting unadvertised files, generator scripts, or exposed credentials
- Editing the test harness or crafting output that exploited how the test checked success
We observed a related case with K2 Horizon 7B, which found and downloaded SWE-bench answers and consequently produced an inflated score of 82. The score does not represent genuine software-engineering performance, but the behavior is scientifically revealing: benchmark hacking emerged as an unintended consequence of broader planning, tool use, environment exploration, and persistence.
Because we release intermediate checkpoints alongside the final models, these behaviors can be studied rather than hidden. Researchers can determine when a strategy first appears, connect it to changes in training, and measure its effect on reported performance.
K2 Horizon is therefore more than a collection of model weights. It is an open experiment in how capabilities—and their unintended consequences—develop throughout training.
Get Started with K2 Horizon Today!
All six K2 Horizon sizes are released as open weights under Apache 2.0, with day-zero support from vLLM, SGLang, and Ollama. Further, K2 Horizon supports deployment on NVIDIA, AMD, and Cerebras hardware, providing options from local inference to large-scale serving. Get the models from our repository: https://huggingface.co/IFM, download the models, inspect intermediate checkpoints, study training data and mixtures, reproduce training stages, or build new models and agents using the Horizon code and infrastructure.
K2 Horizon provides more than six final models. It provides an open blueprint for understanding, adapting, and advancing the next generation of AI systems.
Full Result Table
375B-A23B
36B-A4B
32B
K2 Horizon 7B
K2 Horizon 3.7B
K2 Horizon 0.9B