伴随 Kimi K2.6 模型的发布，我们同步开源了 Kimi Vendor Verifier（KVV）项目，旨在帮助开源模型用户验证其推理实现的准确性。
这并非事后补救，而是因为我们从实践中深刻认识到：开源模型只是成功的一半，另一半在于确保它在任何地方都能正确运行。
官方评测结果
思考模式
非思考模式
| 基准测试 | 指标 | 温度 | TopP | 最大 Token 数 | Kimi API |
|---|---|---|---|---|---|
OCRBench | 准确率 | 1.0 | 0.95 | 16384 | 91.0 |
AIME2025 | avg@32 | 1.0 | 0.95 | 98304 | 98.4 |
MMMU Pro Vision | 准确率 | 1.0 | 0.95 | 65536 | 78.8 |
您可以点击此处访问 Kimi API K2VV 评测结果，用于计算 F1 分数。
我们为何构建 KVV
从孤立事件到系统性问题
自 K2 Thinking 发布以来，我们频繁收到社区关于基准测试分数异常的反馈。经调查确认，相当一部分案例源于解码参数使用不当。为立即缓解此问题，我们在 API 层面构建了第一道防线：在思考模式下强制设定 Temperature=1.0 和 TopP=0.95，并强制验证思考内容是否正确回传。
然而，更细微的异常很快触发了我们的警报。在 LiveBenchmark 的一项特定评测中，我们观察到第三方 API 与官方 API 之间存在显著差异。经过对多家基础设施提供商的广泛测试，我们发现这种差异普遍存在。
这暴露了开源模型生态中一个更深层的问题：权重越开放，部署渠道越多样化，质量就越难以控制。
如果用户无法区分"模型能力缺陷"与"工程实现偏差"，对开源生态的信任将不可避免地崩塌。
我们的解决方案
六项关键基准测试（精选以暴露特定基础设施故障）：
- 预验证：验证 API 参数约束（temperature、top_p 等）是否被正确执行。所有测试必须通过，才能进入基准评测阶段。
- OCRBench：多模态管线的 5 分钟冒烟测试。
- MMMU Pro：通过测试多样化的视觉输入，验证视觉输入预处理。
- AIME2025：长输出压力测试。能够捕捉短基准测试所隐藏的 KV 缓存错误和量化精度下降问题。
- K2VV 工具调用：衡量触发一致性（F1 值）和 JSON Schema 准确性。工具错误会在智能体中累积放大；我们能够及早发现它们。
- SWE-Bench：完整的智能体编码测试。（由于沙箱依赖，未开源）
上游修复：我们与 vLLM/SGLang/KTransformers 社区合作，修复根本原因，而不仅仅是检测症状。
发布前验证：我们提供模型的早期访问测试，而不是等待部署后的用户投诉。这使得基础设施提供商能够在用户遇到问题之前验证其技术栈。
持续基准测试：我们将维护一个公开的供应商结果排行榜。这种透明度鼓励供应商优先考虑准确性。
测试成本估算
我们在两台 NVIDIA H20 8-GPU 服务器上完成了完整的评估工作流验证，顺序执行耗时约 15 小时。为提高评估效率，脚本已针对长时推理场景进行了优化，包括流式推理、自动重试和检查点恢复机制。
公开邀请
权重是开放的。正确运行它们的知识也必须开放。
我们正在扩大供应商覆盖范围，并寻求更轻量的智能体测试。联系我们：contact-kvv@kimi.com
Alongside the release of the Kimi K2.6 model, we are open-sourcing the Kimi Vendor Verifier (KVV) project, designed to help users of open-source models verify the accuracy of their inference implementations.
Not as an afterthought, but because we learned the hard way that open-sourcing a model is only half the battle. The other half is ensuring it runs correctly everywhere else.
Official Evaluation Results
Think
Non-Think
| Benchmark | Metric | Temperature | TopP | MaxTokens | Kimi API |
|---|---|---|---|---|---|
OCRBench | acc | 1.0 | 0.95 | 16384 | 91.0 |
AIME2025 | avg@32 | 1.0 | 0.95 | 98304 | 98.4 |
MMMU Pro Vision | acc | 1.0 | 0.95 | 65536 | 78.8 |
You can click here to access the Kimi API K2VV evaluation results for calculating the F1 score.
Why We Built KVV
From Isolated Incidents to Systemic Issues
Since the release of K2 Thinking, we have received frequent feedback from the community regarding anomalies in benchmark scores. Our investigation confirmed that a significant portion of these cases stemmed from the misuse of Decoding parameters. To mitigate this immediately, we built our first line of defense at the API level: enforcing Temperature=1.0 and TopP=0.95 in Thinking mode, with mandatory validation that thinking content is correctly passed back.
However, more subtle anomalies soon triggered our alarm. In a specific evaluation on LiveBenchmark, we observed a stark contrast between third-party API and official API. After extensive testing of various infrastructure providers, we found this difference is widespread.
This exposed a deeper problem in the open-source model ecosystem: The more open the weights are, and the more diverse the deployment channels become, the less controllable the quality becomes.
If users cannot distinguish between "model capability defects" and "engineering implementation deviations," trust in the open-source ecosystem will inevitably collapse.
Our Solution
Six Critical Benchmarks (selected to expose specific infra failures):
- Pre-Verification: Validates that API parameter constraints (temperature, top_p, etc.) are correctly enforced. All tests must pass before proceeding to benchmark evaluation.
- OCRBench: 5 minutes smoke test for multimodal pipelines.
- MMMU Pro: Verify Vision input preprocessing by testing diverse visual inputs.
- AIME2025: Long-output stress test. Catches KV cache bugs and quantization degradation that short benchmarks hide.
- K2VV ToolCall: Measures trigger consistency (F1) and JSON Schema accuracy. Tool errors compound in agents; we catch them early.
- SWE-Bench: Full agentic coding test. (Not open sourced due to dependency of sandbox)
Upstream Fix: We embed with vLLM/SGLang/KTransformers communities to fix root causes, not just detect symptoms.
Pre-Release Validation: Rather than waiting for post-deployment complaints, we provide early access to test models. This lets infrastructure providers validate their stacks before users encounter issues.
Continuous Benchmarking: We will maintain a public leaderboard of vendor results. This transparency encourages vendors to prioritize accuracy.
Testing Cost Estimation
We completed full evaluation workflow validation on Two NVIDIA H20 8-GPU servers, with sequential execution taking approximately 15 hours. To improve evaluation efficiency, scripts have been optimized for long-running inference scenarios, including streaming inference, automatic retry, and checkpoint resumption mechanisms.
An Open Invitation
Weights are open. The knowledge to run them correctly must be too.
We are expanding vendor coverage and seeking lighter agentic tests. Contact Us: contact-kvv@kimi.com