# Kimi K3 技术报告发布：百万 token 上下文与 2.5 倍效率提升

- 来源：Rohan Paul (@rohanpaul_ai)
- 发布时间：2026-07-28 02:37
- AIHOT 分数：57
- AIHOT 链接：https://aihot.virxact.com/items/cms3l8tg30071roskx2v8jmal
- 原文链接：https://x.com/rohanpaul_ai/status/2081811110504882195

## AI 摘要

Kimi K3 发布技术报告，这是一款 2.8T MoE 模型，原生支持视觉理解与百万 token 上下文窗口。其架构和训练方案相比 Kimi K2 将整体扩展效率提升了约 2.5 倍，通过 Kimi Delta Attention、NoPE、跨深度注意力残差及 896 专家 MoE 等设计实现。模型权重、技术报告及配套高性能内核与基础设施已开源。

## 正文

Kimi K3 just released their technical paper.

one of the most detailed and exhaustive one.

A million-token context window does not create long-horizon agency if the training system cannot preserve a task across thousands of tool calls.

Kimi K3 is built around that constraint.

Reports that the architecture and training recipe improve overall scaling efficiency by roughly 2.5X over Kimi K2.

- Most layers no longer use full softmax attention. Each block runs 3 Kimi Delta Attention （KDA） layers and one global Multi-head Latent Attention （MLA） layer；

- No Position Encoding （NoPE）： removes Rotary Position Embedding （RoPE） extension tricks， while bounded KDA decay makes the difficult causal tiles Tensor Core-friendly.

- Their agents are trained not to depend on one fixed scaffold. Tools， prompts， memory， context management， skills， and subagents are varied during RL， while partial rollouts and resumable microVM sandboxes allowed unfinished trajectories continue across training iterations with their environment state intact.

- The model attends across depth， not only across tokens. Attention Residuals let every layer selectively retrieve earlier block representations instead of squeezing the entire network history through one accumulated residual stream.

- The million-token context was trained as an actual dependency problem. The curriculum grows from 8K to 64K， then 256K to 1M， using synthetic multimodal tasks whose evidence is deliberately scattered across the full sequence.

- The 896-expert MoE required a complete stability redesign. Kimi K3 combines a half-width latent expert path， 16 active experts per token， RMSNorm， bounded SiTU-GLU activations， and Quantile Balancing that directly computes the routing bias needed for each expert's target load.

### 引用推文

> Kimi.ai：Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token...