Kimi K3 Defines the Open Frontier Intelligence
Moonshot AI released Kimi K3 with a 47-page technical report that is, quite honestly, one of the most information-dense documents the open model community has produced. It is not a marketing sheet with a benchmark table stapled to the end. It is a full account of an architecture, a training recipe, a post-training pipeline, and, most unusually, the systems engineering that made all of it run.
The headline: 2.8 trillion total parameters, 104 billion activated, 1 million token context, native vision, open weights. That makes K3 the first open model in the 3T class, at a moment when most of the open ecosystem had settled comfortably into the 1T neighborhood.
But the parameter count is the least interesting thing about this report. What makes it worth reading closely is that Moonshot pushed on both scaling axes at once, and then wrote down, in detail, everything that broke when they did.
Read the full technical report
The Thesis: Two Axes, Not One
The introduction frames the problem cleanly. For most of the LLM era, scaling meant pre-training compute: bigger models, more data. Then reasoning models established test-time compute as a second axis, and the open ecosystem chased it hard. DeepSeek-R1, Kimi K1.5, and everything after showed that large-scale RL can pull sophisticated reasoning out of a strong base model.
The problem, as the Kimi team puts it, is that open models advanced rapidly on the second axis while stalling on the first. Increasingly sophisticated RL recipes kept getting applied to pre-trained foundations of roughly the same size. If everyone runs better and better post-training on similarly sized bases, open progress converges while the gap to the strongest proprietary systems widens.
K3’s answer is to move both together: scale the pre-trained foundation to 3T-class parameters and scale RL, reasoning effort, and long-horizon interaction at 1M context. The architecture is organized around a single idea expressed three ways: scale information flow along sequence length, network depth, and model width.
- Sequence length: Hybrid attention, three Kimi Delta Attention layers to one Gated MLA layer
- Depth: Attention Residuals, so each layer selectively retrieves from all preceding layers
- Width: Stable LatentMoE, activating 16 of 896 routed experts per token
Together with refined data and training recipes, these deliver roughly a 2.5× improvement in scaling efficiency over Kimi K2. That number is doing a lot of work in this report, and it is worth understanding where it comes from.
Architecture
The Kimi K3 architecture, organized around token, channel, and layer mixing, with a native vision pathway at the input. Each block contains three KDA layers followed by one Gated MLA layer, each paired with a Stable LatentMoE feed-forward network. Attention Residuals use learned pseudo-queries (w) to derive attention weights (α) over the embedding and preceding block outputs. Top left: Stable LatentMoE with shared and routed experts. Bottom left: the KDA module. Bottom right: the native vision pathway. Source: Kimi K3 technical report, Figure 2.
Here is the shape of the model, next to K2 for context:
| Spec | Kimi K2 | Kimi K3 | Change |
|---|---|---|---|
| Layers | 61 | 93 | +52% |
| Total parameters | 1.04T | 2.78T | +167% |
| Activated parameters | 32.6B | 104.2B | +220% |
| Hidden dimension | 7,168 | 7,168 | same |
| Latent MoE dimension | n/a | 3,584 (0.5×) | new |
| MoE hidden dim per expert | 2,048 | 3,072 | +50% |
| Routed experts | 384 | 896 | +133% |
| Active experts per token | 8 | 16 | +100% |
| Shared experts | 1 | 2 | +100% |
| Attention heads | 64 | 96 | +50% |
| Vocabulary | 160K | 160K | same |
| Training context | 128K | 1M | 8× |
| Attention | MLA | Hybrid KDA + MLA | new |
| Attention composition | 61 MLA | 69 KDA + 24 MLA | new |
| Activation | SwiGLU | SiTU-GLU | new |
| Vision encoder | n/a | MoonViT-V2, 401M | new |
Notice what did not change: hidden dimension stayed at 7,168, vocabulary stayed at 160K, and the model still has exactly one dense layer and one MTP layer. The growth went into depth (93 layers), expert count, and the number of experts fired per token. This is a deliberate bet on sparsity and depth over width.
Hybrid Attention: 3 KDA + 1 Gated MLA
Each block contains three Kimi Delta Attention layers followed by one Gated MLA layer, a 3:1 ratio repeated throughout the backbone. An extra Gated MLA layer sits at the very end, guaranteeing the final layer performs global attention.
The division of labor is clean: KDA handles efficient long-sequence token mixing with a fixed-size recurrent state, while the periodic MLA layers preserve unrestricted global content interaction. You get most of linear attention’s cost profile without giving up the ability to attend anywhere.
Kimi Delta Attention
KDA extends the delta rule with a channel-wise forget gate. For hidden states $x_t$, with query and key $q_t, k_t \in \mathbb{R}^{d_k}$, value $v_t \in \mathbb{R}^{d_v}$, and recurrent state $S_t \in \mathbb{R}^{d_k \times d_v}$:
$$S_t = \left(I - \beta_t k_t k_t^\top\right)\text{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^\top, \qquad \tilde{o}_t = S_t^\top q_t$$Here $\alpha_t \in (0,1)^{d_k}$ is a channel-wise retention factor (each key channel forgets at its own rate) and $\beta_t \in (0,1)$ controls the delta-rule write strength. The projections apply a short convolution followed by Swish, with L2 normalization on queries and keys.
K3 makes two changes to KDA relative to Kimi Linear, and the first one is a lovely example of numerics driving architecture.
Lower-bounded decay. The chunkwise parallel form of KDA rescales keys within each chunk by the reciprocal cumulative decay $1/\Gamma^{1 \to C}_{[t]}$. Since $\Gamma$ is a product of factors in $(0,1)$, that reciprocal can grow without bound and overflow in finite precision. Kimi Linear worked around this by computing relative decay in log space and splitting chunks into 16-token tiles, but the diagonal tiles still needed explicit position-pair computation, which was the main intra-chunk bottleneck.
K3 fixes it at the source by changing the mapping from decay logits to log-decay. Instead of the unbounded negative-Softplus used by GDN, Mamba-2, and Kimi Linear, K3 uses a scaled sigmoid:
$$g^h_t = g_{\min}\,\text{Sigmoid}\!\left(e^{A_h} z^h_t\right) \in (g_{\min}, 0)^{d_k}, \qquad \alpha^h_t = \exp(g^h_t)$$with $A_h$ a learnable per-head log-scale and $g_{\min} = -5$ fixed. Every retention factor now satisfies $\alpha^h_{t,j} > e^{-5} \approx 6.7 \times 10^{-3}$, so cumulative log-decay over a 16-token tile lies in $(-80, 0)$ and the reciprocal rescaling factor stays under $e^{80}$, comfortably inside BF16’s dynamic range.
The payoff is architectural: with the range bounded, both diagonal and off-diagonal tiles can use dense Tensor Core matmuls, eliminating the position-pair diagonal path entirely. A one-line change to an activation function removes a kernel bottleneck.
Full-rank output gate. K3 replaces KDA’s low-rank output gate with an input-dependent full-rank projection applied after head-wise RMSNorm:
$$y_t = W_o\left[\text{Sigmoid}(W_g x_t) \odot \text{RMSNorm}(\tilde{o}_t)\right]$$Gated MLA with NoPE
The global layers keep Multi-head Latent Attention from DeepSeek-V2, which compresses each token’s key and value into a low-dimensional latent and reconstructs them at attention time. Two departures from K2:
First, No Position Encoding on all MLA layers. No RoPE, no positional signal at all on queries or keys. The intervening KDA layers already provide position-sensitive, recency-aware mixing through their gating and decay, so the MLA layers are free to do pure global content interaction. The practical consequence is significant: extending context requires no positional-encoding surgery, no RoPE base retuning, no YaRN interpolation. K3 extrapolates to 1M tokens directly.
Second, MLA also gets the same input-dependent full-rank output gate as KDA.
There is also a small precision detail worth flagging, because it is the kind of thing that usually never makes it into a paper: to correct biased rounding error in flash attention, K3 keeps the attention output in FP32 during training. That doubles the on-chip footprint of the output tile, so they redesigned the training kernel to overlap it with the KV staging buffers instead of the query tile, freeing shared memory for a deeper KV pipeline.
Attention Residuals
I wrote about Attention Residuals when the paper came out, so I will keep this brief: K3 is the production deployment of that idea.
Standard residual connections compress everything into one state as depth increases, a bottleneck structurally identical to what RNNs suffered over time. AttnRes applies the Transformer’s own fix to the depth dimension. Each layer gets a learnable pseudo-query $w_l$ and attends over all preceding layer outputs with a softmax kernel:
$$\alpha_{i \to l} = \frac{\phi(q_l, k_i)}{\sum_{j=0}^{l-1}\phi(q_l, k_j)}, \qquad h_l = \sum_{i=0}^{l-1}\alpha_{i \to l} \cdot v_i$$where $\phi(q,k) = \exp(q^\top \text{RMSNorm}(k))$. The RMSNorm on keys is what prevents large-magnitude layers from swamping the weights.
At 93 layers the full $O(L^2 d)$ arithmetic is affordable, but the $O(Ld)$ memory (and cross-stage communication under pipeline parallelism) is not. So K3 uses Block AttnRes: layers are partitioned into blocks, outputs are summed within a block, and full softmax attention runs across only the block representations. Memory and communication drop from $O(Ld)$ to $O(Nd)$.
K3 uses 8 blocks of 12 layers, giving a partial final block and 9 total blocks counting the embedding. The embedding is always included as a source, which means every layer in the network has direct, weighted access to the raw token representation.
Stable LatentMoE
This is the section where you can feel the 2.8T scale pushing back.
Conventional MoE sends the full $d$-dimensional token to every selected expert, so communication and expert-weight traffic scale with routing multiplicity. Doubling active experts from 8 to 16 would have doubled that traffic. LatentMoE decouples model width from routed-expert width: shared experts keep a full-width path, while routed experts operate in a compact latent space of width $\ell$ (3,584 here, exactly half the hidden dimension).
$$u = \sum_{i \in T_k(x)} p_i E^{\text{routed}}_i(W_\downarrow x), \qquad y = \sum_{j=1}^{N_s} E^{\text{shared}}_j(x) + W_\uparrow \text{RMSNorm}(u)$$That is how you get to 896 experts with 16 active, a sparsity ratio of 56, without the communication bill exploding.
But extreme sparsity at 2.8T scale amplified two failure modes, and the fixes for both are interesting.
Failure mode 1: activation explosion. The routed path composes $W_\downarrow$, a gated multi-branch expert FFN, and $W_\uparrow$ into a chain of nearly four consecutive matmuls. That ill-conditioned structure at 2.8T scale produced exploding internal activations. Two fixes:
RMSNorm before the up-projection (visible in the equation above). The aggregated routed representation $u$ varies in scale with which experts fired and how they were weighted, so normalizing it before combining with the shared branch reduces the routed path’s scale sensitivity. Beyond stability, this consistently improved validation loss and downstream benchmarks.
SiTU-GLU, a new activation. Both multiplicative factors in SwiGLU are unbounded, so coincident large coordinates produce outliers and overflow risk in low precision. Sigmoid Tanh Unit GLU applies a smooth cap $\text{softcap}(x, \beta) = \beta \tanh(x/\beta)$ to the linear factor of the Swish gate and independently to the up branch:
$$\text{SiTU-GLU}(x) = \left[\beta_1 \tanh\!\left(\tfrac{W_g x}{\beta_1}\right) \odot \text{Sigmoid}(W_g x)\right] \odot \beta_2 \tanh\!\left(\tfrac{W_u x}{\beta_2}\right)$$With $\beta_1 = 4$ and $\beta_2 = 25$, every output coordinate is bounded: $\|\text{SiTU-GLU}(x)\|_\infty \leq \beta_1 \beta_2 = 100$. Near the origin the scaled tanh satisfies $\beta \tanh(z/\beta) = z + O(z^3/\beta^2)$, so SiTU-GLU matches SwiGLU to first order and recovers it exactly as $\beta_1, \beta_2 \to \infty$. The appendix notes they compared against hard clamping of gate pre-activations and found the smooth cap trains better, because it preserves nonzero gradients away from the saturation boundary.
Failure mode 2: load balancing at 896 experts. Auxiliary-loss-free routing adds a per-expert bias $b_j$ to the router score before Top-$k$ selection, with $b$ excluded from the mixture weights so it regulates dispatch without touching gradients. DeepSeek’s original update rule is a fixed-step sign update, $b^{(t+1)}_j = b^{(t)}_j + \gamma\,\text{sign}(\bar{\ell} - \ell^{(t)}_j)$, where $\gamma$ trades slow adaptation against oscillation. At nearly $10^3$ experts per layer, that regime breaks down.
Quantile Balancing replaces the heuristic with something exact. The insight: run Top-$(k{+}1)$ instead of Top-$k$ on the biased score. The first $k$ entries are the actual routes; the $(k{+}1)$-th entry is precisely the cutoff $\alpha^{(t)}_i$ that an expert must exceed to enter token $i$’s Top-$k$. Given those cutoffs, the token count reaching expert $j$ under a candidate bias is monotonically decreasing in the threshold, so setting it to the target load $q = mk/n$ makes the answer a quantile:
$$\hat{b}^{(t+1)}_j \leftarrow -\text{quantile}_{1-k/n}\left(s_{:,j} - \alpha^{(t)}\right), \qquad b^{(t+1)} \leftarrow \hat{b}^{(t+1)} - \text{mean}\!\left(\hat{b}^{(t+1)}\right)$$Appendix C derives this from the maximum-score balanced assignment LP, whose relaxation is exact by integrality of the bipartite b-matching polytope. The bias is the dual variable. It is a genuinely satisfying result: a load-balancing heuristic turns out to be an approximate dual solve, and computing it properly is cheaper than tuning the heuristic.
The practical obstacle is that this quantile spans the full global batch, millions of margins spread across ranks and gradient-accumulation steps. Gathering them is not viable. So K3 estimates it from a per-expert histogram: each rank scatter-adds its local values into a count matrix, one integer all-reduce pools them at step end, and the quantile is read off the pooled counts with linear interpolation inside the selected bin. Because counts are additive, the histogram is exactly invariant to how tokens shard across ranks, so the estimate is the true global-batch quantile rather than an average of per-rank quantiles. With $B = 1000$ bins the error is at most a few $10^{-3}$, communication is under 1% of the natural alternative, and they report no measurable residual imbalance.
Native Vision: MoonViT-V2
K3 is natively multimodal. Text, images, and video go through one shared backbone in one context, with no post-hoc alignment stage. The architectural payoff shows up in agentic work: rendered outputs and the code that produced them live in the same token stream, so the model can write code, look at a screenshot of the result, and iterate, with no cross-model handoff.
The notable departure from K2.5 is that MoonViT-V2 is trained entirely from scratch with next-token prediction, not initialized from a contrastively pre-trained encoder like SigLIP.
The stated reason is stability, and they show the evidence: the SigLIP-initialized MoonViT-3D shows persistently higher vision-tower gradient norms with frequent spikes, while the from-scratch MoonViT-V2 stays smooth throughout training. Training under the language-modeling objective also lets the encoder’s representations be shaped by what the LM actually needs, rather than by a contrastive loss that favors global semantics over fine-grained textual and structural cues.
And the result: MoonViT-V2 matches the SigLIP-initialized baseline across vision evaluations. Their conclusion is worth quoting in spirit, because it overturns standard practice: contrastive pre-training is unnecessary as an initialization for multimodal language models at scale.
The encoder is 27 layers, roughly 401M parameters, RMSNorm with all bias terms removed from linear and attention projections. Images and video share parameters completely, with attention factorized into intra-frame spatial and inter-frame temporal passes plus temporal pooling. A 2×2 pixel-shuffle before projection cuts visual tokens 4×, keeping inputs up to 3584×3584 pixels affordable inside a 1M context.
Per-Head Muon
K3 keeps Muon for matrix parameters but refines it for attention projections. Instead of running Newton-Schulz orthogonalization on the full $Q$, $K$, $V$ matrices, it partitions the momentum matrices along the head dimension and orthogonalizes each head’s block separately.
The reasoning: full-matrix orthogonalization treats all heads as one coupled block, so heads with larger gradient or momentum scales dominate the shared update direction while smaller-scale heads get insufficiently normalized updates. Per-head orthogonalization equalizes update scale across heads. It also happens to be cheaper, since Newton-Schulz on tall per-head blocks costs less than on the full projection.
Pre-Training
Data
Four text domains (Web Text, Code, Mathematics, Knowledge) plus a large vision corpus, each filtered by rule-based heuristics, classifier quality scoring, and deduplication, with domain sampling rates set by ablations on smaller models. Knowledge and mathematics corpora are rephrased following K2’s recipe: style and perspective-diverse prompting, chunk-wise autoregressive generation, and fidelity verification against the source.
One vision-data detail stands out. Alongside classical captioned images, K3 substantially scales programmatic multimodal data: code snippets paired with their rendered visuals, across SVG, 3D assets, webpages, games, and CAD schematics. Coordinate supervision is provided in both absolute and normalized $[0,1]$ formats for resolution-robust localization. If you want a model that can write a shader and then look at the shader, this is where that capability is planted.
Scaling Law, and a Note on Cosine vs WSD
The architectural and data changes alter the optimal training regime, so Moonshot re-ran scaling-law studies to retune batch size, learning rate, tokens-per-parameter ratio, and model shape. The fitted curves on held-out OOD validation data are where the 2.5× scaling efficiency gain over K2 comes from.
Buried in this section is a methodological point I think deserves more attention than it will get. The study consistently favored cosine decay over Warmup Stable Decay, contradicting prior reports that WSD matches or beats cosine. Their explanation: the two schedules have substantially different optimal peak learning rates and batch sizes, even at fixed model size and token budget. Comparing them under a shared hyperparameter set unfairly favors whichever schedule those hyperparameters happen to suit. Moonshot ran an independent scaling-law search for each schedule, and under their respective optima, cosine consistently won.
That is a good template for how to compare training recipes, and a reminder of how many published comparisons are quietly measuring hyperparameter alignment instead of the thing they claim to measure.
Training uses Per-Head Muon with K2’s weight clipping, Quantile Balancing for MoE load, cosine schedule with 1% linear warmup, weight decay 0.1 throughout.
Getting to 1M Tokens
Pre-training starts at 8K context and extends to 64K in a later phase. The jump to 1M happens during cooldown, in a four-stage curriculum: 8K to 64K during pre-training, then 256K to 1M during cooldown. Concentrating the expensive long-sequence computation into a small slice of the total budget keeps the curriculum economical.
Because K3 uses NoPE, there is no positional-encoding modification at any stage. The context just extends.
The long-context data pipeline is more involved than the schedule. Naturally occurring long documents and videos are full of near-duplicates, binary blobs, truncated files, and machine-generated logs, so they go through exact and fuzzy dedup, perceptual hashing over video frames, quality filtering, and structural validation. Genuinely long coherent documents are scarce relative to short text, so they get upsampled during cooldown.
And then the part that actually matters: length alone does not confer long-range capability. A 1M-token sequence made of concatenated unrelated documents teaches the model nothing about attending across 1M tokens. So K3 synthesizes long-context data by carefully permuting and concatenating multimodal documents and sub-tasks such that the embedded tasks are only solvable by attending to information scattered across the full context. This trains attention at the intended scale instead of letting it degenerate into local patterns.
Post-Training
The pipeline is three stages: SFT for a cold-start agentic policy, RL to develop specialized domain experts at varying reasoning effort, and Multi-Teacher On-Policy Distillation to consolidate everything back into one model.
SFT and the XTML Chat Template
SFT synthesizes trajectories using domain-specialized models from prior Kimi generations, then applies multi-stage verification and human-in-the-loop annotation. Everything is serialized with a new chat template built on XTML (eXtensible Token Markup Language), and the design goals behind it tell you a lot about what Moonshot expects agentic serving to look like.
XTML is XML-like, but angle brackets are replaced by three reserved special tokens, [open], [sep], and [close], plus [end_of_msg] as the stop marker. Every structural boundary is an explicit token, which removes tokenization ambiguity at element boundaries and simplifies grammar-constrained decoding.
The context layout is organized around KV cache invalidation, which is a very production-minded way to design a chat format:
- Global options (tool declarations, reasoning-effort setting) go before all input messages. They govern the whole session and rarely change, so modifying them invalidates the cache anyway.
- One-shot options (
tool_choice,response_format) go after the input messages, so per-request changes leave the history KV cache intact. - Input option messages interleave mid-session, which is how dynamically loaded tools get announced: the toolset expands without rebuilding the preceding context.
Assistant messages are split into channels (think, response, tools), and the two generation modes are selected purely by generation prefix rather than separate templates. K3 supports only preserved thinking: the think channel is always retained in history, kept even when empty, so the model sees a consistent structure across turns. This is why the README insists you echo reasoning_content back in multi-turn calls. It is not a quirk, it is what the model was trained on.
Tool calls carry tool and index attributes so parallel calls can be matched to their results unambiguously, and arguments are typed: string arguments appear as raw text rather than escaped JSON. Code is a first-class citizen instead of a string with backslashes in it.
RL: Nine Experts
Rather than training task-specific RL models, K3 scales RL across three broad domains:
- General tasks: general experience, vision, reasoning, faithfulness, search, knowledge work
- General agents: long-horizon assistant tasks, deep research, paragraph-level writing
- Coding agents: software engineering, coding experience, kernel tasks, web development
Crossing three domains with three reasoning-effort levels (low, high, max) yields nine expert models. The report shows that scaling RL FLOPs consistently increases both tool-call step counts and capability scores across every evaluated area, which is a nice empirical statement of what “agentic RL scaling” actually buys.
Partial rollouts. Long-horizon tasks have brutal tail latency. K3 extends the partial rollout scheme: sample $K$ completions for each of $N$ prompts, then pause generation as soon as a fraction $\lambda$ of the $N \times K$ trajectories completes, instead of waiting for stragglers. Paused rollouts are queued and prioritized for resumption next iteration.
The consequence is that a single long-horizon trajectory naturally spans multiple training iterations, which introduces serious data staleness. K3’s policy optimization tolerates this extreme off-policy regime through per-token regularization that constrains updates to a localized neighborhood. This is a real dependency worth noting: the infrastructure trick (partial rollouts) only works because the algorithm was designed to survive it.
Reasoning effort RL. Each problem $x$ gets an initial token budget $b_0(x)$ estimated from the cold-start model. Trajectories exceeding a scaled threshold $\tau \cdot b_0(x)$ have their task reward overridden to $-1$. For general tasks the budget counts thinking tokens; for agentic tasks it counts cumulative output tokens including tool-call arguments. Training follows a stage-wise curriculum over $\tau$: train the max-budget variant first with a large $\tau$ (still capped, to suppress overthinking), then anneal $\tau$ down to get the high and low effort experts.
Agentic generative reward model. For non-verifiable general tasks, K3 uses tournament-style group rewards with binary comparisons, where the judge follows a mandatory protocol: read the output, generate a rubric, score each candidate against it, record scores in a scorepad. To stop reward hacking toward verbosity, the same budget mechanism applies: a candidate whose output exceeds $\sigma \cdot \ell_0$ automatically loses its comparison.
Multi-Teacher On-Policy Distillation
Nine experts, one model. For domain $d$ and sampled effort level $e$, the student is guided by the corresponding teacher via a per-token reward:
$$r^d_{\text{opd}}(y_t \mid e, x, y_{Deployment-Aware Post-Training
Two pieces here, both about making the model cheap to serve, and both applied during training rather than after.
MXFP4 quantization-aware training. MoE expert weights, which dominate parameter memory, are quantized to MXFP4 with MXFP8 activations. Attention projections, latent MoE projections, shared experts, and routers stay in higher precision. QAT runs through the entire post-training stage, both SFT and RL. The crucial detail: during RL, rollout and training share the same quantization scheme, which eliminates train-inference mismatch. You are not training an FP8 model and then hoping it survives quantization; you are training the deployed model directly.
Draft model fine-tuning. K3 is pre-trained with an MTP layer mirroring a backbone block. Since EAGLE-3’s draft model is a single decoder layer with matching structure, they fine-tune the pre-trained MTP layer into an EAGLE-3-style draft with the target frozen, unrolled seven steps during training so the draft learns to consume its own outputs.
The draft input fuses low, mid, and high level features from the outputs of the 1st, 4th, and final AttnRes blocks, projected by a bias-free matrix initialized as $[0\ 0\ I]$ so that at initialization the fused representation exactly equals the high-level feature the MTP layer was pre-trained on, then gradually learns to use the others. That is a careful initialization.
And rather than minimizing KL divergence, they directly optimize the acceptance rate, since KL is only a surrogate and does not guarantee maximizing acceptance for a capacity-limited draft:
$$\mathcal{L}_{\text{LK}} = -\log \sum_{x \in V} \min(p(x), q(x))$$The Environments
This is the part of the report I would point a skeptic to. The RL results are only as good as the environments, and Moonshot built a lot of them.
Unified white-box RL environment. Training on one fixed agent harness overfits the model to that harness’s tool schema, system prompt, and context management. So K3 represents a harness as composable, configurable modules: tool interfaces, system prompts, context strategies, skills, memories, subagents. The environment can instantiate Kimi Code, Claude Code, Codex, OpenClaw, and Hermes, plus entirely new ones, and RL training dynamically varies the configuration across task groups. The model learns the general shape of agent harnesses rather than the conventions of one.
Knowledge-graph-guided task synthesis. Task quality is bounded by source material quality. K3 builds a self-evolving hierarchical knowledge graph as a DAG, expanded recursively by agents: each node gets an agent that performs web searches to investigate the concept, checks the existing graph for equivalent nodes before adding new ones, and stops expanding a branch when the concept is judged sufficiently atomic. Sampling nodes at varying granularity (individually or in related combinations) produces keyword sets that drive web retrieval, and the retrieved real-world material becomes training tasks. Fine-grained concept retrieval surfaces specialized and underrepresented knowledge; sampling across diverse concepts controls coverage.
Kernel optimization tasks. Sourced from repos like Flash Linear Attention, spanning CUDA, Triton, CuTe DSL, Gluon, ThunderKittens, and TileLang, across BF16, FP8, and FP4. Rewards cover correctness and performance: solutions past a numerical error threshold get zero, matching an expert implementation yields 0.5, and approaching the hardware roofline pushes toward 1. They also built a hacking-detection system that penalizes CUDA graph replay, input caching, and precision reduction, extended continuously as new hacks appeared during development. That parenthetical is the honest part: the model kept finding new ways to cheat.
Personal assistant tasks. Realistic mock implementations of Gmail, Notion, Slack, and Canvas, preserving core semantics without external APIs or rate limits. Tasks run over multiple simulated days with dozens of interdependent events across applications. A single rollout may involve thousands of tool calls and millions of context tokens. Each event has its own evaluation criterion. The initial workspace is itself constructed by agents that search the web for reference material and transform it into a coherent environment.
Autonomous Execution Tasks. Each task specifies an initial state, a constrained goal, a tool-based action space, execution budgets, and an independent verifier. The agent sees only the objective and the verification interface: no reference trajectory, no predefined procedure. It must decompose the task, select tools, plan, recover from errors, and decide when to stop. Rewards come from the verifier’s assessment of the final environment state, not the agent’s self-reported completion.
Reward hacking is mitigated structurally: agents are isolated from verifiers, public verifiers giving diagnostic feedback are paired with hidden verifiers evaluating held-out scenarios, and submission budgets carry penalties. Verifier types include black-box system replication, quantitative factor discovery, and tax auditing.
Web development tasks. Inputs from one-line scene descriptions to multi-paragraph specs; artifacts spanning websites, games, 3D/WebGL scenes, data visualization, SVG, and full-stack apps. Every task runs containerized and is rolled out under diverse scaffolds. Rewards combine deterministic functional checks (including structural and pixel-level similarity for replication tasks) with model judging that inspects source or interacts with the rendered artifact. The reward is zeroed when a project fails to build, runs with errors, or fakes rather than implements the artifact.
Infrastructure
K3 combines three systems challenges that rarely appear together: hybrid recurrent-plus-global attention, 3T-class sparse multimodal training, and million-token agentic workloads. This section is a third of the report and I suspect it will be the most reused part.
KDA Systems Co-Design
FlashKDA. The chunkwise form is parallel within a chunk and serial across chunks. Run naively, the two phases alternate and the SMs sit idle during serial state propagation. FlashKDA is a CUTLASS-based kernel that overlaps intra-chunk computation with cross-chunk state propagation, decomposing work into token-parallel stages and a head-parallel recurrence, each scheduled and tuned independently. It serves both training and prefill and is auto-dispatched as a flash-linear-attention backend.
Intra-device context parallelism for prefill. Tensor parallelism partitions heads but never shortens the recurrence, so prefilling an ultra-long sequence leaves most SMs idle when each rank holds few heads. Since each segment’s state transition can be evaluated independently of the incoming state and composed exactly afterward, an SM-level planner partitions the sequence across the SMs of a single rank, evaluates transitions in parallel, and merges. No cross-device communication at all.
KDA Context Parallelism. This one is genuinely elegant. For softmax attention, context parallelism means exchanging KV blocks whose size grows with sequence length. For vanilla linear attention, ranks can compute their local state from $S = 0$ and just sum. But KDA’s delta rule applies a token-dependent matrix $M_t = (I - \beta_t k_t k_t^\top)\text{Diag}(\alpha_t)$ to the incoming state before writing, so a segment’s effect depends on the state entering it and cannot be recovered from the zero-initialized state alone.
KCP decomposes each segment’s effect into two locally computable pieces: a cumulative transition $M^{t \leftarrow 1}_{[i+1]} = \prod_r M_r$ acting on the incoming state, and a state $\tilde{S}^t_{[i+1]}$ generated locally from zero:
$$S^t_{[i+1]} = \tilde{S}^t_{[i+1]} + M^{t \leftarrow 1}_{[i+1]} S^{T_i}_{[i]}$$Both quantities depend only on local tokens and can be computed before the incoming state arrives. These rank-level updates compose associatively, so incoming states are recovered by a prefix scan after a single fixed-size all-gather. Communication does not grow with sequence length, and compute scales linearly. This is what makes 1M-token training tractable for the KDA layers.
3T-Class Pre-Training
The parallelism stack is pipeline parallelism with virtual stages, expert parallelism, ZeRO-1 data parallelism, Pipeline ZeRO-2 gradient sharding, and context parallelism. Three problems had to be solved.
MoonEP: perfectly balanced expert parallelism. In conventional EP, token loads are imbalanced across ranks, which costs throughput and, because routed-expert activation shapes vary dynamically, causes memory fragmentation. MoonEP achieves perfect balance using dynamic redundant experts, planned online from the current micro-batch’s router outputs and prefetched before expert computation.
The central question is how many redundant experts are needed to guarantee balance, and Appendix E answers it with a proof: with $E$ experts and EP size $R$, at most $E/R$ redundant experts per rank always suffice, and the bound is essentially tight (there exist router outputs requiring $\lceil E(R-1)/R^2 \rceil \approx E/R$). The proof is a nice constructive argument: repeatedly pick an underloaded and an overloaded rank and migrate tokens to fill the underloaded one exactly to balance; each fill permanently balances one rank so the process terminates in at most $R-1$ steps, and each rank is filled at most once so its remote tokens come from a single source rank, which holds at most $E/R$ local experts.
Why this matters practically: reserving $E/R$ slots per rank guarantees planning always finds a feasible solution, so training is never interrupted. Prior work like ECHO and UltraEP presets a redundant-expert count or a per-rank token cap, which means training halts whenever no feasible plan exists within the cap, and the cap needs manual tuning while still leaving residual imbalance. Exact solutions are computed offline via integer linear programming as references, and a GPU planning kernel provides a near-optimal online solution that always respects the bound.
Perfect balance then cascades into three more wins:
- Zero-copy communication: the planner precomputes every token’s destination, so tokens go directly to their expert-grouped positions on remote ranks and views of the communication buffer feed computation directly. Under worst-case imbalance, DeepEP would need a buffer of size $S \times K \times R$ for the same copy-free path; MoonEP needs only $S \times K$.
- Sync-free execution with static shapes: normally the host must synchronize with the device every layer to learn actual computation shapes before launching expert compute, stalling the pipeline. With every rank receiving exactly $S \times K$ tokens, all shapes are statically known and the per-layer host sync disappears.
- Workload-aware GEMM scheduling: aggregate balance does not fix per-expert skew within a rank, so the routed-expert GEMM uses a scheduler that adapts its parameters to the current token distribution before launch using an analytical cost model with offline-calibrated coefficients.
Memory-efficient training. A unified activation manager treats every tensor saved for backward as having a pluggable storage backend, so recomputation, quantization, and offload/remote-offload become composable storage policies declared by lightweight annotations, fully decoupled from model code. Most activations use block-wise FP8 plus offload; element-wise operators use recomputation. All GPU memory is allocated on the main compute stream in a single pool, avoiding multi-stream fragmentation.
Under interleaved 1F1B pipelining, activations are unevenly distributed across PP ranks because of warmup, with later ranks holding fewer. Rather than sizing for the worst rank, K3 remotely offloads activations to the memory of other PP ranks via the Mooncake Transfer Engine, balancing activation memory across the pipeline. Gradients are sharded with Pipeline ZeRO-2 and stored in CPU memory, with only the double grad buffer on GPU.
Muon’s Newton-Schulz needs full parameter matrices, but the distributed optimizer shards them. The naive fix is an all-gather over the whole parameter buffer on every rank, which is both a memory hog and the primary communication bottleneck at scale. Instead each rank retrieves only the shards of its locally owned parameters via P2P with the owner ranks, eliminating the full-parameter buffer, with communication pipelined against computation at model-chunk granularity.
Hiding the vision encoder. Large images and long videos make the vision encoder’s cost highly variable and expose it on the critical path. K3 extends context parallelism into the encoder (partitioning a single large image along the patch dimension with gather-KV across CP ranks, and splitting CP groups into sub-groups to distribute large images in a load-balanced way). Then, observing that under interleaved 1F1B the text forward passes of the first micro-batches all schedule at the very beginning while the text backwards of the last finish at the very end, they decompose ViT computation so that most of it lands in pipeline bubbles, largely eliminating the encoder’s effective overhead.
Million-Token Agentic RL
Co-located RL training keeps each 1M-context experiment within a few hundred GPUs. But long-context rollouts demand DRAM for KV retention that competes directly with training state.
External KV cache pool. At 1M-context multi-step rollout, a prefix cache miss is extremely expensive. Partial rollout makes it worse: at each iteration’s start, many unfinished long prefill requests from the previous iteration arrive simultaneously. Speculative decoding accelerates request turnover within relatively fixed tool-call intervals, increasing prefix-block churn. All of this triggers preemption and tanks the hit rate.
The fix decouples prefix retention from GPU residency with a write-back design: active decoding blocks stay in GPU KV cache, while reusable idle prefixes are written back to a CPU DRAM pool only when evicted from GPU, then prefetched before reuse. KDA recurrent states are offloaded and prefetched alongside their corresponding MLA KV blocks so lifecycles stay aligned. Compared to write-through, this pays CPU DRAM and bandwidth only for prefixes that actually left the active decode path. To free the DRAM, training states (weights and optimizer state) are offloaded to NVMe once a training iteration finishes.
Auto-throttling. In multi-step rollout, contexts grow as trajectories advance, so fixed concurrency based on full-trajectory average length is both hard to estimate and overly conservative early. Too high, and you get KV pressure and preemption late. So the scheduler dynamically controls how many requests reach the inference engine using runtime signals: active request count, queued count, KV utilization.
Gradient-buffer reuse. RL loss computation needs forward-only non-policy models (reference models) too large to keep resident. K3 keeps them in CPU memory and materializes them on demand, backing their parameter tensors with the policy model’s FP32 gradient-buffer storage. This reuses existing GPU memory with no extra allocation or fragmentation, and it is safe because the buffers get overwritten when real gradients arrive. With ZeRO-2 sharding, each GPU holds gradient buffers for only two VPP chunks, so reference weights stream chunk by chunk with one slot computing while the other prefetches.
AgentENV. The sandbox story deserves its own paragraph. Container-based runtimes were not enough: in early experiments they observed kernel panics and deadlocks caused by unintended agent operations. Meanwhile, they wanted to permit aggressive exploration, and complex tasks need environments where agents can mount disks, run containers, even launch VMs. So AgentENV runs isolated Firecracker microVMs.
The RL-specific features are what make it interesting:
- Incremental checkpointing: only memory pages dirtied since the last checkpoint are saved. Checkpoint latency 133 ms, resume 49 ms.
- Pause and resume: a paused sandbox consumes no memory or CPU. Since a sandbox spends as much as 98% of its lifetime waiting for model inference, this is enormous.
- Fork: creates a new sandbox from the exact state of the original while the original keeps running, useful for reward judging without side effects.
- Snapshot: periodic saves for error recovery.
For density, they use OverlayBD as the image format with a custom ublk driver, storage-layer sharing, and P2P transport, hitting sub-second launch at scale, plus copy-on-write memory and page-cache optimizations for a 6.5× memory overcommit ratio in real workloads.
The number that gives you a sense of scale: across K3’s training and evaluation, 51,219,741 sandboxes were created across 1,505,678 distinct images.
Serving
KDA-aware prefix caching. The hybrid architecture makes prefix caching genuinely hard. MLA’s KV cache grows with sequence length and pages per token; KDA’s recurrent state is fixed-size with one copy per request. A cached prefix is reusable only if both can be restored at the same boundary.
First, they pack KDA states into the same paged block pool as MLA KV, unified to the same byte size so allocation, reference counting, and eviction share one implementation. Within a page, all heads’ states are stored contiguously head by head so each head’s byte stream is self-contained and serves as the minimal unit of cross-node transfer, which lets prefill/decode disaggregation with different TP degrees re-layout on the transfer path with zero GPU-side reshuffling. There is a wry aside here: the type asymmetry means any type-confused access yields garbage rather than plausible data, “a zero-overhead sanity check on the pooled layout.”
The harder problem: block-hash prefix caching requires one block size shared by all layers, and a hit is only usable if the KDA state at that boundary was persisted. KDA keeps one large state per sequence, so snapshots are affordable only at sparse boundaries, forcing the shared block size to 1024 to 6144 tokens. At that granularity caching is nearly useless: requests shorter than one block can never be reused, and chunked prefill exports nothing cacheable until it crosses a full block boundary.
So K3 decouples the granularities. Prefix hashing runs on fine 512-token hash blocks inside MLA pages while the physical block stays the coarse allocation unit; KDA checkpoints are saved only at a sparse subset of MLA’s hash endpoints, the only positions a lookup can ever reference. Checkpoints superseded as a request advances are recycled, while those at conversation-turn boundaries are retained for cross-request reuse. Lookup then runs in two stages: MLA matches whole physical blocks by chained hash and falls back to hash endpoints inside the first missing block, then KDA requires a checkpoint at the candidate boundary in every cache group. The hit is the longest boundary satisfying both.
The result: any shared prefix is reusable at any 512-token boundary, independent of request length, chunking, or scheduling interleaving. Hybrid models reach the same prefix-caching generality as full-attention models.
The consistency mechanisms are each a fix for a specific failure of sharing partially filled blocks: every hit block is pinned across all cache groups before anything is allocated (otherwise allocating a private copy for one group could evict a block another group just hit); blocks allocated or registered within the current scheduling step are excluded from matching until their copies land (otherwise a reader gets the previous owner’s bytes); and evicting one group’s checkpoint atomically invalidates its siblings, so a checkpoint is hittable in every group or none.
Speculative decoding with a recurrent state. KDA decoding has a rollback problem: the state updates in place every step, so if verification rejects some drafted tokens, the state has already advanced past the last accepted token. Snapshotting per draft position would enable rollback but multiply state traffic, which dominates at serving batch sizes.
The observation: the state after any accepted prefix is fully determined by the projected inputs of the draft tokens, which are far smaller than the state. So K3 caches only those inputs, rebuilds accepted-token states on-chip, and writes back only the verified and bonus token states. Replayed tokens, the bonus token, and the next draft window share one recurrent loop inside a single fused kernel covering short convolution, input normalization, gating, the recurrence, and output normalization. Verification latency grows sub-linearly in tokens verified. (The report notes this was independently proposed as ReplaySSM by Dao AI Lab, which is a good sign for the idea.)
Fleet-level scheduling. Two policies. Cache-aware affinity: at 1M context, a typical coding request carries a 400K-token prefix but only a 4K prefill increment, so a hit is orders of magnitude cheaper than a miss, and moving a cache between clusters means crossing links far slower than the intra-cluster fabric. Requests route to the cluster holding their cache. But that binds sessions to single clusters, so consistent hashing pins each session to a primary and a pre-assigned secondary; the secondary must re-prefill on failover, but since consistent hashing spreads secondary assignments uniformly, that re-prefill work distributes across many clusters instead of concentrating on one.
Budget-based admission control: production traffic mixes sub-2K requests with 1M-token requests, so per-request cost spans three orders of magnitude and capacity planning based on an “average request” is meaningless. In the typical failure mode, a burst of long-context requests saturates compute and short requests behind them see degraded time-to-first-token. Separate resource budgets per request class bound the blast radius.
Results
K3 is evaluated at max reasoning effort, temperature 1.0, top-p 0.95 for single-step tasks and 1.0 for agentic ones, against Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 (xhigh), and GLM-5.2.
The report’s own summary is refreshingly unspun: K3 trails Claude Fable 5 and GPT-5.6 Sol overall, and consistently beats everything else evaluated.
Reasoning and Knowledge
| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 | GPT-5.5 |
|---|---|---|---|---|---|
| GPQA Diamond | 93.5 | 92.6 | 94.1 | 91.0 | 93.5 |
| CritPt | 23.4 | 28.6 | 32.3 | 20.9 | 27.1 |
| AA-LCR | 74.7 | 70.0 | 73.7 | 67.7 | 74.3 |
| HLE-Full (no tools / tools) | 43.5 / 56.0 | 53.3 / 63.0 | 44.5 / 58.0 | 49.8 / 57.9 | 41.4 / 52.2 |
Competitive at graduate level, clearly behind at research level. The report names this directly: CritPt and HLE-Full are where the gap lives, and research-level reasoning remains a key direction for improvement.
Coding
| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 | GPT-5.5 | GLM-5.2 |
|---|---|---|---|---|---|---|
| ProgramBench | 77.8 | 76.8 | 77.6 | 71.9 | 70.8 | 63.7 |
| SWE-Marathon | 42.0 | 35.0 | 39.0 | 40.0 | 14.0 | 13.0 |
| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 | 84.6 | 83.4 | 82.7 |
| FrontierSWE | 81.2 | 86.6 | 71.3 | 66.7 | 64.9 | 67.3 |
| DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 | 67.0 | 46.2 |
| PostTrainBench | 36.6 | 41.4 | 34.6 | 34.1 | 28.4 | 34.3 |
| MLS-Bench-Lite | 48.3 | 49.9 | 46.2 | 42.8 | 35.5 | 40.4 |
| SciCode | 58.7 | 60.2 | 56.1 | 53.5 | 56.1 | 50.5 |
SWE-Marathon is the standout: a GPU-kernel-oriented suite where K3 leads by 7 points over Fable 5, which is consistent with all that kernel-optimization RL. FrontierSWE at 81.2 is second, but ahead of the third-place model by ten points.
Read the caveats, though. The report is unusually forthcoming about them: Fable 5 hits fallbacks on 35% of SWE-Marathon tasks; the SWE-Marathon evaluation runs an H20-calibrated branch of the official tasks; PostTrainBench runs on H20 instead of the official H100. These are stated plainly, but they mean the comparisons are not perfectly apples-to-apples.
Agentic
This is K3’s strongest axis, with state of the art on a broad set:
| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 |
|---|---|---|---|---|
| BrowseComp | 91.2 | 88.0 | 90.4 | 84.3 |
| DeepSearchQA (F1) | 95.0 | 94.2 | n/a | 93.1 |
| MCPMark-Verified | 94.5 | 87.4 | 92.9 | 76.4 |
| ResearchRubrics | 76.2 | n/a | 73.8 | 73.5 |
| Harvey Lab-AA | 94.6 | 93.6 | 87.2 | 91.1 |
| $\tau^3$-Banking | 33.4 | 26.8 | 33.0 | 27.6 |
| AutomationBench | 30.8 | 29.1 | 29.7 | 27.2 |
| SpreadsheetBench 2 | 34.8 | 34.7 | 32.4 | 31.6 |
| GDPval-AA v2 (Elo) | 1686 | 1747 | 1736 | 1593 |
| AA-Briefcase (Elo) | 1548 | 1583 | 1495 | 1354 |
| OSWorld 2.0 | 58.3 | 66.1 | 62.6 | 55.7 |
The pattern: K3 wins on search, tool use, and structured professional tasks, and loses on the Elo-rated open-ended knowledge-work suites and the harder computer-use benchmarks. One config note worth knowing: BrowseComp’s 91.2 uses context compaction triggered at 300K tokens. With the full 1M window and no context management, K3 gets 90.4. Only 0.8 points, but it tells you compaction is doing real work even at 1M.
Vision
Best-in-suite on OmniDocBench (91.1), Video-MME with subtitles (90.0), and MMVU (82.1). On Math-Vision, 94.3 rising to 97.8 with Python tools. On ZeroBench-main, tied with Fable 5 at 23.0 pass@5, jumping to 41.0 with Python. The tool-augmented deltas are the interesting part: they are large, and they are exactly what the vision-in-the-loop agentic RL was designed to produce.
Third-Party Evaluations
| Source | Kimi K3 | Rank |
|---|---|---|
| Artificial Analysis Intelligence Index v4.1 | 57.1 | #4 of 580 |
| Vals Index | 74.7 | #2 of 39 |
| WebDev Arena (Elo) | 1,678 | #1 of 99 |
| Text Arena (Elo) | 1,486 | #8 of 200 |
| Agent Arena | 9.1 | #4 of 37 |
The WebDev Arena result is a first: the first open model to top that leaderboard, ahead of Claude Fable 5 at 1,634. On their in-house Kimi Webdev Bench, blind expert judges preferred K3 over Claude Opus 4.8 by +31.0 points overall, with the largest margin on 3D/WebGL/Shader tasks at +59.1.
Cost Efficiency
This is where K3 makes its actual argument, and it is the section most likely to change someone’s deployment decision.
- Kimi Code Bench 2.0: 4.0 points behind Fable 5 at 38% of the cost. At
higheffort it already matches Opus 4.8’smaxeffort score at roughly a third of the cost. - BrowseComp: best score (91.2) at 2.03 USD per task, half the cost of GPT-5.6 Sol and an order of magnitude cheaper than the Claude models at max effort.
- GDPval-AA v2: within 50 Elo of GPT-5.6 Sol at 13% lower cost, and 2.6× cheaper than Fable 5.
- AA-Briefcase: second-best score behind Fable 5, at roughly half the cost.
K3 sits on or near the cost-efficiency frontier across all four. If your workload is agentic and you are paying per token, that combination is the whole pitch.
Internal Benchmarks, Including the Losses
Moonshot maintains an in-house suite and, notably, publishes where it loses. Strengths: Swarm Bench (76.3, clear lead), Deep Research Bench (90.0, clear lead), Coding Experience (best), Kimi Code Bench 2.0 (second only to Fable 5), Finance Bench (essentially tied with GPT-5.6 Sol).
Weaknesses, stated outright: Agent Behavior Bench, MIRA Bench, 24/7 ClawBench 2.0, Agentic Vision Bench, and KWV Bench. Agent Behavior Bench is the one I would watch, since it scores process quality (tool-use behavior, efficiency, discipline) rather than outcome correctness, and K3 sits at 65.0 against 75.5 and 76.4 for Fable 5 and GPT-5.6 Sol. Getting the right answer via a messier path is a real and measurable difference.
Cyber Security Evaluation
This section is unusual and I want to represent it carefully, because it is one of the more candid capability disclosures in an open model release.
Moonshot evaluated K3 along two tiers. Tier 1 is vulnerability discovery with proof-of-concept development, which they characterize as primarily defensive research. Across dozens of widely deployed systems (OS kernels, databases, AI services, web frameworks, blockchain, VPN software), the model surfaced hundreds of candidate vulnerabilities. Of findings that underwent human review, roughly 70% were confirmed genuine, including 16 previously unknown vulnerabilities across six projects.
Two Linux kernel findings illustrate the depth: a remotely triggerable heap out-of-bounds write introduced by an incomplete upstream fix and affecting all subsequent releases up to current upstream, confirmed by security experts as a remote denial-of-service primitive; and a Dirty-COW-class vulnerability in the RDMA subsystem where an earlier upstream fix had dropped a permission check, confirmed as a deterministic local privilege-escalation primitive.
Tier 2 is end-to-end exploit development, the tier most relevant to misuse risk. They could not compare against frontier proprietary models here because those models refuse cyber tasks, so GLM-5.2 is the baseline. On an in-house 36-task suite (16 user-space CVEs against PostgreSQL, XWiki, Apache HTTP Server and others; 20 Linux kernel privilege escalations in reproducible QEMU environments with progressively enabled mitigations), K3 solved 14 of 36 (38.9%) versus GLM-5.2’s 8 of 36 (22.2%).
Every task is verified solvable by human experts, at an estimated 540 expert-hours total, roughly 15 hours per task. So the 22 unsolved tasks are a direct measurement of remaining gap to human capability. And the successes are lopsided: 10 of the 14 come from the user-space track, and on the kernel track neither model solves three-quarters of the tasks.
Trajectory analysis attributes the gap to four recurring failure modes: difficulty completing the final stage of an exploit chain from primitives already obtained, poor strategy selection under mitigations (persisting with control-flow hijacking when a data-only attack would be simpler), getting stuck in prolonged unproductive debugging loops, and insufficient verification of the final deliverable before submission.
An independent joint assessment by the UK AI Security Institute and NIST’s CAISI reached consistent conclusions: K3 beats GLM-5.2 on exploit development (32% vs 24% on ExploitBench; 17 vs 11 steps on a 32-step simulated enterprise network that takes a human expert about 20 hours), but trails frontier cyber-capable models on end-to-end completion, achieving arbitrary code execution on 0 of 41 tasks.
Moonshot explicitly frames their own evaluation as a lower bound on capability, conditioned on the current version and evaluation coverage, and commits to revisiting at each major update. Publishing “here is what our open-weight model can do offensively, here is the independent assessment, here is where it falls short” is the right norm, and I hope it becomes standard.
Case Studies
The case studies are where the abstract capability claims become concrete, and several of them are striking.
GPU kernel optimization. Each model got an identically configured sandbox and up to 24 hours per task to profile, rewrite, and benchmark four kernels: AttnRes, DeepSeek Sparse Attention, KDA, and MLA with head dimension 512, on an NVIDIA Hopper GPU and an alternative-vendor GPGPU. K3 cut AttnRes latency from 283.6 ms to 114.4 ms, reduced DSA runtime by 55.1% and KDA by 73.6%, and reached over half of peak TFLOPS on MLA. It matched Claude Fable 5 and substantially beat Opus 4.8, GPT-5.6 Sol, and GPT-5.5.
Then this line, almost in passing: an early K3 checkpoint was already handling most of Moonshot’s kernel optimization work during late-stage development. The model helped build itself.
GPU compiler development. K3 built MiniTriton, a compact Triton-like compiler with a custom tile-level Python frontend and layout system, a warp-level MLIR annotation and optimization layer, and a PTX codegen pipeline, plus a dual-mode tensor library with reverse-mode autograd, NN modules, distributed primitives over NCCL, and sparse/visualization primitives.
On an NVIDIA L20 it beats PyTorch eager and torch.compile in geometric mean over its core benchmark suite. Its from-scratch tensor-core matmul approaches cuBLAS at the largest shapes, hitting about 90% of the measured machine roof. Its DSL-level KDA prefill kernel beats a matched Triton reference. And it trains a GPT end to end with a loss curve tracking the PyTorch reference, with full-model gradients differing from torch autograd by no more than torch’s own fp32 rounding error ($10^{-4}$), measured against an fp64 reference.
That last detail is what convinces me. A model can produce a plausible-looking compiler. Producing one whose gradients match torch to within torch’s own numerical noise means the whole stack, frontend through IR through codegen through runtime, is actually correct.
Chip design. As a proof of concept, K3 designed an inference-chip prototype for a nano model following K3’s own architecture (hybrid KDA and NoPE-MLA, Block AttnRes with block size two, sigmoid MoE routing with one shared expert) under group-wise INT4 weight quantization. In a single 48-hour autonomous run with Kimi Code, using open-source EDA tools and the Nangate45 standard-cell library, it built, optimized, and verified a design that closes timing at 100 MHz within a 4 mm² area budget, achieving RTL-simulated decode throughput above 8,700 tokens/s, integrating 1.46M standard cells, 0.277 MiB of SRAM, and an INT4 MAC array with fused dequantization. The RTL is on GitHub.
Research coding. Reproducing the I-Love-Q universal relations in computational astrophysics: reviewed 20+ papers and cross-validated their results, implemented the full numerical pipeline, evaluated over 300 equations of state, identified inconsistencies in published formulas, wrote 3,000+ lines of Python, and produced an interactive HTML dashboard. In about two hours, against a typical one to two weeks for an experienced researcher.
Knowledge work. An interactive research site covering 42 years of the AI ASIC industry: 120+ rounds of iterative refinement over 87 quarterly reports and 99 original PDFs (11,000+ pages), driven by 2,800+ web searches and 1,100+ terminal queries. Separately, analyzing 391 gravitational-wave events in GWTC-5 using 20+ concurrent subagents, producing seven visualizations, two summary tables, and a literature synthesis over ten papers.
Video. K3 created a 3Blue1Brown-style motion-graphics explainer of its own architecture and edited its teaser video from 56 source clips, including clip selection, motion-matched cuts, frame-accurate beat synchronization, and audio processing.
What I Take Away
Full-stack co-design is the actual result. Nearly every architectural decision in K3 has a systems consequence, and several were clearly made for the systems consequence. Bounded decay exists so diagonal tiles can use Tensor Cores. NoPE exists partly so context extension does not require positional surgery. Block AttnRes exists so pipeline stages do not have to ship every layer’s output. LatentMoE exists so 16 active experts do not double communication. Perfect EP balance exists so computation shapes are static and the host sync disappears. You could not have designed this model without the infra team in the room, and the report reads like they were.
The bottleneck has moved to environments. The architecture section is 8 pages. The environments and infrastructure sections together are closer to 20. Moonshot built mock Gmail and Slack, a self-expanding knowledge graph, a kernel suite with an adversarial anti-hacking system, verifier-isolated autonomous execution tasks, and 51 million Firecracker microVMs. That is where the differentiated capability came from, and it is much harder to replicate than an architecture diagram.
Reward hacking is a first-class engineering problem now. Look at how many mechanisms exist purely to stop the model from cheating: hacking detection for CUDA graph replay and precision reduction, verifier isolation with paired public and hidden verifiers, submission budgets with penalties, verbosity caps in the generative reward model, zeroed rewards for faked artifacts. None of this is theoretical. It is all scar tissue.
Honest reporting is a feature. The report states that K3 trails Fable 5 and GPT-5.6 Sol. It names its weak internal benchmarks. It flags that Fable 5 hit fallbacks on 35% of SWE-Marathon and that PostTrainBench ran on H20 instead of H100. It publishes offensive cyber capability alongside an independent government assessment and calls its own numbers a lower bound. Set against the usual release-day benchmark theater, this is a genuinely different posture, and it makes the parts I cannot verify more credible.
The gap is real but specifically shaped. K3 is at or near the frontier on agentic search, tool use, structured professional workflows, and web development, and it is meaningfully behind on research-level reasoning (CritPt, HLE), open-ended knowledge work (GDPval, AA-Briefcase), and process discipline (Agent Behavior Bench). That is a coherent profile: it says the RL environments were superb at verifiable long-horizon execution and thinner on open-ended judgment. Which is exactly what you would predict from reading the environments section.
And the cost curve is the story for practitioners. Near-frontier scores at 38% of Fable 5’s cost on coding, half on BrowseComp, 2.6× cheaper on GDPval, with open weights and a 1M context. Whether K3 is the best model in the world matters less than whether it is the best model per dollar for agentic work at scale. On the evidence here, for a lot of workloads, it is.
The report closes by calling K3 “a new open frontier within everyone’s reach.” Given that they shipped the weights, the kernels (FlashKDA, MoonEP), the sandbox (AgentENV), and 47 pages explaining how all of it works, that is not just a slogan.
References
- Kimi K3: Open Frontier Intelligence, Technical Report
- Kimi K3 on Hugging Face
- Kimi K3 tech blog
- Attention Residuals, Kimi Team and my write-up
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Kimi K2: Open Agentic Intelligence
- LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
- MoonEP, FlashKDA, AgentENV
- MiniTriton and nano-kpu, both built by K3
Share with friends