Where 2.5x comes from: key updates across three information flow axes

Information inside a model flows along three distinct axes: horizontally across sequences (passing context from earlier tokens to later ones), vertically across depth (propagating features from lower to higher layers), and across layer width (routing inputs among functional modules).

Each axis presents its own computational bottlenecks. K3 addresses each with a targeted architectural change.

Combined with refined training and data recipes, these modifications yield the 2.5x efficiency gain. The architecture overview below illustrates how these layers are stacked together throughout the model.

The backbone stack (right) repeats blocks of 3 KDA layers plus 1 Gated MLA layer, each followed by a sparse MoE layer. Thin lines on the right represent Attention Residual skip connections. Top-left shows MoE internals (green: shared experts; purple: selected routed experts); bottom-left shows KDA internals; bottom-right outlines the visual stream. Source: Technical Report Figure 2.

Sequence length: replacing most attention layers with fixed-size memory

Standard attention mechanisms store preceding context in a KV cache, which expands with every new token. As context lengths stretch toward 1 million tokens, VRAM consumption quickly becomes unsustainable.

K3 addresses this by replacing standard KV caching across most layers with a fixed-size memory state. As new input arrives, the memory state updates, allowing older context to decay gradually across channels. Memory footprint remains constant whether processing 10,000 tokens or 1,000,000 tokens. This variant of linear attention is termed Kimi Delta Attention (KDA).

To mitigate memory decay over long sequences, K3 preserves a full-context Gated MLA (Multi-Head Latent Attention) layer every four layers to re-scan the entire sequence, with a final global attention layer placed at the end of the backbone.

This 4-layer block repeats across the 93-layer backbone, totaling 69 KDA layers and 24 Gated MLA layers.

Key advantage: extending to 1M tokens without modifying positional embeddings

Rather than relying on explicit positional encodings, K3 completely removes positional embeddings from its global attention layers. Sequence ordering is encoded implicitly through KDA's channel decay, where distance naturally correlates with decay intensity.

As a result, expanding the context window from 64k to 1M tokens required zero modifications to positional parameters. Standard context-extension techniques like RoPE frequency retuning or YaRN interpolation were completely unnecessary, allowing direct zero-shot sequence extrapolation.

Context length was expanded progressively across four stages: scaling from 8k to 64k during pre-training, and from 256k to 1M during the annealing phase. Concentrating long-context compute within a small fraction of the training budget kept total training costs manageable.

Depth expansion: enabling Layer 93 to pull directly from Layer 1

Standard residual connections accumulate output representations additively layer by layer. As networks deepen, individual layer outputs get blended into a running sum, making it difficult for deep layers to isolate specific representations generated by much earlier layers.

Attention Residuals solve this by equipping each layer with a learned query vector that attends over the uncombined outputs of all preceding layers. Instead of taking just the cumulative sum, Layer 93 can directly query Layer 7's output with dynamically calculated attention weights.

Retaining uncompressed representations across all 93 layers in VRAM would introduce prohibitive memory overhead. To keep computation manageable, K3 uses a block-wise implementation: the 93 layers are grouped into 8 blocks (12 layers per block). Representations are summed locally within each block before cross-block attention is applied, reducing memory scaling from $O(L)$ to $O(B)$, where $B$ is the number of blocks.

The report notes that the final block is partial: 8 full 12-layer blocks would total 96 layers, so the final block contains the remaining 9 layers.

Width scaling: activating 16 out of 896 experts per token

The third axis scales layer width using Mixture-of-Experts (MoE). While K2 relied on MoE layers, K3 scales expert sparsity significantly: expanding routed experts from 384 to 896 while doubling per-token activations from 8 to 16.

Increasing expert counts typically drives up inter-node communication overhead. K3 avoids this bottleneck by operating routed experts in a half-dimension latent space (3,584 dimensions versus the model's 7,168 hidden dimensions). Two persistent shared experts retain full hidden width for general transformations. This architecture doubles total expert capacity without inflating communication payload.

Maintaining stability under extreme sparsity

Operating at extreme sparsity introduces severe training instability, which Moonshot AI addressed through three architectural safeguards:

• Post-Expert Normalization: Output activations from selected experts undergo layer normalization prior to upward projection. Because expert routing and weighting vary dynamically per token, normalization prevents variance spikes across layer outputs.
• SiTU-GLU Activation Function: Standard SwiGLU activations use unbounded multiplicative branches that can trigger gradient explosions when large values coincide. SiTU-GLU introduces a smooth saturation cap (via a tanh function), matching SwiGLU behavior at standard ranges while clamping extreme activation spikes.
• Quantile Balancing: To balance load across 896 experts, traditional methods iteratively adjust routing bias terms, which risks either slow convergence or routing oscillations. K3 replaces this with Quantile Balancing, deriving bias offsets directly from global score quantiles. To compute quantiles efficiently across thousands of GPUs, worker nodes generate local histograms that are aggregated into an empirical global distribution.

The team also updated their optimizer framework: the Muon optimizer used in K2 was adapted to operate head-wise. Previously, orthogonalizing full projection matrices allowed heads with larger gradients to dominate updates; partitioning matrix orthogonalization per attention head ensures balanced update magnitudes across all heads.

K2 vs. K3 specification breakdown: doubling depth and expert counts at constant hidden width

Comparing K2 and K3 specifications side by side clarifies how these individual updates fit together. The data below is extracted from the technical report.

Notice the highlighted row: Hidden dimension width remains completely unchanged at 7,168. K3 expands along depth (more layers) and routing capacity (more experts) rather than inflating hidden dimensions, optimizing information routing instead of widening individual channels.

Training vision encoders from scratch: departing from industry standards

K3 was developed with native multimodality: text, images, and video frames were trained jointly from step one within the primary backbone under a unified objective, skipping the common multi-stage alignment phase.

The vision module itself breaks with conventional design. MoonViT-V2 (401M parameters, 27 layers) was trained entirely from scratch. Common practice—including Moonshot AI's own prior K2.5 release—initializes vision modules using contrastively pre-trained weights (such as SigLIP) to jumpstart visual representation learning.

The team opted against pre-trained initialization to prioritize training stability.

Comparison of vision encoder gradient norms: the SigLIP-initialized baseline (blue) exhibits elevated gradient norms with frequent spikes (signaling training instability), whereas MoonViT-V2 trained from scratch (red) maintains steady trajectories. Right inset zooms in on steps 14,000 to 16,000. Source: Technical Report Figure 6.

Critically, the version trained from scratch matched performance across vision benchmarks compared to the SigLIP baseline. The report concludes that at this compute scale, contrastive vision pre-training is no longer necessary as an initialization step.

If replicated by other labs, this finding could reshape standard multimodal pre-training workflows.

Post-training pipeline: distilling 9 specialized models into one

Following base pre-training, K3 underwent a two-phase post-training process: specialized domain training followed by model consolidation.

The 3×3 matrix spans three domains (general tasks, general agent workflows, and coding agents) across three reasoning effort levels (low, high, and max). RL training produced nine distinct expert checkpoints. Through Multi-Teacher Online Distillation, a unified student model learned dynamically from all nine teacher models simultaneously during generation.

Enforcing reasoning effort via token budgets

Reasoning length during RL training was bounded strictly using token budgets. If reasoning trajectories exceeded assigned budget multipliers, rewards defaulted to -1 regardless of answer correctness. Training began with generous budgets for max reasoning, followed by progressive budget tightening to construct high and low reasoning tiers under human supervision.

Crucially, quantization was integrated directly into training rather than applied post-hoc. Expert weights were trained natively in MXFP4 format with activations computed in MXFP8 across both SFT and RL phases. Training under low-precision constraints prevented accuracy degradation when deploying quantized production models.

Multi-harness RL: Preventing agent overfitting to proprietary interfaces

Training functional agents requires robust, diverse execution environments.

If trained strictly on a single interface, agents risk overfitting to specific prompt structures and tool call formats. Moonshot AI addressed this by modularizing the harness architecture—decoupling tool schemas, system prompts, context management policies, skills, memory, and sub-agents into configurable components. This modular environment simulated interfaces including Kimi Code, Claude Code, OpenAI Codex, OpenClaw, and Hermes, randomly rotating harnesses during training runs.

Key environment features built into the training framework include:

• Simulated Application Suites: Mock implementations of Gmail, Notion, Slack, and Canvas were built to mirror production API logic offline. Tasks spanned multi-day simulated workflows containing dozens of interdependent events, requiring thousands of tool calls across millions of tokens.
• Anti-Cheating Verification for Kernel Optimization: Training tasks sourced from open-source GitHub repositories evaluated code output on both correctness and execution speed. To prevent shortcut behaviors like graph replay, input caching, or silent precision demotion, automated anti-cheat validators inspected generated kernels during evaluation.
• Goal-Oriented Autonomous Tasks: Prompts provided high-level objectives, constraints, and validation APIs without step-by-step instructions. Rewards were awarded strictly based on environmental state verification, ignoring self-reported task completion. Hidden evaluation suites were held out to prevent models from gaming public validation feedback.

Infrastructure efficiency: Spin-up metrics across 51.22 million agent sandboxes

Executing continuous RL training runs required high-density, low-latency agent sandbox orchestration.

Sandboxes provide secure isolation for agent code execution. Moonshot AI deployed AgentENV—an open-source system built on Firecracker microVMs—replacing standard container runtimes after unhandled model execution crashed host kernels during early training runs.

The primary architectural optimization targeted idle execution windows:

A paused sandbox consumes zero memory and CPU resources. Consequently, sandboxes can be paused while agents await model inference outputs—a waiting phase that accounts for up to 98% of total sandbox lifecycle duration.

Kimi K3 Technical Report §5.3.2

This shift prioritized eliminating idle compute costs. Fast pause (133 ms) and resume (49 ms) latencies enabled instant state freezing. Sandboxes could also be cloned on-the-fly for evaluation branching or snapshot-restored following execution errors.

Serving long contexts introduced KV cache prefix matching challenges due to architectural differences between Gated MLA (per-token scaling) and KDA (fixed memory states). Because snapshotting large KDA states at every token is impractical, prefix caching risks coarse alignment boundaries. The engineering team decoupled lookup and storage granularities, matching shared prefixes at fine-grained 512-token boundaries while pinning memory snapshots to segment endpoints.

Public benchmarks: Ranking behind Fable 5 and GPT-5.6 Sol

How does K3 perform empirically? The technical report presents clear findings: K3 trails Claude Fable 5 and GPT-5.6 Sol overall, but consistently outperforms other evaluated models, including Claude Opus 4.8, GPT-5.5, and open-weights baseline GLM-5.2.

Blue bars represent Kimi K3 across max reasoning effort settings (xhigh for GPT-5.5). Report notes: Fable 5 evaluations included fallback behaviors, and GPT-5.6 Sol scores reflected active safety guardrail intervention. Source: Technical Report Figure 1.

K3 leads top scores across several domains: ProgramBench (77.8), SWE-Marathon (42.0—7 points ahead of Fable 5), BrowseComp (91.2), MCPMark (94.5), legal research benchmark Harvey Lab-AA (94.6), and document parsing suite OmniDocBench (91.1). On command-line agent test Terminal-Bench 2.1, K3 scored 88.3, trailing GPT-5.6 Sol (88.8) by half a point.

The report explicitly details K3's performance limitations:

On the complex benchmark HLE-Full, K3 achieved 43.5 without external tools and 56.0 with tool assistance, trailing both Fable 5 and GPT-5.6 Sol. The report acknowledges that "research-grade reasoning remains a primary focus for future improvement."

Third-party evaluations

Independent benchmarks as of July 23 show strong standing: Artificial Analysis Intelligence Index scored 57.1 (4th of 580 models); Vals AI Industry Index scored 74.7% (2nd of 39 models); and WebDev Arena blind user preference scored 1,678 ELO—ranking 1st out of 99 models, making K3 the first open-weights model to top the leaderboard. It ranked 8th of 200 models on Text Arena and 4th of 37 on Agent Arena.

Internal evaluations: Strong output accuracy vs. lower process discipline

Beyond public leaderboards, Moonshot AI reported internal benchmark results designed to test unaligned edge cases, revealing clear operational trade-offs.

Strengths: K3 led multi-agent orchestration on Swarm Bench (76.3) and deep research workflows on Deep Research Bench (90.0). In expert blind side-by-side web development evaluations, K3 outperformed Claude Opus 4.8 by 31 percentage points overall, and by 59.1 percentage points on 3D graphics and shader generation tasks.

Weaknesses highlighted in the report:

Agent Behavior Bench measures process execution quality: tool call discipline, efficiency, and adherence to operational protocol, independent of final output correctness. K3 trails top models by over 10 points on this metric (scoring 65.0 vs. Fable 5's 75.5). K3 also lagged on multi-role enterprise collaboration (MIRA Bench: 64.1 vs. Fable 5: 72.9), virtual assistant tasks, and two vision-centric benchmarks.

This performance gap highlights a practical reality: while K3 delivers correct final outputs, its step-by-step agent execution discipline requires further refinement.

Cost efficiency: Delivering tasks at a fraction of competitor pricing

Beyond raw capability scores, task completion cost determines practical deployment feasibility.

Score vs. per-task cost across four benchmarks (top-left: Kimi Code Bench; top-right: BrowseComp; bottom-left: GDPval-AA v2; bottom-right: AA-Briefcase). Red star indicates Kimi K3; top-left placement indicates optimal cost-efficiency. Source: Technical Report Figure 13.

• BrowseComp (Web retrieval): K3 achieved the top score of 91.2 at $2.03 per task—half the cost of GPT-5.6 Sol and an order of magnitude cheaper than Claude at max reasoning effort.
• Kimi Code Bench 2.0 (Coding suite): K3 scored 4 points below Claude Fable 5 while consuming only 38% of the cost. Configured to 'high' effort, K3 matched Claude Opus 4.8 max effort performance at one-third the execution cost.
• GDPval-AA v2 (Knowledge work): Scored within 5% of GPT-5.6 Sol at 13% lower cost, and delivered 2.6x cost savings over Claude Fable 5.
• AA-Briefcase: Ranked 2nd overall behind Fable 5 at approximately half the per-task expenditure.

Cost methodology notes: K3 execution costs on Kimi Code Bench and BrowseComp reflect internal measurements, whereas Claude and GPT baseline costs are cited from published benchmark figures. GDPval-AA v2 and AA-Briefcase pricing metrics are sourced from Artificial Analysis token rate tables as of July 23.

Cybersecurity evaluation: Discovering 16 zero-days while struggling with exploit generation

Omitted from the initial launch blog, the technical report devotes a complete section to empirical cybersecurity testing divided into two risk tiers.

Tier 1: Vulnerability discovery

Models were assigned to audit production codebases across OS kernels, databases, AI frameworks, web stacks, blockchain projects, and VPN services to discover unpatched zero-day vulnerabilities and construct proof-of-concept triggers.

Results: K3 generated several hundred candidate vulnerability reports. Upon manual audit, approximately 70% were verified as valid bugs, uncovering 16 novel zero-day vulnerabilities across 6 projects. Two Linux kernel examples are highlighted in detail:

1. A remotely triggerable heap out-of-bounds write resulting from an incomplete upstream security patch, affecting all subsequent kernel builds up to the latest release. Security researchers confirmed its viability for remote denial-of-service attacks.
2. A permission check flaw in the RDMA subsystem reminiscent of Dirty COW, caused when a prior patch inadvertently omitted a check, allowing arbitrary writes to read-only memory pages. Researchers confirmed this as a reliable local privilege escalation vector.

Tier 2: End-to-end exploit development

Tier 2 evaluates offensive capability across 36 challenge tasks: 16 user-space targets (e.g., PostgreSQL, Apache HTTP Server, XWiki) provided with source code and live containers, and 20 Linux kernel privilege escalation tasks requiring functional C exploit payloads against hardened VM environments.

K3 solved 14 of 36 tasks (38.9%), outperforming GLM-5.2 (8 of 36). However, performance was uneven: 10 of K3's 14 solves occurred in user-space targets, while both models failed over 75% of kernel-level tasks, illustrating a significant gap between model capabilities and human security experts.

The report summarizes four recurring failure modes: failing to complete multi-step exploit chains after identifying primitives; selecting sub-optimal exploitation paths under modern mitigations (e.g., attempting control-flow hijacking instead of data-only attacks); getting stuck in repetitive debugging loops; and submitting unverified payload drafts.

Joint evaluation by the UK AISI and US NIST CAISI reached consistent conclusions: K3 demonstrated stronger exploitation capabilities than GLM-5.2 (ExploitBench: 32% vs. 24%; completing 17 steps on a 32-step simulated enterprise network vs. 11 for GLM-5.2). However, it trailed leading frontier models, achieving zero arbitrary code execution successes across 41 challenge tasks.

Real-world case studies: Chip design in 48 hours and paper replication in 2 hours

The report outlines several end-to-end autonomous tasks accomplished by K3 during testing.

Autonomous chip design

K3 autonomously designed an inference accelerator prototype for a small model using open-source EDA toolchains and public standard cell libraries. Over a continuous 48-hour execution run, it completed RTL layout, optimization, and verification: achieving 100 MHz timing closure within a $4 \text{ mm}^2$ area footprint, simulating >8,700 tokens/sec decode throughput, integrating 1.46M standard cells, 0.277 MiB on-chip SRAM, and a 4-bit MAC array with fused dequantization. The complete design repository has been open-sourced.

Building a custom GPU compiler

K3 constructed MiniTriton, a GPU compiler spanning a custom Python frontend down to code generation for GPU assembly, alongside PyTorch-compatible tensor primitives, reverse-mode autodiff, and distributed communication kernels. Benchmarked on an NVIDIA L20 GPU, MiniTriton achieved geometric-mean speedups over standard PyTorch execution modes, reaching ~90% of theoretical hardware memory bandwidth on large GEMM operations. It successfully trained a GPT model end-to-end, with gradient divergence matching PyTorch native floating-point rounding margins.

GPU kernel optimization

Execution trajectories over a 24-hour sandbox run for a GPU kernel optimization task. Horizontal axis shows active work duration; vertical axis plots speedup relative to baseline. K3 (red) achieved a final +59.7% speedup versus +57.1% for Claude Fable 5 (blue), exhibiting faster initial convergence. Source: Technical Report Figure 14.

Across four GPU kernel tasks, K3 reduced Attention Residual kernel latency from 283.6 ms to 114.4 ms, while accelerating two other kernels by 55.1% and 73.6%. The report notes:

In later stages of development, an early checkpoint of Kimi K3 was already handling the majority of our internal GPU kernel optimization tasks.

Kimi K3 Technical Report §7

Replicating an astrophysics paper

To replicate universal relations in neutron star astrophysics, K3 parsed over 20 reference papers, cross-verified theoretical equations, built numerical simulation pipelines across 300+ equations of state, identified mathematical inconsistencies in published formulas, wrote over 3,000 lines of Python code, and output an interactive dashboard in two hours—a task typically requiring 1-2 weeks for domain researchers.

Additional achievements: In knowledge research, K3 compiled an interactive 42-year history of the AI hardware industry across 120 iterative turns—analyzing 87 quarterly reports, 99 primary PDFs (>11,000 pages), and running >2,800 web queries. In data processing, it deployed 20 parallel sub-agents to process 391 gravitational wave event datasets. It even edited its own video trailer from 56 raw video clips.

Deployment notes: Licensing boundaries, API behavior, and report omissions

For developers planning integration, the report outlines key operational details:

Licensing boundaries

Kimi K3 is released under a permissive custom license allowing commercial use, modification, redistribution, and fine-tuning, subject to two specific commercial provisions:

• If offering Model-as-a-Service (MaaS) API access to third parties and total revenues exceed $20 million over any rolling 12-month period, a separate commercial agreement with Moonshot AI is required.
• If an end-user product exceeds 100 million MAU or $20 million in monthly revenue, prominent attribution to "Kimi K3" must be displayed within the user interface.

Neither restriction applies to internal enterprise usage or access via official API partners. Products embedding model capabilities into specific end-user features do not count as Model-as-a-Service.

API & Deployment specifications

Critical integration requirement: K3 was trained to rely on complete reasoning history across multi-turn conversations. When making multi-turn API calls, developers must pass back the assistant's full response payload—including raw internal reasoning traces and tool call logs—rather than stripping out intermediate thoughts.

Omissions in the technical report

Notable omissions: The 47-page report omits details regarding cluster GPU counts, total training token volumes, wall-clock training duration, and total financial cost. The 2.5x scaling law chart lacks absolute numerical scale values on its compute axis, and hardware memory requirements for local self-hosting are unlisted.

Information inside LLMs flows across sequence length, layer depth, and layer width. K3 introduces targeted modifications across all three axes while maintaining constant hidden width.

Task execution cost is one of the report's most pragmatic findings.

On coding benchmarks, K3 scored 4 points below Claude Fable 5 while consuming only 38% of the cost. At 'high' effort settings, it matched Claude Opus 4.8 max effort performance at one-third the cost.

The report clearly positions K3 behind Claude Fable 5 and GPT-5.6 Sol overall, while outperforming all other baseline models. It details both strengths and weaknesses, including a full cybersecurity assessment omitted from the launch blog.