Kimi K3 / Architecture
Read technical report

Open frontier intelligence · visualized

Inside Kimi K3

A source-grounded map of how text, images, and video enter a shared 93-layer backbone—then move through hybrid KDA / gated-MLA attention, Attention Residuals, and a 2.8-trillion-parameter sparse expert system.

2.8Ttotal parameters
104Bactive parameters
93decoder layers
1Mtoken context

Click a block for details · drag to pan · wheel to zoom

Interactive Kimi K3 architecture flow graph Seven switchable diagrams show Kimi K3's multimodal input path, decoder layer schedule, Kimi Delta Attention, its short causal convolution, Gated Multi-head Latent Attention, Stable LatentMoE, and MoonViT-V2 vision encoder. Native multimodal input Shared 93-layer language backbone Text prompt / tools / code Tokenizer 160K vocabulary Embedding d = 7,168 Image / video native visual input Patchify 14 × 14 patches MoonViT-V2 27 layers · 401M spatial + temporal 2×2 merge + MLP project to d = 7,168 Shared stream interleaved visual + text embeddings up to 1,048,576 tokens Layer 1 · dense attention + dense FFN only dense decoder layer HYBRID GROUP × 23 KDA + MoE KDA + MoE KDA + MoE Gated MLA + MoE 69 KDA 23 global MLA Layer 93 · Gated MLA final global attention + Stable LatentMoE Norm + LM head next-token logits AttnRes selectively retrieves embedding + prior block outputs Layerwise hybrid attention · pre-norm decoder Embedding shared stream LAYER 1 Attention Dense FFN 33,792 hidden REPEAT 3× KDA Stable LatentMoE sparse channel mixing THEN 1× Gated MLA Stable LatentMoE global content mixing AFTER 23 GROUPS Final Gated MLA LatentMoE layer 93 Final norm untied LM head × 23 92 layers = 69 KDA + 23 Gated MLA Block Attention Residuals 8 decoder blocks · size 12 · learned depth attention α 9 sources when the embedding block is counted No explicit positional embedding KDA encodes position through recurrent gating and decay; MLA is NoPE. Kimi Delta Attention · one head at token t xt normalized input Linear projections 96 heads qt, kt short conv (kernel 4) Swish → L2 norm 128-d per head vt short convolution value write content 128-d per head αt / βt channel-wise retention + write strength bounded log-decay ≥ −5 RECURRENT MEMORY State St decay old channels correct with delta rule recurrent across chunks parallel within a chunk StTqt read memory head RMSNorm σ gate full rank yt Wo Why it scales Linear-time recurrent state for long sequences; dense Tensor Core tiles in the chunkwise form. Position signal Implicit in decay and recurrence—no RoPE or explicit positional embedding. KDA short convolution · one channel c at token t Projected Q / K / V each [B,T,96,128] one independent stream per feature channel CAUSAL WINDOW · WIDTH 4 One projected channel c xₜ₋₃,c xₜ₋₂,c xₜ₋₁,c xₜ,c × w₀ × w₁ × w₂ × w₃ Only time is mixed Channel c never reads another channel. Left padding preserves sequence length T. Σ 4 taps SiLU Swish Convolved Q / K / V same [B,T,96,128] Q,K → L2 normalize V → write content no downsampling KDA scan AUTOREGRESSIVE ROLLING STATE previous 3 values / channel combined QKV [B,36864,3] kernel width − 1 = 3 LEARNED FILTER BANKS Wq · Wk · Wv each [12288,1,4] distinct in every KDA layer What it adds Four-token local context before recurrent memory. No channel mixing and no sequence shortening. Gated multi-head latent attention · one token at position t xt normalized input width 7,168 QUERY PATH Q projection low-rank 1,536 96 heads · NoPE KV DOWN-PROJECTION ct = Wcxt compressed KV latent rank 512 Latent KV cache store c≤t, not full K / V 512 values / token LEARNED UP-PROJECTIONS Reconstruct K, V content keys · 128 / head values · 128 / head GLOBAL TOKEN MIXING Causal attention QKT → causal softmax weights × V unrestricted global content interaction õt ungated output FP32 in training σ(Wgxt) full-rank gate one value / output channel ⊙ yt Wo Why the cache is smaller Cache one 512-wide latent per token, then reconstruct head-specific content K / V when attention runs. Why NoPE works here Interleaved KDA layers supply position-sensitive, recency-aware mixing; MLA focuses on global content. Stable LatentMoE · sparse channel mixing Token x full width d = 7,168 COMMON FULL-WIDTH PATH Shared 1 Shared 2 both experts always active W↓ projection 7,168 → 3,584 compact latent path Router sigmoid top-k QB bias SPECIALIZED BANK 1 2 … 896 16 active SiTU-GLU · h = 3,072 Weighted sum then RMSNorm stabilizes scale W↑ 3,584 → 7,168 Σ merge paths Output residual add Stable at extreme sparsity RMSNorm controls routed-branch scale · bounded SiTU-GLU suppresses activation explosion Quantile Balancing derives expert biases from router-score quantiles to target balanced loads. 56× sparse 896 routed / 16 selected MoonViT-V2 · native image and video pathway image / video Patch embed 14 × 14 pixels bias-free projection VISION TRANSFORMER MoonViT-V2 spatial attention / frame temporal attention / frames 27 layers · 12 heads · width 1,024 Temporal pool compress video tokens shared image/video weights 2 × 2 merge pixel shuffle 4× fewer tokens MLP projector 1,024 → 7,168 GELU Shared stream + text 401M vision parameters 27 encoder layers 3,584² max stated image resolution from scratch trained jointly by next-token prediction One objective, one backbone Visual and textual tokens are jointly optimized from the start—no post-hoc modality-alignment stage. Rendered artifacts and their generating code can occupy the same context for vision-in-the-loop work.
System flow symbols
KDA / linear attention Global MLA MoE / channel path Vision path AttnRes depth retrieval

The architecture at a glance

Counts below reconcile the official technical report, model card, and published checkpoint configuration. The diagram collapses repeated layers; it is structural, not drawn to parameter scale.

3 : 1

Hybrid attention cadence

Three recurrent KDA layers, then one global gated-MLA layer; 23 groups plus a final global layer.

69 + 24

Attention composition

69 KDA layers provide efficient long-sequence mixing; 24 NoPE MLA layers preserve global interaction.

16 / 896

Routed experts

Each token selects 16 specialized experts in a 3,584-wide latent space, alongside two shared experts.

8 × 12

Attention Residual blocks

Decoder layers are grouped into size-12 blocks; the last is partial, with the embedding as another source.

7,168

Backbone hidden width

96 attention heads feed a 160K-token output vocabulary through an untied language-model head.

27

MoonViT-V2 layers

A 401M-parameter, 12-head vision transformer trained from scratch with next-token prediction.

1,048,576

Maximum context

No explicit positional embedding: KDA carries position through recurrent decay and gating.

MXFP4 / 8

Native quantization

Quantization-aware training uses MXFP4 weights with MXFP8 activations from the SFT stage onward.

Primary sources

Architecture claims are grounded in Moonshot AI's own report and release artifacts. Last reconciled against the public sources on 2026-08-10.

Technical report

Architecture §2, model comparison Table 1, and Figure 2.

Open PDF ↗ Checkpoint config

Exact layer indices, dimensions, heads, experts, and vision settings.

Open config ↗ Official model card

Released model summary, quantization, context, and deployment notes.

Open repository ↗