4× / 128×
Two compression regimes
CSA compresses overlapping blocks 4× and sparsely selects them; HCA compresses non-overlapping blocks 128× and attends densely.
Million-token intelligence · visualized
Inside DeepSeek V4
A source-grounded map of the V4 family: hybrid compressed attention, four-stream manifold-constrained residuals, fine-grained DeepSeekMoE, and a shared-head multi-token prediction path.
1.6Ttotal parameters
49Bactivated parameters
61transformer layers
1Mtoken context
System symbols
CSA
HCA
mHC
DeepSeekMoE
MTP
The V4 family at a glance
Pro and Flash share the same architectural ideas but scale different axes. The interactive switch updates every variant-sensitive count and tensor contract.
128
Local window
Every CSA and HCA layer supplements compressed history with 128 recent uncompressed KV entries.
4 streams
mHC residual width
Dynamic input, output, and doubly-stochastic residual mappings surround both attention and MoE sublayers.
6 + 1
Activated experts
Six fine-grained routed experts run beside one always-on shared expert in every transformer block.
3
Hash-routed layers
The first three MoE layers use token-ID hash routing; later layers use learned Sqrt(Softplus) affinity scores.
1 MTP
Next-token lookahead
One shared-head MTP module learns the next future token and can drive speculative decoding.
2%
KV cache vs GQA8
The report estimates roughly 2% of a BF16 GQA8 cache at one-million-token context.
FP4 + FP8
Mixed inference precision
In instruct checkpoints, MoE expert parameters use FP4 while most remaining parameters use FP8.
Primary sources
Architecture claims are reconciled against DeepSeek’s technical report, released configs, and open reference inference implementation.
Architecture §2, model setups §4.2.1, and cache layout §3.5.
Open report ↗ Pro / Flash configsExact layer schedules, widths, heads, experts, mHC, and compression settings.
Open Pro config ↗ Reference implementationExecutable tensor packing for compressors, indexer, grouped output, mHC, MoE, and MTP.
Open model.py ↗