DeepSeek V4 Flash

A 284B / 13B-active MoE model built for million-token context with hybrid CSA/HCA attention, mHC residual streams and one MTP layer.

Checkpoint284Btotal parameters

Token-active13B4.6% of checkpoint

Decoder43transformer blocks

Context1Mmaximum sequence

Token path

From token id to next-token logits

Follow the structural path first, then inspect one decoder block at full resolution.

01 · Input spaceToken embedding129,280 vocabulary × 4,096 hidden530M matrix elements

02 · Repeated compute43 decoder blocksSWA + CSA + HCA43 MoE · 0 dense FFN

03 · Output spaceHidden state → logits4,096d hidden → 129,280 token scoressampling chooses the next token

Embedding matrix size is reported as shape × elements only. This view does not assume whether the output head is tied to the input embedding unless the manifest explicitly records it.

Decoder microscope

Layer 1

Select any layer to see the checkpoint-backed attention, feed-forward and residual anatomy.

AttentionSWA

Mechanism: SWA

Sliding-window attention bounds the visible token neighborhood for this layer.

Feed-forwardMixture of Experts

Routed experts: 256
Active routed: 6 / token
Shared experts: 1
Expert width: 2048
Routing: hash

sqrt(softplus) scores; expert bias affects learned selection only; first 3 MoE layers select expert IDs by token-ID hash

Residual topologymHC

Topology: mHC
Expansion: 4 streams
Sinkhorn iters: 20

The residual path carries the block output forward; this panel reports only topology represented by the manifest.

Parameter & memory surface

What is stored, and what participates per token?

Source-backed parameter pools and active-path quantities; stored inventory is not assumed to be read in full for every token.

Stored weight inventory284B

The full checkpoint parameter set exists in storage / device memory planning; this does not imply every stored weight is read for one token.

Active path13B

Parameters reported as active per token by the model manifest.

Selected layer stateSWA

Sliding-window attention bounds the visible token neighborhood for this layer.

Expert fan-out6 + 1

Active routed + shared experts for the selected layer.

Evidence for layer 1

Trace every structural claim back to source

5 sources currently support this layer view.

paperDeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Official technical report for CSA, HCA, mHC, MoE/MTP inheritance and the Flash model setup.

papermHC: Manifold-Constrained Hyper-Connections

DeepSeek's mHC paper: four-stream Hyper-Connections, non-negative pre/post maps and Birkhoff-manifold projection of the residual mixing matrix via Sinkhorn-Knopp.

config · rev fd53f94DeepSeek-V4-Flash config.json

Released checkpoint configuration: 43 backbone layers, 256 routed experts, top-6 routing, first 3 hash-routed MoE layers, route scale 1.5, compression schedule and one next-token prediction layer.

implementation · rev 2b2bebcDeepSeek-V4-Flash inference/model.py

Official inference implementation used to resolve attention scheduling, MoE score/selection/weight semantics, hash routing and the exact hc_pre/hc_post topology around both attention and FFN sublayers.

implementation · rev 553034dDeepSeek-V4-Flash inference/kernel.py

Official TileLang kernel for splitting 24 dynamic routing values into 4 pre weights, 4 post weights and a 4×4 residual matrix projected with 20 Sinkhorn iterations.