01KDALinear attention
Follow projection, gated finite-state memory, delta update and readout through an eight-chapter cinematic walkthrough.
02MLALatent attention
Follow Ling's periodic full-attention block through the 256d query latent, 512d KV latent, 16-head reconstruction, decoupled 64d RoPE path and compact-cache readout.
03Gated DeltaNetRecurrent linear attention
Follow Qwen3.6 from packed QKV projection and four-token causal convolution through state decay, memory prediction, delta correction, recurrent read and z-gated output.
04Gated AttentionGrouped-query full attention
Trace the every-fourth full-attention block through 16Q/2KV grouped heads, partial 64/256 RoPE, causal softmax read and the dedicated sigmoid output gate.
05Qwen MoESoftmax top-k experts
Follow global softmax routing across 256 experts, top-8 selection and renormalization, sparse 512d SwiGLU execution, the sigmoid-gated shared expert and the final routed/shared merge.
06Multimodal boundaryVision → language
Trace image/video patches through the 27-block vision subsystem, 2×2 patch merger and 2,048d visual-feature handoff into the 40-layer language tower.
07Vision towerMultimodal encoder
Inspect Conv3D patch embedding, learned spatial positions, vision RoPE, all 27 pre-norm 16-head attention/MLP blocks and the final 4,608→2,048d patch merger.
08Gated DeltaNetRecurrent linear attention
Inspect the 36 recurrent blocks in Qwen3.8's 3:1 hybrid rhythm, using 16 key heads and 48 value heads around the four-token causal-convolution state update.
09QSACompressed-block sparse attention
Inspect per-layer Qwen Sparse Attention through four-token key blocks, a 4Q/1K × 128d indexer, top-512 complete-block retrieval (= 2,048 tokens) plus the uncompressed causal tail, and the 24Q/2KV grouped-query read.
10Qwen3.8 MoEUltra-sparse softmax experts
Inspect a 512-expert routed bank with top-10 activation, selected-weight normalization, 640d SwiGLU experts and one independently sigmoid-gated shared expert beside the routed mixture.
11Gated ResidualDynamic multi-stream residual
Inspect Qwen's four residual branches through a rank-320 elementwise input mixer and four data-dependent branch-wise 2×sigmoid injection gates around each attention and MoE sublayer.
12PLE / N-gramPer-layer lexical side memory
Inspect the 51B published N-gram embedding pool: hashed bigram and trigram features, eight heads per represented N-gram order, convolutional post-processing and the one-based layer-2 injection into the decoder.
13SWALocal attention
Follow DeepSeek V4 Flash's two-layer local stem through the moving 128-token exact KV window, 64×512d query heads, partial RoPE, causal masking and the handoff into CSA/HCA.
14CSACompressed sparse attention
Trace local exact attention, 4× compressed history, indexer retrieval, sparse read and output merge.
15HCAHierarchical compressed attention
Compare DeepSeek's much more aggressively compressed 128× history path against CSA inside the same cinematic language.
16MoESparse feed-forward
See score formation, expert selection, sparse execution and shared-expert merge, including V4's hash-routed early blocks.
17mHCResidual topology
Expand one hidden state into four residual streams, project the residual matrix onto the Birkhoff manifold, then read and write through the block.
18MTPPrediction side block
Follow hidden state and shared embedding through dual projection, additive fusion, a full inherited block, mHC collapse and vocabulary logits.
19DSAIndexed sparse latent attention
Follow GLM-5.2 from its MLA-backed low-rank path through the 32×128d causal indexer, top-2048 sparse retrieval, IndexShare and the independent 64-head attention read.
20IndexShareLong-context attention reuse
Jump directly to the released four-layer IndexShare rhythm and see why only sparse token indices are reused while every layer keeps its own hidden state, weights and attention computation.
21GLM MoEBias-corrected sigmoid top-k experts
Follow 256 routed experts through sigmoid scoring, correction-biased top-8 ID selection, unbiased selected-score renormalization, sparse SwiGLU execution, 2.5× routed scaling and one always-on shared expert.
22DSparkSpeculative decoding
Open DeepSeek's released V4 Flash speculative sidecar: late target features, semi-autoregressive draft stages, Markov correction and confidence scheduling.
23DFlashSpeculative decoding reference
Inspect the clearly labelled DeepSpec Qwen3-4B block-diffusion reference recipe without conflating it with the V4 checkpoint.