Guided architecture paths · source-backed

Learn the mechanisms as a connected system.

The Atlas now contains enough independent cinematics that a flat index is no longer the best way to enter. These paths order the existing source-backed mechanisms by concept, keep model boundaries explicit, and use comparison as the bridge between implementations.

4 guided tracks14 cinematic mechanisms23 indexed systems

Track 01 · attention

From recurrent state to indexed sparse history

Build a comparative mental model of how current architectures trade recurrent state, exact local or full attention, compressed history and token-level sparse retrieval.

Not startedStart path

Learning outcome

You leave able to read KDA/MLA, Qwen's GDN/full-attention and GDN/QSA rhythms, SWA/CSA/HCA, and GLM-5.2 DSA/IndexShare as distinct ways of retaining, selecting and reading context.

01Compare layer rhythmscompareStart with the shared Attention axis so recurrent, full/local, compressed and indexed-sparse systems are aligned before diving into implementation detail.Cross-model comparison 02KDAcinematicSee finite-state linear attention as a recurrent memory update rather than a token-to-token matrix.Ling 3.0 tiny · Linear attention 03MLAcinematicStay inside Ling and contrast recurrent KDA with the periodic full-attention block built from compact query/KV latents and decoupled RoPE.Ling 3.0 tiny · Latent attention 04Gated DeltaNetcinematicMove to Qwen3.6 and compare its gated delta-rule recurrent state update with the KDA memory you just saw.Qwen3.6-35B-A3B · Recurrent linear attention 05Gated AttentioncinematicInspect Qwen3.6's every-fourth return to causal full attention through grouped 16Q/2KV heads, partial RoPE and output gating.Qwen3.6-35B-A3B · Grouped-query full attention 06QSAarchitectureNow keep the same 3:1 recurrent rhythm but replace the periodic full-attention refresh with Qwen3.8's independently indexed QSA layer.Qwen3.8-Flash-Next · Compressed-block sparse attention 07SWAcinematicEnter DeepSeek through its exact-local stem: a moving 128-token causal KV window before any compressed long-range machinery appears.DeepSeek V4 Flash · Local attention 08CSAcinematicExtend that exact local read with 4× compressed history, indexer retrieval and sparse long-range access.DeepSeek V4 Flash · Compressed sparse attention 09HCAcinematicIncrease the history compression hierarchy and compare HCA directly against CSA.DeepSeek V4 Flash · Hierarchical compressed attention 10DSAcinematicMove to GLM-5.2 and contrast QSA/DeepSeek retrieval with DSA's MLA-backed top-2048 retrieval of original causal token positions.GLM-5.2 · Indexed sparse latent attention 11IndexSharecinematicFinish the attention track by following how GLM-5.2 amortizes sparse selection: one full indexer result can feed the following layers in a four-layer IndexShare group.GLM-5.2 · Long-context attention reuse

Track 02 · sparse experts

How routers turn width into conditional compute

Compare source-backed sparse-FFN systems without flattening their routing rules into a generic MoE diagram.

Not startedStart path

Learning outcome

You leave able to separate router score function, selection adjustment, top-k choice, mixture weighting, expert execution and shared paths as independent design decisions — including Qwen3.8's 512-expert ultra-sparse bank.

01Compare routing systemscompareUse the normalized MoE axis to line up expert pool, active share, routing rule and shared path across the available checkpoints.Cross-model comparison 02Qwen MoEcinematicTrace Qwen3.6 global softmax over 256 experts, top-8 selection, renormalization and the sigmoid-gated shared expert.Qwen3.6-35B-A3B · Softmax top-k experts 03Qwen3.8 MoEarchitectureScale the same conditional-compute idea to Qwen3.8's 512 routed experts while activating only ten routed experts per token.Qwen3.8-Flash-Next · Ultra-sparse softmax experts 04MoEcinematicContrast DeepSeek's top-6 learned routing and the early token-ID hash path with the Qwen softmax designs.DeepSeek V4 Flash · Sparse feed-forward 05GLM MoEcinematicThen inspect GLM-5.2's 256-expert top-8 router, where a learned correction bias changes expert-ID selection but the selected mixture weights come from the original sigmoid scores before 2.5× scaling.GLM-5.2 · Bias-corrected sigmoid top-k experts

Track 03 · residual, side memory & prediction

What happens around the transformer block

Move beyond attention and FFN internals into the topology that carries state between blocks, lexical side memory injected into a layer, and auxiliary prediction paths attached to a backbone.

Not startedStart path

Learning outcome

You leave able to distinguish standard residuals, Qwen Gated Residual, DeepSeek mHC, Qwen's deterministic PLE side memory and DeepSeek's MTP prediction side block as separate architectural axes.

01Compare residual topologycompareAlign conventional single-stream residuals against Qwen3.8's four-branch Gated Residual and DeepSeek's four-stream mHC.Cross-model comparison 02Gated ResidualarchitectureFollow four residual branches through Qwen3.8's rank-320 data-dependent read and write gates around a sublayer.Qwen3.8-Flash-Next · Dynamic multi-stream residual 03PLE / N-gramarchitectureStep outside the ordinary embedding matrix and inspect Qwen3.8's 51B deterministic N-gram lookup side memory injected at decoder layer 2.Qwen3.8-Flash-Next · Per-layer lexical side memory 04mHCcinematicContrast GR with DeepSeek mHC: four residual streams plus constrained cross-stream mixing through one block.DeepSeek V4 Flash · Residual topology 05MTPcinematicThen follow the prediction side block from shared embedding fusion through an inherited transformer block and vocabulary logits.DeepSeek V4 Flash · Prediction side block

Track 04 · inference

After the checkpoint: speculative generation systems

Separate the released checkpoint architecture from serving-time systems that try to produce more accepted tokens per target-model step.

Not startedStart path

Learning outcome

You leave able to distinguish the model itself from its speculative sidecar and from a clearly labelled external reference recipe.

01DSparkinferenceOpen the released V4 Flash speculative sidecar: late target features, semi-autoregressive drafting, correction and confidence scheduling.DeepSeek V4 Flash · Speculative decoding 02DFlashinferenceUse the DeepSpec Qwen3-4B block-diffusion recipe as a labelled reference rather than conflating it with the V4 checkpoint.DeepSpec reference · Speculative decoding reference

Already know the mechanism you want?

Return to the full Atlas index ↗Open cross-model comparison ↗