Qwen · Qwen3.8 / Qwen4-Exp
A multimodal ultra-sparse MoE preview of the Qwen4 architecture: a 125B main model, 51B bigram/trigram PLE capacity and a separately published 4B MTP auxiliary parameter pool, with 6B parameters active per token, 48 language blocks repeating 3 × Gated DeltaNet + 1 × Qwen Sparse Attention, and four-branch Gated Residual.
Each blueprint column is one transformer block. Hover or keyboard-focus a linked cell to read attention, feed-forward and residual structure as one aligned layer; linked cells open the corresponding walkthrough or inspector. DSA cells additionally preserve whether the layer runs a full indexer or reuses a shared IndexShare selection. QSA cells preserve their per-layer sparse retrieval identity, while PLE is marked on the decoder layer receiving N-gram features. MoE cells preserve any layer-level routing transition encoded by the checkpoint manifest. Structural encoding only — it does not imply benchmark quality, throughput or FLOPs.