Qwen3.8-Flash-Next

A multimodal ultra-sparse MoE preview of the Qwen4 architecture: a 125B main model, 51B bigram/trigram PLE capacity and a separately published 4B MTP auxiliary parameter pool, with 6B parameters active per token, 48 language blocks repeating 3 × Gated DeltaNet + 1 × Qwen Sparse Attention, and four-branch Gated Residual.

Checkpoint180Bstored parameters

Token-active6.0B4.8% of main model

Decoder48transformer blocks

Context256Kmaximum sequence

Token path

From token id to next-token logits

Follow the structural path first, then inspect one decoder block at full resolution.

01 · Input spaceToken embedding248,320 vocabulary × 2,560 hidden636M matrix elements

02 · Repeated compute48 decoder blocksGated DeltaNet + QSA48 MoE · 0 dense FFN

03 · Output spaceHidden state → logits2,560d hidden → 248,320 token scoressampling chooses the next token

Per-Layer Embedding is a side-memory injection, not part of the ordinary token-embedding matrix: hashed 2-gram + 3-gram features from a 51B published parameter pool enter decoder layer 2. The released implementation supports host-memory offload for this deterministic lookup table.

Decoder microscope

Layer 1

Select any layer to see the checkpoint-backed attention, feed-forward and residual anatomy.

AttentionGated DeltaNet

Key heads: 16
Value heads: 48
Key / value dim: 128 / 128
Short conv: kernel 4

Recurrent state: token history is compressed into a fixed-size state update rather than a full KV history for this layer.

Feed-forwardMixture of Experts

Routed experts: 512
Active routed: 10 / token
Shared experts: 1
Expert width: 640
Routing: learned

router logits → softmax → top-10 routed experts → selected-weight normalization; one shared expert is computed alongside the routed path and modulated by its own sigmoid gate

Residual topologyGated Residual

Topology: GR
Expansion: 4 streams
Mixer rank: 320
Gate: input-dependent

Four residual branches are read and written through fine-grained dynamic gates around each sublayer.

Parameter & memory surface

What is stored, and what participates per token?

Source-backed parameter pools and active-path quantities; stored inventory is not assumed to be read in full for every token.

Stored weight inventory180B

125B main model + 51B N-gram embedding + 4.0B MTP (outside ordinary inference graph). Stored inventory is a deployment/storage boundary; pools outside ordinary inference are not counted as per-token execution.

Active path6.0B

Published active-per-token count for the main model. Side-memory lookups and auxiliary pools are accounted separately.

Selected layer stateGated DeltaNet

Recurrent state: token history is compressed into a fixed-size state update rather than a full KV history for this layer.

Expert fan-out10 + 1

Active routed + shared experts for the selected layer.

Evidence for layer 1

Trace every structural claim back to source

3 sources currently support this layer view.

model-cardQwen3.8-Flash-Next official release

Official release description: early Qwen4 architecture preview, 125B main model + 51B N-gram embeddings, 6B active parameters, GDN + QSA, four-branch Gated Residual, N-gram Embedding, 262,144 native context and optional YaRN extension to 1M.

config · rev 34567a4Qwen3.8-Flash-Next released config.json

Concrete 48-layer 3:1 linear/full schedule, 2,560d hidden width, QSA 4×128d indexer with 4× compression and 2,048-token budget, 16-key/48-value-head GDN, 512 routed experts with top-10 activation, GR hc_count=4/hc_lowrank=320, PLE layer 2, and the 27-block vision encoder.

implementation · rev fc5c5bdTransformers Qwen4-Exp implementation contract

Public Qwen4-Exp model definition and forward contract, including QSA compressed-block selection, one-key-head indexer, incomplete trailing-block handling, Gated Residual elementwise input mixing plus branch-wise injection, PLE one-based layer IDs, sigmoid-gated shared-expert execution and cached-generation state behavior.