Model anatomy

GLM-5.2

A 744B / 40B-active MoE model with a 78-layer MLA-based DSA backbone, four-layer IndexShare groups and a released 1M-token context.

Checkpoint744Btotal parameters

Token-active40B5.4% of checkpoint

Decoder78transformer blocks

Context1Mmaximum sequence

Token path

From token id to next-token logits

Follow the structural path first, then inspect one decoder block at full resolution.

01 · Input spaceToken embedding154,880 vocabulary × 6,144 hidden952M matrix elements

02 · Repeated compute78 decoder blocksDSA75 MoE · 3 dense FFN

03 · Output spaceHidden state → logits6,144d hidden → 154,880 token scoressampling chooses the next token

Embedding matrix size is reported as shape × elements only. This view does not assume whether the output head is tied to the input embedding unless the manifest explicitly records it.

Decoder microscope

Layer 4

Select any layer to see the checkpoint-backed attention, feed-forward and residual anatomy.

AttentionDSA

Attention heads: 64
Q / KV low-rank: 2048 / 512
QK split: 192 no-PE + 64 RoPE
Sparse retrieval: top-2048
Indexer: 32 × 128d
IndexShare role: reuse prior full indexer

MLA-backed sparse attention retrieves at most 2,048 causal token positions; this layer reuses the previous full IndexShare selection.

Feed-forwardMixture of Experts

Routed experts: 256
Active routed: 8 / token
Shared experts: 1
Expert width: 2048
Routing: learned

Sigmoid router scores; an expert-score correction bias affects top-k selection, while un-biased selected scores are renormalized and scaled by 2.5 before expert outputs are merged with one always-on shared expert.

Residual topologyStandard stream

Topology: standard

The residual path carries the block output forward; this panel reports only topology represented by the manifest.

Parameter & memory surface

What is stored, and what participates per token?

Source-backed parameter pools and active-path quantities; stored inventory is not assumed to be read in full for every token.

Stored weight inventory744B

The full checkpoint parameter set exists in storage / device memory planning; this does not imply every stored weight is read for one token.

Active path40B

Parameters reported as active per token by the model manifest.

Selected layer stateDSA

MLA-backed sparse attention retrieves at most 2,048 causal token positions; this layer reuses the previous full IndexShare selection.

Expert fan-out8 + 1

Active routed + shared experts for the selected layer.

Evidence for layer 4

Trace every structural claim back to source

3 sources currently support this layer view.

config · rev b4734deGLM-5.2 config.json

Released checkpoint configuration: 78 layers, 6144 hidden width, 1,048,576 maximum positions, first 3 dense MLPs then sparse MoE, 256 routed experts with top-8 selection, MLA dimensions and the exact per-layer IndexShare schedule.

implementation · rev eb4d9e2Transformers GLM-MoE-DSA implementation

Released Transformers forward implementation used to resolve MLA projection flow, the 32×128d DSA indexer, causal top-2048 retrieval, sigmoid MoE routing and the always-on shared-expert merge.

paperGLM-5.2: Built for Long-Horizon Tasks

Official architecture release explaining four-layer IndexShare, 1M-context training and the separate GLM-5.2 MTP/serving improvements.