Frontier Model Atlas

router projection

hidden6144d

→

gate_proj256 logits

independent routed scoressigmoid

scores do not need to sum to one

selection-only correction

sᵢ+bᵢ=aᵢ

bias changes expert ranking only

schematic expert banktop-8 / 256

1234567891011121314151617181920212223242526272829303132

highlighted IDs are illustrative · count and rule are exact

selected mixture weightsreturn to unbiased sᵢ

→

gather original sigmoid scores → renormalize selected set

sparse execution8 × SwiGLU

E12048dE22048dE32048dE42048dE52048dE62048dE72048dE82048d

always-on dense path1 shared expert

6144d token→shared SwiGLU

not part of the routed top-k pool

MoE output

2.5× routed mixtureshared expert

residual stream

selection bias ≠ mixture weight3 dense layers → 75 MoE layers

0007

GLM-5.2 · MoE router

00 · Router fan-out

One token scores all 256 routed experts

After the first three dense FFN layers, each GLM-5.2 decoder block uses a sparse MoE. The token's 6,144-dimensional hidden state is projected into one router logit per routed expert before any expert MLP executes.

6144d token → 256 routed-expert logits

Implementationhidden_states → gate_proj → 256 router logits

The compact expert grid in this scene is a schematic subset. The released checkpoint has exactly 256 routed experts.