← Back to 3D modelInclusionAI · Ling 3.0
Model anatomy
Ling 3.0 tiny
A 7.9B hybrid-linear MoE model that alternates three KDA layers with one MLA layer, while activating about 1.3B parameters per token.
Token path
From token id to next-token logits
Follow the structural path first, then inspect one decoder block at full resolution.
Embedding matrix size is reported as shape × elements only. This view does not assume whether the output head is tied to the input embedding unless the manifest explicitly records it.
Decoder microscope
Layer 2
Select any layer to see the checkpoint-backed attention, feed-forward and residual anatomy.
AttentionKDA
- Heads
- 16
- Head dim
- 128
- Short conv
- kernel 4
- Decay gate
- safe gate · lower bound -5
Recurrent state: token history is compressed into a fixed-size state update rather than a full KV history for this layer.
Feed-forwardMixture of Experts
- Routed experts
- 128
- Active routed
- 8 / token
- Shared experts
- 1
- Expert width
- 512
- Routing
- learned
sigmoid scores; expert bias affects selection only; 8 groups → keep 4 → top-8 routed experts
Residual topologyStandard stream
- Topology
- standard
The residual path carries the block output forward; this panel reports only topology represented by the manifest.
Parameter & memory surface
What is stored, and what participates per token?
Source-backed parameter pools and active-path quantities; stored inventory is not assumed to be read in full for every token.
The full checkpoint parameter set exists in storage / device memory planning; this does not imply every stored weight is read for one token.
Parameters reported as active per token by the model manifest.
Recurrent state: token history is compressed into a fixed-size state update rather than a full KV history for this layer.
Active routed + shared experts for the selected layer.
Evidence for layer 2
Trace every structural claim back to source
4 sources currently support this layer view.
model-card · rev 405ba96InclusionAI/Ling-3.0-tiny model card
Official overview: 7.9B total / 1.3B active, 3:1 KDA–MLA, 128 routed experts, 8 routed + 1 shared active per token.
config · rev 405ba96Ling-3.0-tiny config.jsonCheckpoint truth for layer count, dimensions, MoE routing, KDA and MLA hyperparameters.
implementation · rev 405ba96modeling_bailing_moe_v3.pyOfficial forward implementation. It proves the 3 KDA + 1 MLA schedule, dense layer 0, and MoE routing semantics: sigmoid scores, bias-only selection, grouped top-k, normalized/scaled mixture weights and additive shared expert.
paperKimi Linear: An Expressive, Efficient Attention ArchitectureCanonical KDA mechanism reference: finite-state matrix recurrence, fine-grained diagonal decay gate, delta-rule update and chunkwise implementation.