Frontier Model Atlas

SWASWACSAHCACSAHCA43-layer backbone · opening stem → alternating compressed attention

older history

1234567891011121314qexact local KV · 128-token horizon

future masked

hidden4096d

→

Q LoRA1024

→

64 × 512d Q heads

448d content64d RoPE

causal ∩ localj ≤ i · i − j < 128

exact local attention readQKᵀ → local causal mask → softmax → V mixture

SWAexact local only→CSAlocal + 4× compressed top-512→HCAlocal + 128× compressed dense

0006

DeepSeek V4 Flash · SWA

00 · Local-first stem

Begin V4 Flash with two exact local-attention blocks

DeepSeek V4 Flash starts its 43-layer backbone with two sliding-window attention blocks before switching into the alternating CSA/HCA regime. SWA keeps the recent causal neighborhood exact: no compressed-history branch and no Lightning Indexer are needed in these opening layers. The released local horizon is 128 tokens.

backbone stem = 2 × SWA · window 128

Implementationlayers 0–1 · attention = SWA

The token count drawn on screen is schematic; layer placement and the released window size are source-backed.