Qwen3.6-35B-A3B · exact vision encoder
Inside the 27-block vision tower
Image and video patches are projected to 1,152 dimensions, receive learned spatial position features plus vision RoPE, pass through 27 pre-norm Transformer-style vision blocks, then merge in 2×2 groups before a 2,048-dimensional handoff to the language model.
Released forward path
Pixels to language-width features
Every dimension below comes from the released checkpoint configuration or the inherited Transformers implementation. Patch tiles, rays and motion cues are explanatory geometry only.
Patch embedding
Conv3D turns local pixels into 1,152d tokens
RGB input is grouped with a temporal span of 2 and a 16×16 spatial patch. Kernel and stride are the same, so each projected patch becomes one vision token.
Conv3D(3 → 1152, kernel=stride=2×16×16)vision tokens
Spatial + rotary position
Learn a 48×48 spatial grid, then rotate Q/K
The learned 2,304-entry position table is bilinearly interpolated to each visual grid and added to patch embeddings. Vision RoPE is then applied to query and key states inside every attention block.
patch + interpolated learned position → vision RoPE(Q, K)27 × same block topology
Vision backbone
27 pre-norm residual blocks at 1,152d
Each block performs non-causal 16-head self-attention and a 1,152→4,304→1,152MLP. Qwen3.6's released vision config has no deepstack visual indexes, so no intermediate vision states are injected into the language tower.
LN → self-attention → +residual → LN → MLP → +residual1,152d per patch
Patch merger
2×2 neighboring patches become one 2,048d feature
LayerNorm is applied before reshaping groups of four 1,152d patches into a 4,608d vector. The merger MLP keeps that width once, applies GELU, then projects to the language model's 2,048d hidden size.
1,152d × 4 → 4,608d → GELU → 2,048dBlock-level explorer
Select any of the 27 vision blocks
The checkpoint uses the same structural template at every vision depth. Selecting a block changes the location being inspected; it does not imply that weights or activations are identical.
Vision block 01 / 27
1,152d pre-norm residual block
Input1,152dpatch-token sequence
→LayerNormε = 0.000001pre-attention norm
→Vision self-attention16 × 72djoint QKV · non-causal · RoPE on Q/K
+ residualLayerNormε = 0.000001pre-MLP norm
→Vision MLP1,152 → 4,304 → 1,152gelu_pytorch_tanh
+ residualOne biased linear projection produces Q, K and V jointly, then reshapes into 16 heads.
1,152 ÷ 16 = 72. Attention scaling is therefore based on √72.
Visual tokens attend within each packed image/video sequence without a causal future mask.
Evidence boundary
Exact structure, illustrative activity
Block topology is now represented at implementation level; learned activation values still are not.
Checkpoint dimensions
Depth, widths, heads, patch kernel, merge factor, position-table size, LayerNorm epsilon and output width come from released config and implementation.
Forward topology
Conv3D patch embed, learned position interpolation, vision RoPE, pre-norm attention/MLP residual blocks and the final patch merger mirror the released-family forward path.
On-screen activity
Patch colors, token travel, highlighted heads and any apparent attention strength are teaching cues. They are not captured Qwen3.6 activations.
No deepstack injection
The released Qwen3.6 vision config records an empty deepstack visual index list, so this Atlas does not invent intermediate vision-to-language injection points.
Primary evidence