Read the code behind the name
Open the config for Qwen2.5-7B and look for its model type. It says qwen2. Open Qwen3.8-27B and the wrapper says qwen3_5. Then open Qwen3.8-Flash-Next: qwen4_exp. A release name tells us where a checkpoint belongs in the product family. Its configuration tells us which implementation will run it. Qwen2.5 config, Qwen3.8 config, Flash-Next config.
That mismatch is a useful starting point. To understand how Qwen evolved, we can follow what changed inside the decoder: the projections, normalization, experts, attention state, and eventually the residual stream itself.
Hugging Face Transformers gives us a particularly readable map. Many models have a modular_*.py file that expresses an implementation through inherited components and local overrides. A short subclass can expose a change that would disappear inside hundreds of lines of repeated code. A converter expands those definitions into the modeling_*.py files used at runtime. Modular converter.
We will follow three branches: the dense text decoder, the mixture-of-experts feed-forward block, and the multimodal input path. Qwen3-Next introduces a hybrid of recurrent linear attention and full attention; Qwen3.5 brings that decoder together with the vision branch. Flash-Next changes how information flows between blocks. The block comparison below is a schematic of those changes; its source links lead to the implementation behind each step.
This is a reading of Transformers at commit a72a1c385297bfaf1c8a07b5f4e689a315b90b77, with 27 saved checkpoint configs. The inheritance map describes that snapshot: later refactoring can change which class inherits from which. It does not establish who invented an idea or when it first shipped. Numbers refer to named checkpoints, and configured context limits are not measurements of usable context. The evidence manifest records local hashes; the original Hub revision and retrieval time of the existing downloads are unknown.
The scope is architecture and inference implementation. We will inspect what a forward pass does, what state it retains, and where a kernel can replace the reference computation. Model quality and training recipes need a different investigation.
Qwen-1: a model file that owns the machinery
The original Qwen (Bai et al., 2023) lives in Hub remote code. Its naming looks familiar to anyone who has opened GPT-2: wte, transformer.h, c_attn, and ln_f. The attention module projects the hidden state into queries, keys and values in one operation:
self.c_attn = nn.Linear(config.hidden_size, 3 * self.projection_size)
This is the exact constructor line from the saved modeling_qwen.py, line 269. nn.Linear defaults to a bias here, even though the config also contains no_bias: true; that flag controls other projections. For Qwen-7B, all 32 query heads have their own keys and values: full multi-head attention. The saved config specifies a hidden width of 4,096 and kv_channels: 128. Remote attention implementation, saved config.
There is a small trap in the MLP dimensions. The config says intermediate_size: 22016, but both expanding projections use intermediate_size // 2. The effective branch width is 11,008, the number later written directly into Qwen1.5-7B’s config. Comparing raw config fields without reading their consumers would make this look like a halving of the MLP. Saved MLP implementation, lines 557–576, Qwen1.5 config.
Long context is visible in the forward pass
Qwen-1 applies two inference-time adjustments. Dynamic NTK scaling changes the rotary frequency base according to sequence length; for a padded batch, the code can compute a separate scaling factor for each sample. LogN attention scaling multiplies queries by a position-dependent factor beyond the configured seq_length of 8,192. The latter uses for its one-based position index . Both paths are disabled during training. Dynamic NTK selection, LogN application; the exact local source is available in the saved snapshot.
These are behaviors of this implementation, with finite buffers and explicit conditions. A context number in a config does not describe all of them.
The first kernel interface is an import
The file also manages optional FlashAttention functions for attention, rotary embeddings and RMSNorm, with compatibility branches for different package versions. Its quantized KV-cache path looks for a custom CUDA extension. Architecture, optional acceleration, cache representation and compatibility checks sit together in the model file. Remote-code imports, cache extension loading.
Keep that arrangement in mind. Later implementations move much of this dispatch into shared library interfaces.
Qwen1.5, Qwen2 and Qwen2.5 share an implementation
Qwen1.5 arrived on February 4, 2024, with support in Transformers that removed the need for trust_remote_code. Its implementation name was already qwen2. Qwen2 followed on June 7. Qwen1.5 announcement, Qwen2 announcement.
In the pinned checkout, Qwen2Attention inherits from LlamaAttention, then explicitly constructs biased query, key and value projections. Unlike Qwen-1’s fused c_attn, these are separate q_proj, k_proj and v_proj modules. Qwen2Model inherits from MistralModel and retains support for full and sliding attention masks. There are also MLP and forward-method overrides, so “Llama plus bias” is a useful summary of the attention change, not the entire file. Qwen2 attention, model and masking.
The release-to-release change is easier to see in the configs:
| Saved checkpoint | Layers | Hidden width | Query / KV heads | Configured positions | RoPE θ |
|---|---|---|---|---|---|
| Qwen1.5-7B | 32 | 4,096 | 32 / 32 | 32,768 | 1,000,000 |
| Qwen2-7B | 28 | 3,584 | 28 / 4 | 131,072 | 1,000,000 |
| Qwen2.5-7B | 28 | 3,584 | 28 / 4 | 131,072 | 1,000,000 |
| Qwen2.5-7B-Instruct-1M | 28 | 3,584 | 28 / 4 | 1,010,000 | 10,000,000 |
Sources: Qwen1.5, Qwen2, Qwen2.5, 1M.
Qwen2-7B uses grouped-query attention, or GQA: several query heads share a smaller set of key/value heads (Ainslie et al., 2023). For this checkpoint, 28 query heads share four KV heads. That reduces the keys and values stored per token relative to giving every query head its own KV pair. The ratio describes this checkpoint; the number of heads is not a constant of the qwen2 class.
Qwen2.5-7B retains Qwen2-7B’s decoder geometry. The 1M checkpoint still uses Qwen2ForCausalLM, but it raises both the rotary base and the configured position limit. It also includes dual_chunk_attention_config, which needs separate support in the chosen serving implementation. Loading the same class does not guarantee identical long-context behavior across runtimes. 1M shipped config.
The sliding-window fields deserve the same care. Every saved Qwen2/2.5 config in this sample has use_sliding_window: false. Support exists in the library, but these configs do not request it. The stronger statement that sliding attention was “never switched on” across all Qwen releases would require a wider audit.
Qwen3: normalize each head’s queries and keys
Qwen3’s attention change is small enough to recognize immediately: queries and keys pass through RMSNorm within each head, before rotary embeddings. Values do not get that normalization. The projections use the inherited attention constructor and the Qwen3 config’s bias setting; the saved Qwen3-8B config sets attention_bias: false. Qwen3 attention, saved config.
Here is the essential operation, with shape handling omitted:
q = q_norm(q_proj(x))
k = k_norm(k_proj(x))
v = v_proj(x)
q, k = apply_rotary_pos_emb(q, k, cos, sin)
Thus “Qwen2 minus QKV bias plus QK-norm” captures an important decoder change. head_dim also becomes an explicit config field; projection width can be specified independently of residual width. In the saved Qwen3-30B-A3B config, 32 query heads of dimension 128 produce a query width of 4,096 from a hidden state of width 2,048. Qwen3 MoE config.
The generated Qwen3 modeling file exposes Hub-kernel hooks for RMSNorm and rotary application, while attention selects an implementation through ALL_ATTENTION_FUNCTIONS. A module can keep the same architectural role while its computation is dispatched elsewhere. These are dispatch opportunities; availability depends on the selected backend and hardware. Generated Qwen3 implementation.
The experts branch changes the feed-forward path
A dense decoder sends each token through one MLP. A mixture of experts, or MoE, gives the router several expert MLPs and evaluates a selected subset for each token. Qwen1.5-MoE was announced on March 28, 2024; both it and Qwen2-57B-A14B use qwen2_moe. MoE announcement, Qwen1.5-MoE config, Qwen2-MoE config.
Qwen2MoeSparseMoeBlock has two paths. The router selects experts using a softmax over all experts followed by top-k selection. Alongside the selected experts, a shared MLP processes every token; a learned sigmoid gate scales its contribution before addition. The schematic expression is:
If norm_topk_prob is false, the selected weights remain their original softmax probabilities; they need not sum to one. If it is true, the router renormalizes the selected weights. Router and shared path.
| Saved checkpoint | Routed experts | Selected per token | Renormalize selected weights? | Shared MLP path? |
|---|---|---|---|---|
| Qwen1.5-MoE-A2.7B | 60 | 4 | No | Yes |
| Qwen2-57B-A14B | 64 | 8 | No | Yes |
| Qwen3-30B-A3B | 128 | 8 | Yes | No |
| Qwen3-Next-80B-A3B-Instruct | 512 | 10 | Yes | Yes |
Counts and routing flags come from the saved checkpoint records and their linked configs; shared-path behavior comes from the Qwen2 block, Qwen3 block, and Next inheritance.
Qwen3-MoE removes the shared path and uses Qwen3’s QK-normalized attention. That is a structural change, distinct from increasing the number of experts. Qwen3-Next later brings the shared path back. The family does not move monotonically toward “more of everything.”
Experts have a dispatch interface too
In this checkout, the expert implementation inherits fused, three-dimensional parameter tensors from Mixtral’s expert module. The @use_experts_implementation decorator permits alternate execution paths, including batched_mm, grouped_mm, deepgemm and sonicmoe, alongside the eager implementation. The router’s selection and the matrix operations that evaluate those experts are separate parts of the design. Fused expert module, experts registry.
The vision branch teaches the decoder about space and time
Qwen2-VL adds a vision encoder and a path for inserting its output into the language model. A Conv3d patch embedding handles spatial and temporal patches; a patch merger combines neighboring visual features. The resulting embeddings replace image or video placeholder tokens using masked_scatter. The decoder receives a sequence of embeddings containing both text and visual content. Patch embedding and merger, embedding insertion.
Two details carry forward through the family.
First, vision attention packs variable-sized inputs and uses cumulative sequence lengths, cu_seqlens, to identify boundaries. FlashAttention can consume those offsets directly. The alternative path splits the sequence into chunks before attention, keeping independent inputs apart. This is a representation choice that affects how kernels can execute the workload. Vision attention dispatch.
Second, multimodal rotary position embeddings, or M-RoPE, use temporal, height and width coordinates for visual tokens. Text uses matching coordinates along the three axes. A decoding offset, rope_deltas, reconciles these multimodal positions with continuation generation. In this checkout, the frequency layout is assembled inside rotary embedding classes through recomposition_frequencies; an older snippet using apply_multimodal_rotary_pos_emb would describe a different version of the implementation. Qwen2-VL rotary embedding, position construction.
Qwen2.5-VL and Qwen3-VL take different routes
Qwen2.5-VL adds window attention to its vision encoder. The implementation reorders tokens into windows, permutes the rotary features with them, and selects either full-input or window sequence boundaries for each layer. It then reverses the token permutation. Calling this “just a second cu_seqlens” would miss the reordering that makes the windows work. Video positions also incorporate elapsed time through a tokens-per-second scale. Window implementation, time coordinates.
Qwen3-VL adds learned absolute position embeddings to the vision path and introduces DeepStack: intermediate vision features are merged and added into early language-model layers. The saved 8B checkpoint selects vision block indices 8, 16 and 24; these indices are zero-based. Its text rotary embedding interleaves the three axes, rather than grouping each axis into one contiguous section. Vision encoder, DeepStack injection, interleaved rotary frequencies, saved config.
The Omni branch extends the composition further: the Thinker handles multimodal understanding, while the Talker and speech-generation components produce audio. Its nested configs describe different networks, so comparing an Omni wrapper’s fields directly with a text decoder would be misleading. The explorer above deliberately reads the Thinker’s text_config. We will keep that branch brief here; the vision encoder and positional layout are the pieces needed to understand Qwen3.5. Omni component configs, saved Omni config.
Qwen3-Next: most layers retain a state matrix
Qwen3-Next changes what a token-mixing layer remembers. Its saved 80B-A3B checkpoint repeats three linear-attention layers followed by one full-attention layer, across 48 layers: 36 linear, 12 full. The linear layers implement Gated DeltaNet (Yang et al., 2024), which updates a recurrent state instead of keeping a growing list of keys and values. Saved layer schedule, layer construction.
Before the state update, a depthwise causal convolution mixes local context into queries, keys and values. The model computes a sigmoid update strength and a nonpositive log-decay . The recurrent implementation normalizes queries and keys, scales the queries, then performs the following operation for each token. This is pseudocode for one head; batching and tensor reshapes are omitted:
state = exp(g) * state
remembered_value = k @ state
delta = beta * (v - remembered_value)
state = state + outer(k, delta)
output = q @ state
The delta is a correction: read what the current state associates with the key, subtract that prediction from the new value, then write the scaled difference. The output reads from the updated state. The reference implementation uses a chunked algorithm with chunk size 64 for longer sequences and a recurrent path for single-token decoding when previous state exists. Recurrent reference, decay and dispatch.
Full attention gains a gate
The remaining full-attention layers also change. Their query projection emits both queries and a gate. After attention, a sigmoid of the gate scales the output before o_proj. The saved checkpoint uses head dimension 256, 16 query heads, two KV heads, and rotary embeddings on one quarter of each head. The normalization uses a zero-centered parameterization, with scale , and the MoE block has 512 routed experts, top-10 selection and a shared expert path. Gated full attention, checkpoint config.
The cache is now heterogeneous. Full-attention layers retain K/V tensors; linear layers retain convolution and recurrent states. Replacing three quarters of the attention layers therefore reduces the number of growing K/V caches, but total cache memory still grows with context because the full-attention layers remain. A useful memory estimate must account for both kinds of state and their dtypes. Linear-attention cache implementation.
Kernels become replaceable functions
The DeltaNet operations use use_kernel_func_from_hub_with_fallback. The dispatch can choose a Hub kernel, an installed package implementation, or the PyTorch reference. The shared wrapper rejects packed-sequence inputs when the fallback cannot keep their boundaries separate. “There is a torch fallback” therefore does not mean every accelerated input layout is supported by it. Kernel fallback logic.
One more distinction between weights and implementation: the Next class ignores unexpected checkpoint keys matching ^mtp.*. Multi-token prediction weights can exist in a checkpoint without being executed by this class’s forward pass. Load rules.
Qwen3.5 brings the branches together
Qwen3.5 combines a hybrid text decoder with a vision encoder and multimodal position handling. Its outer config inherits from Qwen3VLConfig; the text decoder inherits components from Next. Both dense and MoE variants exist. A family label alone no longer tells us whether a checkpoint has routed experts. Composite configuration, MoE configuration.
The vision encoder reuses Qwen3-VL’s implementation but explicitly removes its DeepStack machinery. The language path retains interleaved M-RoPE, with sections [11, 11, 10] over 32 frequency pairs, corresponding to 64 rotary dimensions in the sampled models’ 256-dimensional heads. Their vocabulary size is 248,320. Vision changes, rotary changes, 35B config.
Inside DeltaNet, the projection layout changes. Next’s in_proj_qkvz and in_proj_ba become separate in_proj_qkv, in_proj_z, in_proj_b and in_proj_a modules. The underlying recurrent idea remains recognizable, but checkpoint layout and kernel fusion boundaries have changed. Projection replacement.
| Saved checkpoint | Text decoder | Layers | Routed experts / top-k |
|---|---|---|---|
| Qwen3.5-27B | qwen3_5_text | 64 | Dense |
| Qwen3.5-35B-A3B | qwen3_5_moe_text | 40 | 256 / 8 |
| Qwen3.5-397B-A17B | qwen3_5_moe_text | 60 | 512 / 10 |
Sources: 27B config, 35B config, 397B config. The MoE router always renormalizes its selected weights through the inherited Qwen3-VL-MoE router.
The outer multimodal wrapper and text-only class should also be kept distinct. The ForCausalLM implementations ignore visual checkpoint weights, while ForConditionalGeneration includes the multimodal path. The same family can expose both entry points. Dense loading rules, MoE loading rules.
A whole layer can become the kernel boundary
The decorator @use_kernel_forward_from_hub("Qwen3_5GatedDeltaNet") marks the entire DeltaNet layer as replaceable. The pinned registry maps it to device-specific revisions of Atlas-Inference/gdn for NVIDIA GB10 and AMD Strix Halo. That is more extensive than replacing a rotary operation, and more specific than saying a fast implementation runs on every GPU. Layer hook, device mappings.
The sampled Qwen3.6 checkpoints and Qwen3.8-27B still use these classes. That establishes implementation reuse. It does not imply the releases changed only config, or that their weights and input processing are identical. Qwen3.8-2.4T-A95B is another useful example: its saved config directly selects qwen3_5_moe_text, with 512 experts and top-10 routing. 3.6 config, 3.8-27B config, 2.4T config.
Flash-Next changes the residual stream
Qwen3.8-Flash-Next loads as qwen4_exp. We will use the checkpoint’s release name and keep the implementation name visible: exp is part of the class family, not evidence of a separately announced Qwen4 release. Its saved config enables indexed attention, four residual streams and hashed n-gram embeddings. Checkpoint config.
Indexed attention chooses blocks of visible tokens
Where Next has a full-attention layer, this implementation uses QSA indexed attention. The raw checkpoint still labels those layers full_attention; config normalization converts them to indexed_attention. An explorer that only reads the unprocessed strings would miss the change. Config conversion.
The indexer groups causally visible keys into blocks of four and mean-pools each complete block. It scores blocks using its own query/key projections, normalization and rotary embeddings. Scores sum rectified dot products across indexer query heads. The saved token budget is 2,048, so up to 512 complete blocks are selected, with any trailing partial block retained. The resulting selection is combined with the main attention mask. Indexer and selection.
This reference path still scores the available compressed blocks and retains the main K/V cache plus indexer keys. Sparse selection is not proof of constant memory or a measured speedup. The class disables FlashAttention and flex attention in this snapshot; eager and SDPA paths remain. A later architecture can have fewer supported backends while its specialized kernels catch up. Backend flags.
Four streams replace a single residual highway
The residual representation expands to four streams. Before an attention or MLP sublayer, GatedResidual normalizes each stream and computes a mixed input using low-rank projections and sigmoid weights. The sublayer’s output is injected back into the streams through learned gates. A final mixer reduces them to one hidden representation before the language-model head. Gated residual module, decoder flow, final mixer.
There are no conventional decoder pre-norm modules or final self.norm here. Normalization has moved into the stream-mixing machinery. This is a change to the block’s wiring, beyond replacing its attention operator.
PLE gives the second layer a large lexical lookup
The config’s ple_layer_ids: [2] is one-based: PLE runs on the second decoder layer. It hashes bigrams and trigrams into prime-sized embedding tables, with eight heads per n-gram order. EOS-aware shifts prevent lookups from crossing sequence boundaries. The retrieved features are gated against the residual streams and passed through a short dilated convolution. Layer indexing, embedding allocation and hashing, feature injection.
The allocation deserves an explicit calculation. Sixteen hash heads share a total embedding width of 2,560, so each table row has width 160. Summing the prime-sized vocabularies and padding to a multiple of 128 yields 320,001,536 rows. The combined embedding therefore has 51,200,245,760 parameters, or about 95.37 GiB at two bytes per parameter, before other model weights or state. This is derived from the saved config and allocation code, without loading the table; it differs from one older size estimate in a source comment. Calculated record.
The implementation supports special placement for this large embedding and explicitly moves lookup inputs and outputs between devices as needed. It can remain on an accelerator when it fits. “Always off-device” would be too strong. Placement rules, lookup placement.
What the library reuses
We can ask a narrower, reproducible question about influence: which non-Qwen modular implementations import Qwen components in this checkout? Counting distinct destination model directories gives the following results. An import may reuse a helper function or a class; it is not necessarily a subclass relationship.
| Qwen source module | Non-Qwen importing model directories |
|---|---|
qwen2_moe | 12 |
qwen3_moe | 9 |
qwen2 | 8 |
qwen3 | 6 |
qwen3_next | 5 |
qwen2_vl | 14 |
qwen2_5_vl | 8 |
qwen3_vl | 5 |
The generated reuse data lists the files, lines and imported symbols, with subclass relations recorded separately. Counts overlap: one destination can import several Qwen modules. They measure direct imports in modular_*.py files, excluding directories whose names begin with qwen. They omit reuse outside that scope and cannot establish historical priority or feature exclusivity.
That limitation matters because the code story has two layers. One is the architecture: biased projections become QK-normalized attention; expert routing changes; vision adds spatial and temporal coordinates; recurrent state replaces many growing attention caches; multiple streams change residual wiring. The other is the library: reusable classes, standardized caches, and dispatch interfaces make those components available to other implementations.
The kernel boundary keeps moving
Across the implementations we inspected, the replaceable unit grows from individual operations to whole layers:
| Implementation | Where acceleration is selected |
|---|---|
| Qwen-1 remote code | Optional imports and model-local compatibility branches |
| Qwen2 / Qwen3 | Shared attention interface; operation-level Hub hooks |
| MoE implementations | Expert execution interface around fused parameter tensors |
| Qwen3-Next | DeltaNet and convolution functions with Hub, package and torch paths |
| Qwen3.5 | Device-specific replacement of the DeltaNet layer |
| Flash-Next | New indexed-attention path, with a narrower backend set in this snapshot |
The linked implementations in the preceding sections establish these dispatch points. They do not supply a controlled speed comparison: hardware, dtype, sequence layout and installed kernels determine which path runs.
The next time a Qwen checkpoint appears, its release number is only the beginning of the inspection. Read the outer model_type, then the nested decoder config. Follow the module overrides into the generated forward pass. Check what is cached and which backend actually executes it. That route tells us whether we are looking at another checkpoint of a familiar decoder, a new composition of existing branches, or a change to the block itself.
Architectural references
Background for the mechanisms discussed here: the original Qwen report (Bai et al., 2023), grouped-query attention (Ainslie et al., 2023), and Gated DeltaNet (Yang et al., 2024). Checkpoint-specific behavior is documented through the source links throughout the article.
- Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv Preprint arXiv:2305.13245. https://arxiv.org/abs/2305.13245
- Bai, J., Bai, S., Chu, Y., & others. (2023). Qwen Technical Report. arXiv Preprint arXiv:2309.16609. https://arxiv.org/abs/2309.16609
- Yang, S., Kautz, J., & Hatamizadeh, A. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv Preprint arXiv:2412.06464. https://arxiv.org/abs/2412.06464