Convert Qwen3.8-27B to a looped model with recurrent depth

1. Verify the actual checkpoint

The official Qwen3.8-27B model card and configuration identify a post-trained vision-language model using the Qwen3.5 architectural family. Do not start from a generic Qwen3 or Llama decoder implementation.

Three different notions must remain separate: DeltaNet recurrence scans across token positions; the new loop repeats computation across depth; Qwen’s reasoning_effort controls its existing thinking behavior. Neither the DeltaNet state nor that API option is a recurrent-depth switch. Multi-token prediction is also a separate mechanism.

Pin the checkpoint revision, tokenizer/processor, chat template, Transformers commit, PyTorch version, and attention/DeltaNet kernels. The saved config mentions 5.8.0.dev0; that is metadata, not a sufficient compatibility test. Before surgery, verify the exact stack can run a forward pass, backward pass, and cached generation with the original checkpoint.

2. Choose the architecture

Use zero-based, half-open ranges: prefix [0:16], core [16:48], suffix [48:64]. The core contains eight complete hybrid groups. Its 32 layers remain different from one another; only repeated executions of the same layer share weights and adapters.

Let C include the core’s existing residual additions, and let S include the suffix, final normalization, and LM head. Use:

Initially, use a shared per-channel gate g = sigmoid(b), with b = −4, giving approximately 0.018. Keep the first pass ungated. With zero-initialized LoRA updates and R=1, this reproduces the original computation. With more passes it is only an approximate warm start. Do not set a sigmoid bias to negative infinity: it prevents gate learning.

Do not use h + C(h), which adds the original residual stream twice. Do not insert a new core-final RMSNorm or depth embedding in version one. Keep the pretrained normalization, RoPE, convolution, and state-update conventions. A small output gate does not guarantee stability inside the repeated block or convergence to a correct answer.

These ratios are not measured latency or whole-model FLOPs. After the first candidate works, compare 24+16R+24 and 8+48R+8. Profile hidden-state movement on held-out calibration text to choose boundaries, but do not equate high cosine similarity with safe reusability. LoopUS motivates middle-block recurrence and damping; this simpler first-pass-preserving design is an adaptation, not a reproduction of its full method.

3. Separate depth from token state

This is the highest-risk implementation detail. The current Transformers implementation indexes both attention caches and DeltaNet histories using layer_idx. Calling the same module repeatedly with the same global cache therefore does not implement independent logical depths.

Use independent cache namespaces for the prefix, each core pass, and the suffix. For example, with zero-based pass index r and local core layer j:

prefix:          logical slots 0 … 15
core pass r:     logical slot 16 + 32*r + j      # j = 0 … 31
suffix layer j:  logical slot 16 + 32*R + j      # j = 0 … 15

The weight identity is the original layer; the cache identity is the logical execution. Allocate the matching expanded layer_types for these slots. Pass the logical slot explicitly into attention and DeltaNet state access, or provide an execution-local cache view. Do not repeatedly mutate the shared module’s layer_idx: it is unsafe under checkpoint recomputation and concurrent requests.

• Full attention: separate K/V for every logical full-attention layer. Append each new token once to each applicable slot.
• DeltaNet: separate convolution history and recurrent matrix state for every logical DeltaNet layer. A depth pass is not another timestep of the same temporal state.
• Positions: a token keeps its original positional coordinates through every pass. Advance token position once per generated token, not once per loop.
• Training: use full-sequence computation with no persistent decode cache. Every logical DeltaNet execution starts from its prescribed fresh sequence state and scans causally through that sequence. Never carry the final state of one full-sequence depth pass into the next: it can leak future tokens.
• Cached inference: prefill every logical slot, then preserve its history across generated tokens and continuation chunks. Gate only the residual-stream output, not cache updates.

Full-sequence, pass-major execution and token-by-token, depth-major decoding are valid evaluation orders of the same causal graph only if these state contracts agree. Use numerical equivalence tests to establish this; plausible generated text is not a correctness test.

4. Build and test the implementation before training

Create a separate research package rather than changing an installed library in place. The official modeling_qwen3_5.py is generated from modular source; if maintaining a Transformers fork, edit its modular source and regenerate. The paths below are a proposed implementation layout, not files created by this plan.

Execution order: config → tests and uncached model → conversion → logical caches → training → evaluation → serving integration. Develop state/loop mechanics on a smaller real hybrid checkpoint with matching semantics; verify its actual config rather than assuming the same layer counts. A small all-attention Qwen3 can exercise generic looping but cannot validate DeltaNet cache correctness.

Write acceptance tests before the corresponding implementation. Cover every supported execution branch without mocks:

1. One-pass equivalence: original versus converted R=1, adapters initially zero, same backend, masks, dtype, and positions. Establish numerical tolerances against the original backend’s own repeatability.
2. Causality: alter later tokens and confirm earlier logits do not change. Include padding and document boundaries.
3. Cache correctness: compare full-sequence logits with token-by-token decode and several chunked-prefill partitions for R=1, 2, and 4; include sequences longer than the convolution history.
4. Gradient correctness: compare checkpointed and non-checkpointed loss/gradients; confirm loss reaches the shared core through the frozen suffix. Zero-initialized LoRA factors need not all have nonzero gradients on the first step.
5. Isolation and persistence: interleave independent requests, reorder batches, reset histories, save/reload, and resume training. Confirm no parameter duplication or lost sharing.
6. Unsupported behavior: reject depth changes on a live cache. Test all other advertised paths, or leave them explicitly disabled.

For packed training documents, reset both full-attention visibility and DeltaNet/convolution state at every boundary. Until the kernels’ packed-sequence semantics are verified, use independent sequences. Initially disable MTP/speculative decoding and beam search; do not imply those optimizations work after architecture surgery.

5. Run staged fine-tuning

Phase A: record the baseline and screen initialization

Record held-out next-token loss, target task accuracy, instruction following, relevant languages, tokens/second, latency, peak memory, and hidden-state/gradient norms. Fix prompts, sampling settings, output limits, and verifier versions. Keep existing thinking and non-thinking modes as separate evaluations.

Before optimization, measure R=1, 2, and 4 with the proposed small gate. Reject a candidate with nonfinite branch activations or severe loss deterioration. Screen smaller cores or gates before increasing training spend. The 16/32/16 partition is a starting point, not an established optimum.

Phase B: learn a usable two-pass path

Freeze all original weights and train the extra-pass gate plus LoRA adapters in the core. Share each physical layer’s adapter across its loop executions. Keep prefix, suffix, embeddings, vision encoder/projector, and output head frozen. Frozen suffix parameters still require differentiation with respect to their input: do not wrap the suffix in no_grad().

Inspect the actual module inventory. Adapt both kinds of mixer, not just Llama-style attention:

• Full attention: q_proj, k_proj, v_proj, o_proj.
• Gated DeltaNet: in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj.
• FFN: verify and target the gate/up/down linear projections in the pinned implementation. Keep convolution kernels, norm parameters, and DeltaNet decay parameters unchanged initially.

Start with full backpropagation through the two core passes. Use non-reentrant activation checkpointing, use_reentrant=False, with no persistent cache mutation. Weight sharing reduces parameter storage, not the number of activations or operations required for backpropagation.

At 65,536 input tokens/update, a 100M-token run is about 1,526 optimizer updates. Count input tokens once, not once per recurrence. The suggested depth mixture averages 2.4 core passes and 1.7× baseline layer executions during multi-depth continuation; actual cost includes backward passes, checkpoint recomputation, evaluation, and teacher work. These budgets are screening allocations, not promises of successful conversion.

Objectives and data

Use shifted next-token targets, mask padding, and stop gradients through the teacher. Start with temperature τ=1 and λ=1 on retention batches; tune on validation. CE learns the target task; teacher-to-student KL limits capability drift. Strong teacher matching alone is not a mechanism for outperforming the teacher. Lower its weight on independently verified reasoning examples when it conflicts with correct targets.

Memory-saving teacher: while all base weights remain frozen, obtain original-model teacher outputs by disabling adapters and using the ungated R=1 path on the same backbone. Run that pass without gradients before the student pass, then restore adapter state before student forward/backward; do not toggle shared module state while checkpointed work is in flight. If any base weights are later unfrozen, preserve a separate original teacher. Compute losses in manageable token/vocabulary chunks; storing full 248,320-way logits for an entire training corpus is generally impractical. Top-k teacher storage is an approximate distillation objective, not exact full-distribution KL.

A concrete initial token mixture is 60% licensed general text covering required languages, 20% verified code/math/problem-solving data, and 20% instruction/chat replay preserving the checkpoint’s template and formatting. During SFT, apply loss to assistant targets rather than user/system text. Keep retention replay throughout domain fine-tuning. Split by document, repository, problem generator, and template family before creating teacher outputs; deduplicate against evaluation.

Phase C: test variable recurrent depth

Sample R during training, but keep it fixed for each sequence’s full computation. Train the final output at the sampled depth. Do not require a prediction loss at every loop exit initially: running the suffix at all exits adds cost and may overconstrain useful intermediate states.

Evaluate every trained depth. If R=4 does not beat R=2 at any useful cost point, keep R=2 rather than forcing more loops. R=6 or 8 can be an extrapolation test only after the trained range is stable. Do not advertise arbitrary-depth generalization based on the architecture alone.

If LoRA stalls while genuine retention/correctness checks pass, test unfreezing the core at approximately 2×10⁻⁶ to 1×10⁻⁵ LR with sharded training. Change one factor at a time. Input-conditioned gates, extra normalization, input reinjection, and depth embeddings are separate ablations, not simultaneous “fixes.”

Phase D: task SFT; optional later RL

After recurrence recovery, fine-tune on the intended application with fixed, verified targets and continued replay. For coding/math, executable tests or exact-answer checks provide stronger evidence than self-reported confidence. Preserve a final-answer-only track and the original visible-reasoning track separately. Loops do not automatically replace chain-of-thought tokens.

Only add reinforcement learning once supervised recurrence is stable and verifiers are reliable. If adding a compute penalty, tune it against measured quality; otherwise the model may learn to exit early without solving the problem. For much larger loop budgets, investigate LoopUS-style sparse supervision or truncated BPTT as distinct algorithms. Detaching hidden states saves memory by changing the gradient path, not by implementing exact full-depth training more efficiently.

6. Prove that extra loops help

The claim to test is not “the modified model generates reasonable text.” It is: after accounting for additional training and inference compute, recurrent depth offers a useful accuracy–cost trade-off without unacceptable capability loss.

Run both data-controlled and compute-controlled comparisons; one cannot substitute for the other. Report unique data, total token exposures, teacher computation, tuning cost, and end-to-end training time. At inference, measure prefill and decoding separately and compare equal answer-token limits as well as equal total latency/compute. A recurrent answer with a much longer visible trace is not evidence that recurrence alone helped.

Use a preselected target suite: executable coding tasks and repository-disjoint tests for a coding goal; exact-answer math with held-out problem families for a math goal; instruction following and held-out language-model loss for retention. If retaining vision capability is a requirement, evaluate it even while visual weights are frozen: changing the text decoder can still degrade multimodal behavior.

Log per-depth task accuracy, next-token loss, gate distribution, residual/update norms, nonfinite values, generated token counts, repetition/format failures, latency, GPU memory, and throughput. A small hidden-state change or high confidence can accompany a wrong answer; neither alone justifies stopping.

Choose depth and hyperparameters on validation, not the final test set. Use paired problem-level comparisons with family clustering where needed; predefine evaluation checkpoints to limit repeated-testing bias. Adjust the proposed thresholds to the application and sample size before seeing results.

7. Budget memory and compute

For approximately 27B language-model parameters, BF16 weights alone require 54GB ≈ 50.3GiB. The full multimodal artifact contains additional components; inspect its actual tensor inventory rather than treating 54GB as its total footprint.

An illustrative mixed-precision Adam layout is 2 bytes of model weight + 2 of gradient + 4 of master weight + 8 of optimizer moments = 16 bytes/trainable parameter. Full 27B training is then approximately 432GB before activations, temporary buffers, and distributed overhead. Actual optimizers differ. This is not the LoRA memory requirement: frozen weights do not need Adam moments or gradients.

For full attention, a BF16 KV cache costs 2 × 4 KV heads × 256 dimensions × 2 bytes = 4,096 bytes/token/logical layer. The proposed layout has 8 + 8R logical full-attention layers:

The DeltaNet column assumes one 48 × 128 × 128 FP32 recurrent matrix per logical layer, about 3MiB. Add convolution histories, batching, metadata, and implementation overhead. These are inference-state calculations, not training activation estimates. At the model’s native 262,144-token context, the R=4 full-attention KV component alone is about 40GiB per sequence. Begin with short contexts.

Run at least 20 warmup steps and 100 measured steps on the exact proposed stack before estimating duration. Record peak allocated/reserved memory, optimizer-step time, input tokens/second, teacher overhead, and communication. Estimate training duration from measured input-token throughput, then add evaluation and checkpoint time. Do not extrapolate from the original model’s inference tokens/second.

8. Only then try compression or adaptive depth

Compression: a separate model conversion

If reducing stored parameters is the actual objective, use a relaxed recursive transformer approach. For example:

Original:    8 prefix + 24 middle-A + 24 middle-B + 8 suffix
Compressed:  8 prefix + 24 shared core, executed twice + 8 suffix
             40 unique decoder layers; 64 executed layers

Preserve groups of four. For each compatible matrix at local offset j, compare initializing the shared weight from the corresponding A layer versus the mean of A and B. An optional SVD-initialized low-rank residual can approximate each pass’s deviation from the shared weight:

Do not average incompatible layer types or claim arithmetic averaging preserves the network’s function. Norms, convolutions, gates, and non-matrix parameters need explicit handling. Use original-model distillation and recovery training. This compression initialization uses two ordinary full core passes to approximate middle-A and middle-B; do not blindly carry over the near-zero extra-pass gate from the parameter-preserving experiment. Pass-specific adapters relax strict sharing but make extrapolation to new pass indices nontrivial.

The input embedding and output projection alone contain about 2.54B parameters. Thus 40 unique decoder layers do not imply a 40/64 reduction of every parameter, nor a “13.5B” result. Count actual unique tensors, including visual/MTP components and adapters. Also distinguish fewer parameters from lower inference cost: this example still executes 64 layers.

Adaptive computation

Start with request-level depth selection before prefill: a fixed R=1, 2, or 4 budget chosen from the prompt and service policy. This avoids mid-sequence cache inconsistency. Calibrate it on held-out accuracy and cost.

Per-token adaptive depth requires a new execution contract for skipped-token state updates, later-pass history, suffix representations, batching, and training. Study adaptive-recursion work only after the fixed-depth model is correct. Do not insert a confidence threshold into the loop and assume the existing cache remains valid.

Stock support for the original Qwen checkpoint in vLLM, SGLang, or another server does not imply support for this custom graph. First use a correctness-oriented custom runner, then implement and test an engine backend with its own cache allocation, batching, save/load, and request-isolation checks.

9. Milestones and stop conditions

Keep an experiment ledger recording the idea, reason, baseline, exact change, data/compute budget, results, and decision. If a candidate fails, distinguish a software defect from a failed modeling hypothesis. Do not repeatedly increase R or train longer without evidence that the failure is data-limited.

Bottom line: port the principle of gated middle-block reuse, not a generic loop around model.forward(). Qwen3.8’s hybrid temporal state makes cache correctness the first engineering milestone; compute-matched held-out improvements are the first research milestone.

10. Sources and what each supports

1. Qwen3.8-27B official model card and config.json: checkpoint identity, hybrid layout, dimensions, vision components, and thinking controls.
2. Transformers Qwen3.5 implementation: decoder residuals, mask selection, physical layer indexing, KV and DeltaNet state access. Pin a commit for implementation; this link tracks main.
3. Park et al., LoopUS, 2026; official code: pretrained-to-looped conversion, selective gates, random deep supervision, and confidence exits. Reported experiments cover earlier Qwen3 models and other backbones, not Qwen3.8-27B. Its default 3B-token experiment is not evidence that this plan will succeed in 100M tokens.
4. Bae et al., Relaxed Recursive Transformers, ICLR 2025: conversion for parameter sharing, depth-wise LoRA, and recovery of compact recursive models.
5. Geiping et al., Scaling up Test-Time Compute with Latent Reasoning, 2025: recurrent-depth pretraining at 3.5B parameters and 800B tokens; useful precedent, not a cheap retrofit recipe. Code.
6. Dehghani et al., Universal Transformers, ICLR 2019: depth recurrence and adaptive halting foundations.
7. Giannou et al., Looped Transformers as Programmable Computers, ICML 2023: constructive computational expressivity, not automatic reasoning gains from pretrained layer reuse.
8. Yu et al., Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning, EACL 2026: supervised alignment of iterations with reasoning steps, a distinct objective beyond simple recurrence.
9. PyTorch activation checkpointing: non-reentrant checkpointing, recomputation trade-offs, and warnings about changed mutable state.
10. DeepSpeed ZeRO: optimizer/gradient/parameter sharding and checkpoint consolidation.
11. Raschka, Looped Transformers, September 2026: technical commentary distinguishing weight storage, execution cost, and visible chain-of-thought. Not used to infer proprietary model internals.

Verified architecture facts and cited research are separate from the proposed experiment. GPU fit, runtime, retained capabilities, and improvements remain to be measured.