Task

How can I modify and fine tune the latest qwen 3.8 27B model to use looped transformer with recurrent depth?

create a detailed plan

Previous tasks and results from the chat session for reference

Task 1

How can I modify and fine tune the latest qwen 3.8 27B model to use looped transformer with recurrent depth?

Result 1

Plan: convert Qwen3.8-27B to recurrent depth

I recommend retaining the original model and repeatedly applying a gated middle block, then fine-tuning that block with shared LoRA adapters. Do not begin by collapsing its layers into a much smaller network or adding adaptive per-token halting. Those introduce separate training and cache-management problems.

Open the full illustrated implementation and training plan. It includes the proposed file layout, detailed acceptance tests, memory calculations, experiment schedule, and references.

The architecture facts below were verified against the official checkpoint. The partition, hyperparameters, and training budgets are proposed starting points, not a demonstrated Qwen3.8-27B recipe. No model training was performed.

1. Start from the correct architecture

The official Qwen/Qwen3.8-27B checkpoint uses the Qwen3.5 architectural family. Its configuration specifies:

• 64 text layers, with hidden dimension 5,120.
• 16 repetitions of three Gated DeltaNet layers followed by one full-attention layer, each with an FFN.
• A separate vision encoder and the Transformers architecture Qwen3_5ForConditionalGeneration.

This matters because there are two different recurrence axes: DeltaNet already maintains state across tokens; you want to add recurrence across depth. Qwen’s reasoning_effort option does not implement this architectural change.

First pin the checkpoint revision, tokenizer/chat template, Transformers commit, PyTorch version, and kernel versions. Verify that the untouched checkpoint supports forward, backward, and cached generation in that environment.

2. Use a 16 + 32R + 16 architecture initially

Partition the text decoder using zero-based, half-open ranges:

Embeddings
    ↓
Prefix: layers [0:16]       Run once
    ↓
Core:   layers [16:48]      Run R times, sharing weights
    ↓
Suffix: layers [48:64]      Run once
    ↓
Original final norm + LM head

All boundaries preserve complete four-layer hybrid groups. The core’s 32 layers remain distinct; only repeated executions of the same layer share its weights and adapters.

The base parameter count stays unchanged. These are execution counts, not equivalent independently parameterized model sizes or measured latency ratios.

Preserve the ordinary first pass; damp additional passes:

h = Prefix(Embed(tokens))
h = Core(h)                        # Original first pass

For each additional core pass:
    proposal = Core(h)
    h = h + g * (proposal - h)
logits = LMHead(FinalNorm(Suffix(h)))

Start with a shared per-channel gate g = sigmoid(b), initialized with b = -4, so g ≈ 0.018. With zero initial LoRA updates, R=1 reproduces the original computation; R>1 is only an approximate warm start.

The core already contains residual connections. Do not use h + Core(h), which adds the residual stream twice. Initially avoid new boundary normalization, depth embeddings, or input reinjection. A small gate reduces the initial perturbation but does not guarantee stability.

3. Implement logical-depth caches correctly

Share weights, not temporal state. In the stock implementation, both attention and DeltaNet cache access depend on layer_idx. Repeatedly calling the same layer with its original cache index is incorrect for this design.

Allocate separate state for every logical execution:

• Full attention: independent KV history for each (core_pass, layer).
• DeltaNet: independent convolution history and recurrent matrix state for each (core_pass, layer).
• Prefix and suffix: their own ordinary histories.

Pass a logical cache index explicitly, or use an execution-local cache view. Do not mutate the shared module’s layer_idx inside the loop: checkpoint recomputation and concurrent requests can observe the wrong value.

Keep token position IDs unchanged across depth passes. During full-sequence training, disable persistent decode caching and start each logical DeltaNet sequence scan from its prescribed fresh state. During cached generation, preserve each logical slot’s history across tokens.

Choose R before prefill and keep it fixed for the sequence. Changing R mid-generation invalidates both later-pass core histories and suffix histories. Adaptive depth is not just an early-exit condition added to the loop.

4. Establish correctness before spending on training

Build the loop and cache mechanics on a smaller real hybrid model first, then repeat the checks on the actual 27B checkpoint. An all-attention Qwen3 model cannot validate DeltaNet behavior.

Required end-to-end tests:

1. Original-path parity: converted R=1 matches original logits within a declared numerical tolerance.
2. Causality: changing later tokens does not change earlier logits.
3. Cache equivalence: full-sequence evaluation matches token-by-token decoding and several chunked-prefill layouts at R=1, 2, and 4.
4. Gradient equivalence: checkpointed and non-checkpointed execution agree; gradients reach the core through the frozen suffix.
5. Persistence and isolation: save/reload preserves sharing; independent requests do not contaminate one another; training resumes correctly.

Initially disable speculative decoding/MTP, beam search, and unverified packed-document execution. Packed documents require boundary handling in both attention masks and DeltaNet/convolution state.

5. Fine-tune in stages

A. Record the baseline

Measure untouched-model task accuracy, held-out next-token loss, instruction following, relevant language/vision capabilities, latency, throughput, and peak memory. Evaluate thinking and non-thinking modes separately.

B. Train the two-pass path with adapters

Freeze all original weights. Train the extra-pass gate and LoRA adapters in the middle core, shared across its executions.

Target both mixer types and the FFN:

• Full attention: q_proj, k_proj, v_proj, o_proj.
• DeltaNet: in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj.
• FFN: the gate/up/down projections, verified against the pinned implementation.

Use rank 64 on large projections and rank 8 on the small, 48-output DeltaNet in_proj_a/b projections. Keep convolution kernels, norms, and DeltaNet decay parameters unchanged initially.

Freezing the suffix does not mean running it under no_grad(): its input gradients are necessary to train the core.

C. Use retention distillation and task supervision

Loss = next-token cross-entropy
     + λ × KL(original teacher || recurrent student)

Start with λ=1 and temperature 1 on retention batches; tune on validation. Mask padding and use shifted next-token targets. Reduce teacher matching on independently verified reasoning targets when it conflicts with correct answers: imitating the teacher alone cannot establish improvement beyond it.

Avoid a second 27B teacher copy during LoRA-only training: use the same frozen backbone with adapters disabled and R=1 to obtain teacher outputs without gradients, then restore adapters before student forward/backward. Once base weights are unfrozen, an independent original teacher is needed.

A starting data mixture by tokens is 60% general text covering required languages, 20% verified code/math, and 20% instruction/chat replay. Preserve the original chat template and use assistant-target masking for SFT. Split and deduplicate by document, repository, and problem family before generating teacher data.

D. Introduce variable training depth

• First 20M tokens: sample R=1 with probability 0.2 and R=2 with probability 0.8.
• Remaining 80M, if stable: sample R=1/2/3/4 with probabilities 0.2/0.4/0.2/0.2.
• Use one depth per global microbatch, coordinated across distributed ranks.
• Evaluate every depth. If four passes do not improve the useful accuracy–cost trade-off, retain two.

Expand to 0.5–3B training tokens only after the pilot supports further investment. If adapters plateau, test low-learning-rate core unfreezing as a separate experiment. Add domain SFT next; consider reinforcement learning only after recurrence and verifiers are reliable.

For larger loop budgets, sparse depth supervision or truncated backpropagation may help memory, but detaching states changes the gradient estimator. It is not equivalent to full-depth training.

6. Budget realistically

• 27B BF16 language-model weights: approximately 54GB, before additional multimodal components.
• Illustrative full-parameter Adam training state: about 432GB at 16 bytes/parameter, before activations and temporary buffers. LoRA does not need these optimizer states for frozen weights.
• LoRA pilot planning: budget around two 80GB GPUs or a 140GB-class GPU, short sequences and checkpointing; actual fit must be measured.
• Full/core adaptation planning: an eight-GPU 80GB-class node with ZeRO-3 or FSDP is a reasonable starting allocation, not a guarantee.

For this layout, BF16 full-attention KV storage at 8,192 tokens and batch size one grows from approximately 0.50GiB at R=1 to 1.25GiB at R=4, plus DeltaNet states and overhead. Training activations are a separate, larger consideration. Parameter sharing does not make extra passes free.

Profile the exact training stack before committing to a long run. Estimate duration from measured training-token throughput, including teacher work and checkpoint recomputation.

7. Require evidence that recurrence is responsible

Compare the untouched checkpoint, equally fine-tuned non-looped LoRA, untrained loops, and the trained recurrent checkpoint at each depth. Run both data-matched and total-training-compute-matched comparisons. At inference, compare accuracy against measured latency, memory, and visible output-token budgets.

Suggested acceptance criteria, agreed before seeing results:

• All correctness tests pass.
• At least a 2-percentage-point gain on the chosen primary validation accuracy over the data-matched non-looped control, with a paired 95% interval excluding zero.
• No more than a 1-point retention-composite loss or 2% relative perplexity increase on the trained R=1 path.
• A measured accuracy–cost advantage before claiming efficiency.
• Confirmation across three training seeds and untouched problem families.

Evaluate multimodal retention if it matters: frozen visual weights do not protect against changes in the text decoder. Do not assume that more loops eliminate visible chain-of-thought or generalize to arbitrarily large R.

8. Treat compression as a later, separate conversion

If the goal is lower weight memory, a second experiment could replace:

8 prefix + 24 middle-A + 24 middle-B + 8 suffix

with:
8 prefix + 24 shared core executed twice + 8 suffix

This gives 40 unique decoder layers but still 64 executions. Initialize compatible shared matrices from selected original layers or their means, optionally with pass-specific low-rank residuals, then distill and retrain. Arithmetic averaging does not preserve the network’s function. The large untied input/output matrices also prevent a simple proportional reduction of total parameters.

Relevant references: LoopUS and its code are the closest starting point for pretrained middle-block recurrence; Relaxed Recursive Transformers addresses compression through sharing; Huginn’s recurrent-depth work demonstrates recurrence learned during substantial pretraining. None of those reviewed results establishes this exact Qwen3.8-27B conversion.

Recommended order: original-model baseline → gated uncached recurrence → correct logical caches → 100M-token adapter pilot and controls → held-out compute-matched confirmation → larger training and deployment. Leave compression and adaptive per-token depth until that sequence succeeds.

Task (work on it now)

how much would the traning cost on modal?

Previous tasks and results from the chat session for reference

Task 1

How can I modify and fine tune the latest qwen 3.8 27B model to use looped transformer with recurrent depth?

Result 1

Plan: convert Qwen3.8-27B to recurrent depth

I recommend retaining the original model and repeatedly applying a gated middle block, then fine-tuning that block with shared LoRA adapters. Do not begin by collapsing its layers into a much smaller network or adding adaptive per-token halting. Those introduce separate training and cache-management problems.

Open the full illustrated implementation and training plan. It includes the proposed file layout, detailed acceptance tests, memory calculations, experiment schedule, and references.

The architecture facts below were verified against the official checkpoint. The partition, hyperparameters, and training budgets are proposed starting points, not a demonstrated Qwen3.8-27B recipe. No model training was performed.

1. Start from the correct architecture

The official Qwen/Qwen3.8-27B checkpoint uses the Qwen3.5 architectural family. Its configuration specifies:

• 64 text layers, with hidden dimension 5,120.
• 16 repetitions of three Gated DeltaNet layers followed by one full-attention layer, each with an FFN.
• A separate vision encoder and the Transformers architecture Qwen3_5ForConditionalGeneration.

This matters because there are two different recurrence axes: DeltaNet already maintains state across tokens; you want to add recurrence across depth. Qwen’s reasoning_effort option does not implement this architectural change.

First pin the checkpoint revision, tokenizer/chat template, Transformers commit, PyTorch version, and kernel versions. Verify that the untouched checkpoint supports forward, backward, and cached generation in that environment.

2. Use a 16 + 32R + 16 architecture initially

Partition the text decoder using zero-based, half-open ranges:

Embeddings
    ↓
Prefix: layers [0:16]       Run once
    ↓
Core:   layers [16:48]      Run R times, sharing weights
    ↓
Suffix: layers [48:64]      Run once
    ↓
Original final norm + LM head

All boundaries preserve complete four-layer hybrid groups. The core’s 32 layers remain distinct; only repeated executions of the same layer share its weights and adapters.

The base parameter count stays unchanged. These are execution counts, not equivalent independently parameterized model sizes or measured latency ratios.

Preserve the ordinary first pass; damp additional passes:

h = Prefix(Embed(tokens))
h = Core(h)                        # Original first pass

For each additional core pass:
    proposal = Core(h)
    h = h + g * (proposal - h)
logits = LMHead(FinalNorm(Suffix(h)))

Start with a shared per-channel gate g = sigmoid(b), initialized with b = -4, so g ≈ 0.018. With zero initial LoRA updates, R=1 reproduces the original computation; R>1 is only an approximate warm start.

The core already contains residual connections. Do not use h + Core(h), which adds the residual stream twice. Initially avoid new boundary normalization, depth embeddings, or input reinjection. A small gate reduces the initial perturbation but does not guarantee stability.

3. Implement logical-depth caches correctly

Share weights, not temporal state. In the stock implementation, both attention and DeltaNet cache access depend on layer_idx. Repeatedly calling the same layer with its original cache index is incorrect for this design.

Allocate separate state for every logical execution:

• Full attention: independent KV history for each (core_pass, layer).
• DeltaNet: independent convolution history and recurrent matrix state for each (core_pass, layer).
• Prefix and suffix: their own ordinary histories.

Pass a logical cache index explicitly, or use an execution-local cache view. Do not mutate the shared module’s layer_idx inside the loop: checkpoint recomputation and concurrent requests can observe the wrong value.

Keep token position IDs unchanged across depth passes. During full-sequence training, disable persistent decode caching and start each logical DeltaNet sequence scan from its prescribed fresh state. During cached generation, preserve each logical slot’s history across tokens.

Choose R before prefill and keep it fixed for the sequence. Changing R mid-generation invalidates both later-pass core histories and suffix histories. Adaptive depth is not just an early-exit condition added to the loop.

4. Establish correctness before spending on training

Build the loop and cache mechanics on a smaller real hybrid model first, then repeat the checks on the actual 27B checkpoint. An all-attention Qwen3 model cannot validate DeltaNet behavior.

Required end-to-end tests:

1. Original-path parity: converted R=1 matches original logits within a declared numerical tolerance.
2. Causality: changing later tokens does not change earlier logits.
3. Cache equivalence: full-sequence evaluation matches token-by-token decoding and several chunked-prefill layouts at R=1, 2, and 4.
4. Gradient equivalence: checkpointed and non-checkpointed execution agree; gradients reach the core through the frozen suffix.
5. Persistence and isolation: save/reload preserves sharing; independent requests do not contaminate one another; training resumes correctly.

Initially disable speculative decoding/MTP, beam search, and unverified packed-document execution. Packed documents require boundary handling in both attention masks and DeltaNet/convolution state.

5. Fine-tune in stages

A. Record the baseline

Measure untouched-model task accuracy, held-out next-token loss, instruction following, relevant language/vision capabilities, latency, throughput, and peak memory. Evaluate thinking and non-thinking modes separately.

B. Train the two-pass path with adapters

Freeze all original weights. Train the extra-pass gate and LoRA adapters in the middle core, shared across its executions.

Target both mixer types and the FFN:

• Full attention: q_proj, k_proj, v_proj, o_proj.
• DeltaNet: in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj.
• FFN: the gate/up/down projections, verified against the pinned implementation.

Use rank 64 on large projections and rank 8 on the small, 48-output DeltaNet in_proj_a/b projections. Keep convolution kernels, norms, and DeltaNet decay parameters unchanged initially.

Freezing the suffix does not mean running it under no_grad(): its input gradients are necessary to train the core.

C. Use retention distillation and task supervision

Loss = next-token cross-entropy
     + λ × KL(original teacher || recurrent student)

Start with λ=1 and temperature 1 on retention batches; tune on validation. Mask padding and use shifted next-token targets. Reduce teacher matching on independently verified reasoning targets when it conflicts with correct answers: imitating the teacher alone cannot establish improvement beyond it.

Avoid a second 27B teacher copy during LoRA-only training: use the same frozen backbone with adapters disabled and R=1 to obtain teacher outputs without gradients, then restore adapters before student forward/backward. Once base weights are unfrozen, an independent original teacher is needed.

A starting data mixture by tokens is 60% general text covering required languages, 20% verified code/math, and 20% instruction/chat replay. Preserve the original chat template and use assistant-target masking for SFT. Split and deduplicate by document, repository, and problem family before generating teacher data.

D. Introduce variable training depth

• First 20M tokens: sample R=1 with probability 0.2 and R=2 with probability 0.8.
• Remaining 80M, if stable: sample R=1/2/3/4 with probabilities 0.2/0.4/0.2/0.2.
• Use one depth per global microbatch, coordinated across distributed ranks.
• Evaluate every depth. If four passes do not improve the useful accuracy–cost trade-off, retain two.

Expand to 0.5–3B training tokens only after the pilot supports further investment. If adapters plateau, test low-learning-rate core unfreezing as a separate experiment. Add domain SFT next; consider reinforcement learning only after recurrence and verifiers are reliable.

For larger loop budgets, sparse depth supervision or truncated backpropagation may help memory, but detaching states changes the gradient estimator. It is not equivalent to full-depth training.

6. Budget realistically

• 27B BF16 language-model weights: approximately 54GB, before additional multimodal components.
• Illustrative full-parameter Adam training state: about 432GB at 16 bytes/parameter, before activations and temporary buffers. LoRA does not need these optimizer states for frozen weights.
• LoRA pilot planning: budget around two 80GB GPUs or a 140GB-class GPU, short sequences and checkpointing; actual fit must be measured.
• Full/core adaptation planning: an eight-GPU 80GB-class node with ZeRO-3 or FSDP is a reasonable starting allocation, not a guarantee.

For this layout, BF16 full-attention KV storage at 8,192 tokens and batch size one grows from approximately 0.50GiB at R=1 to 1.25GiB at R=4, plus DeltaNet states and overhead. Training activations are a separate, larger consideration. Parameter sharing does not make extra passes free.

Profile the exact training stack before committing to a long run. Estimate duration from measured training-token throughput, including teacher work and checkpoint recomputation.

7. Require evidence that recurrence is responsible

Compare the untouched checkpoint, equally fine-tuned non-looped LoRA, untrained loops, and the trained recurrent checkpoint at each depth. Run both data-matched and total-training-compute-matched comparisons. At inference, compare accuracy against measured latency, memory, and visible output-token budgets.

Suggested acceptance criteria, agreed before seeing results:

• All correctness tests pass.
• At least a 2-percentage-point gain on the chosen primary validation accuracy over the data-matched non-looped control, with a paired 95% interval excluding zero.
• No more than a 1-point retention-composite loss or 2% relative perplexity increase on the trained R=1 path.
• A measured accuracy–cost advantage before claiming efficiency.
• Confirmation across three training seeds and untouched problem families.

Evaluate multimodal retention if it matters: frozen visual weights do not protect against changes in the text decoder. Do not assume that more loops eliminate visible chain-of-thought or generalize to arbitrarily large R.

8. Treat compression as a later, separate conversion

If the goal is lower weight memory, a second experiment could replace:

8 prefix + 24 middle-A + 24 middle-B + 8 suffix

with:
8 prefix + 24 shared core executed twice + 8 suffix

This gives 40 unique decoder layers but still 64 executions. Initialize compatible shared matrices from selected original layers or their means, optionally with pass-specific low-rank residuals, then distill and retrain. Arithmetic averaging does not preserve the network’s function. The large untied input/output matrices also prevent a simple proportional reduction of total parameters.

Relevant references: LoopUS and its code are the closest starting point for pretrained middle-block recurrence; Relaxed Recursive Transformers addresses compression through sharing; Huginn’s recurrent-depth work demonstrates recurrence learned during substantial pretraining. None of those reviewed results establishes this exact Qwen3.8-27B conversion.

Recommended order: original-model baseline → gated uncached recurrence → correct logical caches → 100M-token adapter pilot and controls → held-out compute-matched confirmation → larger training and deployment. Leave compression and adaptive per-token depth until that sequence succeeds.

Task 2

how much would the traning cost on modal?

Result 2

Modal cost for the recurrent-Qwen training plan

For the 100M-token recurrent-LoRA pilot I proposed, an illustrative budget is about $1,300 for one run. Depending on training speed, the scenarios below range from approximately $650 to $3,300. These are conditional estimates, not measured performance for this custom architecture.

The larger 0.5–3B-token adaptation would be roughly $6,500–$39,000 per run at the middle throughput assumption. This is fine-tuning the existing checkpoint, not pretraining from scratch.

1. Current Modal rental prices

Modal’s official prices, checked September 5, 2026, list:

The estimates below use 2 × H100, plus an illustrative combined allocation of 8 physical CPU cores and 128GiB host RAM. At standard Modal Function rates, that totals approximately $9.30/hour. Memory fit still needs verification; the lowest hourly GPU price does not necessarily give the lowest training cost.

2. Cost of one 100M-token pilot

These speeds are assumptions, not benchmarks or guaranteed bounds. Throughput means input training tokens per second across both GPUs, with all recurrent passes, backward computation, activation recomputation, and teacher forward passes included in step time. Do not multiply the resulting cost by the loop count again.

The 25% contingency is a planning allowance for evaluation, checkpointing, and restarts. Slow kernels, memory offloading, or substantial debugging could exceed it.

Training compute cost = training tokens ÷ (tokens/sec × 3,600) × hourly cost

3. Cost of the broader experiment

Keeping the same allocation and 250 tokens/sec assumption, including the 25% allowance:

A recurrent candidate and a non-looped control across three seeds means six runs: approximately $7,800 for six equal pilot budgets. The non-looped control may run faster; additional candidates and compute-matched controls add cost.

If you later unfreeze the core or full model, profile that separately. An eight-H100 allocation alone costs $758/day, before CPU/RAM. Its throughput cannot be inferred from these LoRA scenarios.

4. Important billing details

• These estimates use base, preemptible pricing. Modal lists non-preemptible execution at 3× base prices, and explicit region selection at 1.15–1.75×. Checkpoint and resume rather than assuming an uninterrupted multi-day run.
• CPU and host RAM are billed at the higher of requested or actual usage. Sandbox/Notebook host-resource rates differ from the Function rates used here.
• Engineering time, dataset purchase/generation, extensive evaluation, and hyperparameter searches are excluded. Taxes, credits, and paid-plan fees are not applied.
• For academic research, Modal advertises up to $10,000 in compute credits, subject to approval.

My recommendation: allocate $20–$50 for a short profiling run first, measuring the actual loop-depth mixture with teacher distillation enabled. If it sustains around 250 tokens/sec and fits in memory, approve approximately $1,300 for the first 100M-token run. No training resources have been launched.

Task (work on it now)

what is 100M here?

Previous tasks and results from the chat session for reference

Task 1

How can I modify and fine tune the latest qwen 3.8 27B model to use looped transformer with recurrent depth?

Result 1

Plan: convert Qwen3.8-27B to recurrent depth

I recommend retaining the original model and repeatedly applying a gated middle block, then fine-tuning that block with shared LoRA adapters. Do not begin by collapsing its layers into a much smaller network or adding adaptive per-token halting. Those introduce separate training and cache-management problems.

Open the full illustrated implementation and training plan. It includes the proposed file layout, detailed acceptance tests, memory calculations, experiment schedule, and references.

The architecture facts below were verified against the official checkpoint. The partition, hyperparameters, and training budgets are proposed starting points, not a demonstrated Qwen3.8-27B recipe. No model training was performed.

1. Start from the correct architecture

The official Qwen/Qwen3.8-27B checkpoint uses the Qwen3.5 architectural family. Its configuration specifies:

• 64 text layers, with hidden dimension 5,120.
• 16 repetitions of three Gated DeltaNet layers followed by one full-attention layer, each with an FFN.
• A separate vision encoder and the Transformers architecture Qwen3_5ForConditionalGeneration.

This matters because there are two different recurrence axes: DeltaNet already maintains state across tokens; you want to add recurrence across depth. Qwen’s reasoning_effort option does not implement this architectural change.

First pin the checkpoint revision, tokenizer/chat template, Transformers commit, PyTorch version, and kernel versions. Verify that the untouched checkpoint supports forward, backward, and cached generation in that environment.

2. Use a 16 + 32R + 16 architecture initially

Partition the text decoder using zero-based, half-open ranges:

Embeddings
    ↓
Prefix: layers [0:16]       Run once
    ↓
Core:   layers [16:48]      Run R times, sharing weights
    ↓
Suffix: layers [48:64]      Run once
    ↓
Original final norm + LM head

All boundaries preserve complete four-layer hybrid groups. The core’s 32 layers remain distinct; only repeated executions of the same layer share its weights and adapters.

The base parameter count stays unchanged. These are execution counts, not equivalent independently parameterized model sizes or measured latency ratios.

Preserve the ordinary first pass; damp additional passes:

h = Prefix(Embed(tokens))
h = Core(h)                        # Original first pass

For each additional core pass:
    proposal = Core(h)
    h = h + g * (proposal - h)
logits = LMHead(FinalNorm(Suffix(h)))

Start with a shared per-channel gate g = sigmoid(b), initialized with b = -4, so g ≈ 0.018. With zero initial LoRA updates, R=1 reproduces the original computation; R>1 is only an approximate warm start.

The core already contains residual connections. Do not use h + Core(h), which adds the residual stream twice. Initially avoid new boundary normalization, depth embeddings, or input reinjection. A small gate reduces the initial perturbation but does not guarantee stability.

3. Implement logical-depth caches correctly

Share weights, not temporal state. In the stock implementation, both attention and DeltaNet cache access depend on layer_idx. Repeatedly calling the same layer with its original cache index is incorrect for this design.

Allocate separate state for every logical execution:

• Full attention: independent KV history for each (core_pass, layer).
• DeltaNet: independent convolution history and recurrent matrix state for each (core_pass, layer).
• Prefix and suffix: their own ordinary histories.

Pass a logical cache index explicitly, or use an execution-local cache view. Do not mutate the shared module’s layer_idx inside the loop: checkpoint recomputation and concurrent requests can observe the wrong value.

Keep token position IDs unchanged across depth passes. During full-sequence training, disable persistent decode caching and start each logical DeltaNet sequence scan from its prescribed fresh state. During cached generation, preserve each logical slot’s history across tokens.

Choose R before prefill and keep it fixed for the sequence. Changing R mid-generation invalidates both later-pass core histories and suffix histories. Adaptive depth is not just an early-exit condition added to the loop.

4. Establish correctness before spending on training

Build the loop and cache mechanics on a smaller real hybrid model first, then repeat the checks on the actual 27B checkpoint. An all-attention Qwen3 model cannot validate DeltaNet behavior.

Required end-to-end tests:

1. Original-path parity: converted R=1 matches original logits within a declared numerical tolerance.
2. Causality: changing later tokens does not change earlier logits.
3. Cache equivalence: full-sequence evaluation matches token-by-token decoding and several chunked-prefill layouts at R=1, 2, and 4.
4. Gradient equivalence: checkpointed and non-checkpointed execution agree; gradients reach the core through the frozen suffix.
5. Persistence and isolation: save/reload preserves sharing; independent requests do not contaminate one another; training resumes correctly.

Initially disable speculative decoding/MTP, beam search, and unverified packed-document execution. Packed documents require boundary handling in both attention masks and DeltaNet/convolution state.

5. Fine-tune in stages

A. Record the baseline

Measure untouched-model task accuracy, held-out next-token loss, instruction following, relevant language/vision capabilities, latency, throughput, and peak memory. Evaluate thinking and non-thinking modes separately.

B. Train the two-pass path with adapters

Freeze all original weights. Train the extra-pass gate and LoRA adapters in the middle core, shared across its executions.

Target both mixer types and the FFN:

• Full attention: q_proj, k_proj, v_proj, o_proj.
• DeltaNet: in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj.
• FFN: the gate/up/down projections, verified against the pinned implementation.

Use rank 64 on large projections and rank 8 on the small, 48-output DeltaNet in_proj_a/b projections. Keep convolution kernels, norms, and DeltaNet decay parameters unchanged initially.

Freezing the suffix does not mean running it under no_grad(): its input gradients are necessary to train the core.

C. Use retention distillation and task supervision

Loss = next-token cross-entropy
     + λ × KL(original teacher || recurrent student)

Start with λ=1 and temperature 1 on retention batches; tune on validation. Mask padding and use shifted next-token targets. Reduce teacher matching on independently verified reasoning targets when it conflicts with correct answers: imitating the teacher alone cannot establish improvement beyond it.

Avoid a second 27B teacher copy during LoRA-only training: use the same frozen backbone with adapters disabled and R=1 to obtain teacher outputs without gradients, then restore adapters before student forward/backward. Once base weights are unfrozen, an independent original teacher is needed.

A starting data mixture by tokens is 60% general text covering required languages, 20% verified code/math, and 20% instruction/chat replay. Preserve the original chat template and use assistant-target masking for SFT. Split and deduplicate by document, repository, and problem family before generating teacher data.

D. Introduce variable training depth

• First 20M tokens: sample R=1 with probability 0.2 and R=2 with probability 0.8.
• Remaining 80M, if stable: sample R=1/2/3/4 with probabilities 0.2/0.4/0.2/0.2.
• Use one depth per global microbatch, coordinated across distributed ranks.
• Evaluate every depth. If four passes do not improve the useful accuracy–cost trade-off, retain two.

Expand to 0.5–3B training tokens only after the pilot supports further investment. If adapters plateau, test low-learning-rate core unfreezing as a separate experiment. Add domain SFT next; consider reinforcement learning only after recurrence and verifiers are reliable.

For larger loop budgets, sparse depth supervision or truncated backpropagation may help memory, but detaching states changes the gradient estimator. It is not equivalent to full-depth training.

6. Budget realistically

• 27B BF16 language-model weights: approximately 54GB, before additional multimodal components.
• Illustrative full-parameter Adam training state: about 432GB at 16 bytes/parameter, before activations and temporary buffers. LoRA does not need these optimizer states for frozen weights.
• LoRA pilot planning: budget around two 80GB GPUs or a 140GB-class GPU, short sequences and checkpointing; actual fit must be measured.
• Full/core adaptation planning: an eight-GPU 80GB-class node with ZeRO-3 or FSDP is a reasonable starting allocation, not a guarantee.

For this layout, BF16 full-attention KV storage at 8,192 tokens and batch size one grows from approximately 0.50GiB at R=1 to 1.25GiB at R=4, plus DeltaNet states and overhead. Training activations are a separate, larger consideration. Parameter sharing does not make extra passes free.

Profile the exact training stack before committing to a long run. Estimate duration from measured training-token throughput, including teacher work and checkpoint recomputation.

7. Require evidence that recurrence is responsible

Compare the untouched checkpoint, equally fine-tuned non-looped LoRA, untrained loops, and the trained recurrent checkpoint at each depth. Run both data-matched and total-training-compute-matched comparisons. At inference, compare accuracy against measured latency, memory, and visible output-token budgets.

Suggested acceptance criteria, agreed before seeing results:

• All correctness tests pass.
• At least a 2-percentage-point gain on the chosen primary validation accuracy over the data-matched non-looped control, with a paired 95% interval excluding zero.
• No more than a 1-point retention-composite loss or 2% relative perplexity increase on the trained R=1 path.
• A measured accuracy–cost advantage before claiming efficiency.
• Confirmation across three training seeds and untouched problem families.

Evaluate multimodal retention if it matters: frozen visual weights do not protect against changes in the text decoder. Do not assume that more loops eliminate visible chain-of-thought or generalize to arbitrarily large R.

8. Treat compression as a later, separate conversion

If the goal is lower weight memory, a second experiment could replace:

8 prefix + 24 middle-A + 24 middle-B + 8 suffix

with:
8 prefix + 24 shared core executed twice + 8 suffix

This gives 40 unique decoder layers but still 64 executions. Initialize compatible shared matrices from selected original layers or their means, optionally with pass-specific low-rank residuals, then distill and retrain. Arithmetic averaging does not preserve the network’s function. The large untied input/output matrices also prevent a simple proportional reduction of total parameters.

Relevant references: LoopUS and its code are the closest starting point for pretrained middle-block recurrence; Relaxed Recursive Transformers addresses compression through sharing; Huginn’s recurrent-depth work demonstrates recurrence learned during substantial pretraining. None of those reviewed results establishes this exact Qwen3.8-27B conversion.

Recommended order: original-model baseline → gated uncached recurrence → correct logical caches → 100M-token adapter pilot and controls → held-out compute-matched confirmation → larger training and deployment. Leave compression and adaptive per-token depth until that sequence succeeds.

Task 2

how much would the traning cost on modal?

Result 2

Modal cost for the recurrent-Qwen training plan

For the 100M-token recurrent-LoRA pilot I proposed, an illustrative budget is about $1,300 for one run. Depending on training speed, the scenarios below range from approximately $650 to $3,300. These are conditional estimates, not measured performance for this custom architecture.

The larger 0.5–3B-token adaptation would be roughly $6,500–$39,000 per run at the middle throughput assumption. This is fine-tuning the existing checkpoint, not pretraining from scratch.

1. Current Modal rental prices

Modal’s official prices, checked September 5, 2026, list:

The estimates below use 2 × H100, plus an illustrative combined allocation of 8 physical CPU cores and 128GiB host RAM. At standard Modal Function rates, that totals approximately $9.30/hour. Memory fit still needs verification; the lowest hourly GPU price does not necessarily give the lowest training cost.

2. Cost of one 100M-token pilot

These speeds are assumptions, not benchmarks or guaranteed bounds. Throughput means input training tokens per second across both GPUs, with all recurrent passes, backward computation, activation recomputation, and teacher forward passes included in step time. Do not multiply the resulting cost by the loop count again.

The 25% contingency is a planning allowance for evaluation, checkpointing, and restarts. Slow kernels, memory offloading, or substantial debugging could exceed it.

Training compute cost = training tokens ÷ (tokens/sec × 3,600) × hourly cost

3. Cost of the broader experiment

Keeping the same allocation and 250 tokens/sec assumption, including the 25% allowance:

A recurrent candidate and a non-looped control across three seeds means six runs: approximately $7,800 for six equal pilot budgets. The non-looped control may run faster; additional candidates and compute-matched controls add cost.

If you later unfreeze the core or full model, profile that separately. An eight-H100 allocation alone costs $758/day, before CPU/RAM. Its throughput cannot be inferred from these LoRA scenarios.

4. Important billing details

• These estimates use base, preemptible pricing. Modal lists non-preemptible execution at 3× base prices, and explicit region selection at 1.15–1.75×. Checkpoint and resume rather than assuming an uninterrupted multi-day run.
• CPU and host RAM are billed at the higher of requested or actual usage. Sandbox/Notebook host-resource rates differ from the Function rates used here.
• Engineering time, dataset purchase/generation, extensive evaluation, and hyperparameter searches are excluded. Taxes, credits, and paid-plan fees are not applied.
• For academic research, Modal advertises up to $10,000 in compute credits, subject to approval.

My recommendation: allocate $20–$50 for a short profiling run first, measuring the actual loop-depth mixture with teacher distillation enabled. If it sustains around 250 tokens/sec and fits in memory, approve approximately $1,300 for the first 100M-token run. No training resources have been launched.

Task 3

what is 100M here?

Result 3

“100M” means 100 million training tokens

It is the total amount of tokenized text processed during one fine-tuning run. It does not mean 100 million parameters, examples, or unique words.

• At a sequence length of 2,048: about 48,828 full sequences.
• At an effective batch size of 65,536 tokens: about 1,526 optimizer updates.
• If your dataset contains 20M tokens and you train for five epochs, that equals 100M training tokens.

For the earlier cost estimate, recurrent passes were already included in the assumed tokens-per-second measurement. A token processed with multiple depth loops still counts as one dataset training token, although it requires more computation. Teacher-distillation passes also increase compute without increasing the stated 100M-token dataset budget.

Task (work on it now)

hwo do I get those tokens?