How Kimi K3's RL Improves on DeepSWE's Recipe — and Why
A technical comparison of the reinforcement learning in Kimi K3 (Moonshot AI, July 2026) against DeepSWE-Preview (Agentica × Together AI, July 2025), the fully open-sourced coding agent trained with GRPO++ on R2E-Gym.
The two systems in one paragraph each
DeepSWE-Preview trained Qwen3-32B with pure, fully on-policy RL — no SFT, no distillation — on 4,500 software-engineering tasks from R2E-Gym, over six days on 64 H100s. Its algorithm, GRPO++, is GRPO hardened with tricks borrowed from DAPO, Dr. GRPO, and RLOO/LOOP, plus its own compact filtering. The reward is a sparse 0/1 test-pass signal, the agent runs one fixed 4-tool scaffold in Docker containers orchestrated by Kubernetes, and it reached 42.2% Pass@1 (59% with hybrid test-time scaling) on SWE-Bench-Verified — state of the art for open-weight coding agents in mid-2025.
Kimi K3 makes RL the central post-training stage of a 2.8T-parameter MoE with a 1M-token context. It trains nine domain×effort expert policies (general, agentic, coding × low/high/max reasoning effort) with a partial-rollout, extremely off-policy scheme built on the Kimi K2.5 objective, layered verifiable/generative/budget rewards, harness-randomized "white-box" environments, and Firecracker-microVM sandboxes (AgentENV) — then merges the experts back into one model with multi-teacher on-policy distillation.
Scale caveat. This is not an apples-to-apples contest: DeepSWE is a 32B open-recipe research release; K3 is a frontier-lab system a year later. What makes the comparison instructive is that K3's design choices target exactly the failure modes DeepSWE documented — reward collapse, straggler cost, censored long trajectories, and a context-scaling plateau. (Naming note: the "DeepSWE" appearing in K3's evaluation tables is the 2026 datacurve benchmark at deepswe.datacurve.ai, where K3 scores 67.5; it shares only the name with Agentica's agent.)
1. The objective: from patched PPO clipping to a staleness-tolerant trust region
| DeepSWE (GRPO++) | Kimi K3 (K2.5 objective) | |
|---|---|---|
| Update rule | PPO surrogate with DAPO "clip-high" (raised upper ratio bound, sign-dependent) | Sign-agnostic gradient masking: gradients zeroed for tokens whose log-ratio leaves [α,β], regardless of advantage sign |
| Trust region | None beyond ratio clipping (KL loss removed, per DAPO) | Per-token penalty τ(log πθ/πold)² anchored to the sampling policy |
| Advantage | Leave-one-out group baseline, no std division (Dr. GRPO) | Plain group-mean baseline r − r̄, no std division |
| Normalization | Divide by max context length (Dr. GRPO length fix) | Divide by N = total generated tokens in the batch |
| Optimizer | (standard AdamW-family via verl/rLLM) | MuonClip |
K2.5/K3: L = E_x [ 1/N Σ_j Σ_i Clip(ρ_ji, α, β) · (r(x,y_j) − r̄(x)) − τ (log ρ_ji)² ]
ρ_ji = π_θ(token i of response j) / π_old(·) [K2.5 Eq. 1]
Why it's an improvement. Both teams removed GRPO's std-normalization (difficulty bias) and per-response length bias — that part is shared DNA. The difference is what bounds the update. PPO-style clipping only suppresses a token's gradient when the ratio moves outward in the direction its advantage pushes; with heavily stale data it neither detects nor bounds drift in the other direction. K3's masking is a pure off-policy-drift detector on |log ρ|, and the squared log-ratio term is a smooth local trust region around whatever policy generated the data. The K2.5 report calls this combination "essential for maintaining training stability" in long-horizon multi-step tool use — and it is precisely what lets K3 train on trajectories that are several policy versions old (next section). DeepSWE never needed that property because it stayed on-policy; but staying on-policy is itself the limitation.
2. Rollout regime: synchronous on-policy vs. partial-rollout off-policy
Why it's an improvement. DeepSWE's iteration time is set by its slowest of 512 rollouts, and its compact filtering masks out every trajectory that hits max context (64K), max steps (100), or a 20-minute generation timeout. That was the right stability fix at the time — it prevents rewarding accidental passes — but it has two structural costs: straggler compute is wasted, and the hardest, longest tasks are systematically censored from the gradient, capping the horizon the policy can ever learn. K3's partial rollout starts optimization as soon as a fraction λ of trajectories completes and resumes paused trajectories in later iterations with persistent model and sandbox state. A single trajectory legitimately spans several policy versions; the objective of §1 is what makes that staleness safe. Result: RL over million-token, thousands-of-tool-call trajectories becomes economically and numerically feasible — the regime DeepSWE could not enter.
3. Rewards: from one sparse bit to a layered, anti-hacking reward stack
- DeepSWE: a single sparse ORM — 1 if the patch passes selected Pass2Pass + Fail2Pass tests within 5 minutes, else 0. Simple and unhackable-by-construction for verifiable SWE tasks, but it provides no signal for partial progress, no token-efficiency pressure, and nothing for non-verifiable qualities.
- Kimi K3:
- rule-based outcome rewards for verifiable tasks (as in DeepSWE);
- budget-controlled effort rewards: reward overridden to −1 when output exceeds τ·b0(x), a per-problem budget, with τ annealed to produce max/high/low-effort experts (plus K2.5's Toggle alternation to avoid length overfitting; −25–30% tokens at equal accuracy);
- agentic generative reward model for non-verifiable tasks: tournament-style binary comparisons under a mandatory read→rubric→score→record protocol, with a verbosity budget (σ·ℓ0) that auto-fails overlong candidates;
- dense per-token distillation rewards (MOPD): clip(sg(log πteacher/πθ), ±Rmax) turns the sparse-outcome framework into a dense-signal one for merging the nine experts;
- domain-specific shaped rewards: kernel tasks scored against expert implementations and the hardware roofline with an evolving hacking-detection system (penalizing CUDA-graph replay, input caching, precision tricks); web-dev tasks zeroed on faked artifacts; Autonomous Execution Tasks judged by hidden verifiers on held-out scenarios.
Why it's an improvement. Sparse 0/1 rewards limit sample efficiency and leave entire capability classes untrainable (writing quality, efficiency, artifact quality). More capable agents also hack harder: K3's report notes agents caused kernel panics in container sandboxes and continually invented new kernel-task exploits. The reward stack pairs every generative or performance-based signal with an explicit counter-hack (verbosity budgets, rubric protocols, hidden verifiers, detection systems, microVM isolation that permits exploration without letting it corrupt the reward). DeepSWE's compact filtering solved one specific hack — accidental passes reinforced across steps — while K3 generalizes the defense.
4. Environments: one scaffold vs. harness-randomized white-box training
Why it's an improvement. DeepSWE trained and evaluated inside one fixed R2E-Gym scaffold (bash, search, file editor, submit) on 4.5K Python-repo tasks — and its own "attempted experiments" section reports that alternative datasets and environments did not work well. K3 treats the harness itself as data: a unified white-box environment composes tool schemas, system prompts, context-management strategies, skills, memories, and subagents into many configurations (instantiating Kimi Code, Claude Code, Codex, OpenClaw, Hermes, and novel ones), so the policy cannot overfit one tool grammar. Task supply is industrialized by knowledge-graph-guided synthesis plus purpose-built suites (kernels, web dev, mock Gmail/Notion/Slack "living" environments, verify-in-the-loop AET). The observable payoff: K3 posts strong scores under third-party harnesses (Claude Code, mini-SWE-agent, Codex-style tools), where a single-scaffold model typically degrades.
5. Token efficiency: trained in, not bolted on
Why it's an improvement. DeepSWE found output-token scaling saturates (≤2% gain beyond 32K context) and got its headline 59% only through best-of-16 trajectory sampling with two trained verifier models — a 16×+ inference bill per task. K3 makes compute-vs-quality a training-time variable: budget-controlled RL produces low/high/max effort levels exposed as a single API field, and MOPD merges the nine experts so one model serves every operating point in a single pass, no verifier ensemble required. Emergent per-step efficiency in DeepSWE (thinking tokens fell as steps rose) becomes an explicitly optimized objective in K3.
6. Systems: Docker/Kubernetes vs. resumable microVMs + KV-cache economics
| DeepSWE | Kimi K3 | |
|---|---|---|
| Sandbox | Docker containers; dockerd crashed under thousands of containers until a Kubernetes orchestrator (200-core nodes, cluster autoscaler) was added | AgentENV Firecracker microVMs: checkpoint 133 ms / resume 49 ms, pause frees CPU+RAM (agents idle ~98% of sandbox lifetime), fork for side-effect-free reward judging, 9.6× memory overcommit; 51.2M sandboxes over 1.5M images |
| Test budget | 5-minute cap (vs. 30 official) to keep rollouts affordable | Paused sandboxes cost nothing, so long-running verification is affordable |
| Rollout engine | Synchronous batch in rLLM/verl | Co-located RL with an external CPU-DRAM KV-cache pool (write-back), NVMe offload of optimizer state, auto-throttling request scheduler |
| Numerics | Separate training/inference stacks; ratio clipping absorbs the mismatch | MXFP4/MXFP8 quantization-aware training through all of RL; rollout and training share one quantization scheme — the train–inference mismatch is removed at the source |
Why it's an improvement. At 100 steps × 64K tokens, containers-plus-patience works. At thousands of tool calls × 1M tokens, the binding constraints become state persistence (a paused trajectory must resume bit-exact days later), idle-time cost, and prefix-cache hit rate — none of which Docker/K8s addresses. The QAT point is subtle but important: one root cause of RL instability that clipping heuristics paper over is the numerical gap between the inference engine that samples and the trainer that computes gradients; K3 eliminates the gap rather than absorbing it.
7. Cold start: pure RL as a statement vs. SFT+RL+distillation as engineering
Why it's an improvement (for K3's goal). DeepSWE's pure-RL-from-base was a deliberate scientific demonstration — and its SFT-cold-start attempts with Claude-generated data actually hurt. K3 optimizes for capability per FLOP instead: SFT provides the cold start, RL unlocks long-horizon execution, and multi-teacher on-policy distillation consolidates specialists without the compounding-error problem of off-policy SFT on teacher traces. At 2.8T parameters, spending RL compute to rediscover output formatting would be waste, not purity.
What DeepSWE got right that K3 kept
- Critic-free group-relative baselines (no value network) as the backbone of agentic RL.
- Dropping GRPO's reward-std division (difficulty bias) and per-response length normalization (length bias) — DeepSWE via Dr. GRPO, K3 via the K2.5 token-level objective.
- No KL-to-reference-model penalty — both replaced anchoring-to-SFT with other stabilizers.
- Excluding truncated/censored trajectories from naive reward assignment — DeepSWE masks them; K3 goes further and finishes them later.
- Executable, test-verified environments as the substrate for coding RL (R2E-Gym then; K3's sandboxed suites now).
Sources
- DeepSWE Notion post: Training a Fully Open-sourced SOTA Coding Agent by Scaling RL (Agentica × Together AI)
- Kimi K3 tech report, §4.1.2, §4.2, §5.3: arXiv:2607.24653; Kimi K2.5 report, Policy Optimization Eq. 1: arXiv:2602.02276
- DAPO: arXiv:2503.14476; Dr. GRPO: arXiv:2503.20783; R2E-Gym: arXiv:2504.07164
- Kimi-K3 model card; Kimi K3 tech blog; AgentENV (Firecracker sandbox platform)