Review of "KS Gov: A Five-Layer Coding Agent Measured on Its Own Task Database"
ICLR 2027

Summary

KS Gov is a coding agent built from five nested Python classes (ReAct loop, summary-based continuation, tools with sub-agents, chat persistence, a worktree per task) and a 3,161-word prompt of engineering rules. The paper mines six months of the author's own task database, runs a six-arm prompt ablation on ten small tasks with hidden tests, adds a read-only reviewer from a second vendor whose findings are checked by probes, and reports two case studies. Every arm passes every hidden test; the reviewer reports 21 findings, 17 judged real.

Strengths

- The paper is candid. Section 4.1 says the read-before-edit rule is "enforced by nothing", 4.2 says the hidden tests were incomplete, 4.5 says the reviewer's model is not logged. The numbers I recomputed from the released JSON (21/8/9/4, 15/12/11, 2.2x, 2.3x, 0.6%) match.
- Probing each reviewer finding on the other arms' artifacts and on the reference solutions is the right instinct. Few papers on LLM review check whether a reported defect exists outside the reviewed diff.
- Continuation counts, steering, unread edits, mock use, and edits that follow a review verdict (372 of 477 against 48 of 113) are not things a benchmark shows.
- The 2026 related work is read, and Kim's homogeneous-pair control is reported correctly.

Weaknesses

- The prompt ablation cannot detect an effect. Fixtures are 23 to 44 lines and all 120 cells pass. Zhang et al. (arXiv 2604.11088) and Khatri (2607.27250) asked the same question on SWE-bench Verified subsets and real repositories, chose tasks in the band where the agent sometimes fails, and reported equivalence bounds. Both are uncited. On a ceiling, a flat pass rate is not a finding.
- "Real defect" is the authors' own label, assigned after reading the reviewer's report. Eleven of the 17 are present in the authors' reference solutions, and by my count of results.json five of the eight "specification violations" are too (relative_dates, kvstore_ttl, token_bucket NaN, run_cmd_timeout, todo_sqlite). Either the spec was ambiguous or the reference is wrong. The count needs a second rater blind to arm.
- The review arm is confounded with compute. It ran 26 steps against 9 at 2.3x the cost. Without a same-vendor reviewer or a budget-matched self-check, the data show that a second pass helps, and nothing about the second vendor. Panickssery et al. (NeurIPS 2024) and Huang et al. (ICLR 2024) supply the motivation for a different model; the paper cites neither.
- The 0.6% unread-edit rate headlines the abstract, yet Section 4.2 shows the rule sentence has no effect (removing it produced zero unread Edits), and Edit needs an exact string from the file. Claude Code, whose tool names the paper borrows, refuses Edit and Write on an unread file. A tool property is being reported as prompt compliance.
- The no-mocks rule has no justification. Spadini et al. (MSR 2017) is the standard reference on when practitioners mock and why; the paper's own 158 monkeypatch uses show the agent needed doubles anyway.
- The small-codebase thesis has a smaller precedent. mini-SWE-agent is about 100 lines and reports above 74% on SWE-bench Verified. The paper adds browser, memory, decision, and dispatch tools and measures none of them.
- Ma et al. (2606.28430) describe the paper's central observation, hidden tests passing while defects remain, and are uncited. So is Li et al.'s AIDev (2507.15003), 456,000 agent pull requests, the contrast to a one-developer database.

Detailed review

Novelty. Each mechanism has a public precedent, and the paper says so. The new part is the measurement, a framework that logs its own tool calls for six months and a reviewer study in which each finding becomes a probe. That is a methodological contribution; the results built on it are thinner. The ablation is a null on tasks that cannot register an effect. The reviewer result rests on the authors' reading of "defect" and on a design that cannot separate vendor from effort.

Look at the probe table. relative_dates probe 1 fires on all ten non-review artifacts and on the reference; the defect is that a ValueError message uses {text!r}. dupfinder probe 3 needs a 100-level tree under a 64-descriptor limit. These are the findings of a diligent reviewer with time to spend, and a developer would close half of them as won't-fix. The reviewer is still useful; but "17 real defects" should read "17 reproducible behaviors, of which N a blind rater called defects."

The confound is the thing to fix first. The review arm doubles the developer's steps because the developer must act on the report. A fair test has four arms at one budget: no review; the developer re-reads its own diff under the same instruction; a same-vendor reviewer; the cross-vendor reviewer. Judge findings blind to arm. Kim's 69.8% against 53.1% is the only datum here, and it is observational.

What I would do.

1. Choose ablation tasks by discriminative power, as Zhang et al. and Khatri do. Keep tasks on which the lite arm passes 30% to 70% of repetitions, and report an equivalence bound.
2. Run the four-arm review design with a second rater and agreement statistics on the 21 categorizations; release the twelve hand-checked verdicts.
3. Move read-before-edit into the Edit and Write tools, report the change, and drop the 0.6% from the abstract.
4. Justify the no-mocks rule with a measured effect on test quality, or present it as a preference.
5. Cite Zhang, Khatri, Ma, Panickssery, Huang, Li, Spadini, Cihan et al. (ICSE-SEIP 2025, LLM review cost and false positives in industry), and mini-SWE-agent.
6. Move the case studies to an appendix. Single runs against untuned baselines, with framework and model inseparable, add no evidence.

Recommendation: reject, and encourage resubmission. The instrument is worth building on. The experiments run through it do not yet support the claims, and two of the headline numbers (the 0.6% and the 17) measure something other than what the abstract says.
