Review: "Who Catches What?" (ICLR 2027 submission)

Summary

The authors obtained the full event log of an eight-week deployment in which claude-fable-5 wrote code and gpt-5.6-sol reviewed it read-only under a budget cap: 329 tasks, 440,738 events, and 1,115 reviewer findings extracted from the builder's end-of-task narratives. They audit protocol fidelity from the log, report cost and disposition counts, and reproduce three defects.

Strengths

- The corpus is a real asset: per-call, per-cost internals of a two-vendor review loop on shipped code, released with a recomputation script.
- Auditing the protocol before trusting the narrative is the right order; the write-leakage check convinces for the 84 inline tasks.
- The three cases (coverage-gate off-by-one, delete-intent log, eval() on model output) are concrete and instructive.

Weaknesses

- The abstract and introduction report numbers that Section 6 declines to compute.
- The cost share rests on 7 tasks and $93.
- Dispositions are builder self-reports, extracted by an unnamed model; the promised spot checks sit in an empty appendix.
- Conclusion, Ethics, Reproducibility, and Appendix are bare headings.
- Several 2026 bibliography entries have wrong author names and paraphrased titles.
- Related work omits the code-review and buggy-fix literature behind the "dynamics with no published counterpart."

Detailed review

Two papers are stapled together. The abstract and Sections 1 and 7 believe the numbers; Section 6 does not. The abstract says the builder "repaired 89% and rejected 1%"; Section 6.3 says the counts "do not form a partition" and refuses ratios. The abstract says the reviewer "consumed about 9% of task cost" and Section 1 quotes a $3.25 narrative median; Section 6.2 says the share comes from seven tasks and that eleven narrative-dollar figures are budget caps. Section 1 says most later-round findings were "introduced by the fixes"; Section 6.5 says the extraction cannot tell. "Test-blind" in the abstract means a defect the passing tests missed; in 6.4 it includes suites never run. Contribution 2 says spend "stayed far under the budget cap"; 6.2 says cap compliance was never established. If Section 6 is right, rewrite the abstract to match it.

The abstract mentions "a smaller, earlier trace window ... under a different model pair." It appears nowhere else. Add the section or cut the sentence.

"Complete event log" overstates. Subagent reviews (202 of 283 tasks) leave no per-call events, so the 0.14% leakage covers only inline tasks; delegated returns are cut to 3,021 characters, and 44.8% of candidate passes end truncated, timed out, errored, or unrecorded. Say so in the abstract.

Which model read the narratives? If it was claude-fable-5, the builder graded its own homework. Name it, hand-label 60 random tasks, report agreement, and fill the appendix.

The title asks who catches what and never answers. Section 6.3 lists categories without counts. Code a random 200 findings into a fixed taxonomy and compare with the human baseline: Mantyla and Lassenius (TSE 2009) and Beller et al. (MSR 2014) found about 75% of human review findings concern evolvability.

The "no published counterpart" claims fail. Fixes that introduce bugs is Yin et al. (FSE 2011): 14.8 to 24.4% of OS fixes were incorrect, 39% for concurrency, your afc1c52d case. Reviewers catching what tests miss is the premise of Bacchelli and Bird (ICSE 2013). Present both dynamics as replications. Also missing: Monperrus (arXiv 2606.13175), Smith et al. (FSE 2015), and ControlArena.

Verify the bibliography against arXiv. Pinto 2607.06596 is Lucas (not Daniel); Jha 2608.16190 is Anik (not Anand); Karakaya et al. 2604.24525 has three wrong first names and appeared at EASE 2026; the Pinto and Helwig titles are paraphrased. Pinto's effect held for trained monitors only, so it says little about a prompted reviewer. Kim's tap paper (2606.14445), dismissed as "one aggregate metric," contains the homogeneous-pair comparison you lack (53.1% vs 69.8%). Credit it.

Novelty: the corpus and the protocol-fidelity audit are new. The dynamics are old findings in a new setting, and the headline statistics are unsupported by the paper's own results. Section 5 restates Section 4 and contains no experiments; merge them.

Recommendation: reject as submitted. I would welcome a resubmission whose abstract agrees with Section 6, with the empty sections written and the bibliography checked.
