Summary
-------
REVERT turns released paper-repository pairs into failure-recovery environments: it injects a reversible, paper-grounded fault, keeps the reference state private, and scores repairs with a frozen verifier at inference cost. A 9B model trained this way beats budget-matched baselines by 17.2 points on a new 500-task human-authored benchmark and recovers more natural incidents as a sidecar inside two autoresearch systems.

Strengths
---------
- The recovery framing fills a real gap. SUPER, PaperBench, RExBench, and EXP-Bench all run forward; nobody trains agents to work backward from a silent failure.
- Unusual methodological hygiene: frozen verifiers with preregistered precision targets, provider-family separation, leak gates, audits with reported kappa, transfer to execution-scored benchmarks.
- Disclosure ladders as a difficulty dial, and the four-certificate admission, are useful beyond this paper.
- The sidecar study measures what matters: valid experiments and final score, not patch acceptance.

Weaknesses
----------
- No repaired benchmark task is ever executed; both training and test rewards are LLM judgments.
- Every fault is one reversible edit to a working repository; real failures arrive in half-built code, several at a time.
- The mutation-testing literature is absent, and SWE-RL, R2E-Gym, and debug-gym go unmentioned.
- Likely pretraining contamination from public reference repositories is not measured.

Detailed review
---------------
On novelty: the ingredients are known. Fault seeding with hidden ground truth is mutation testing, fifty years old; reference-similarity rewards appear in SWE-RL; execution-free verifiers in R2E-Gym; judge calibration in PaperBench. The composition, though, is new and earns its place: paper-grounded consequential faults, a private reference, and a certified difficulty ladder together make one fault reusable across thousands of rollouts. I judge this a solid, moderately novel systems contribution whose benchmark may outlast its model.

Now the issues. First, the verifier is never checked against reality. GPT-5.6-Sol scores the benchmark, and GPT-5.6-Sol also built the environments and tops the leaderboard; the self-preference audit helps, but the clean fix is cheap: pick the ~50 REVERT-Bench tasks with the smallest native workloads, run them end to end, and report true recovery against verifier-accepted recovery. That one table would remove the paper's largest soft spot.

Second, "generic mutations cost 20.8 points" restages Just et al. (FSE 2014) — are mutants a valid substitute for real faults? — for agent training. Cite that literature and say what changed.

Third, contamination. The reference snapshots are public repositories of highly rated papers; a model that memorized one can diff against memory. Measure it: ask each model to reproduce the changed lines given only file paths, and stratify results by that score.

Fourth, baselines are retrained under REVERT's recipe. TMAX and SWE-Smith tuned their own; report their native-recipe numbers too, or the 17.2-point claim invites a rebuttal you can avoid.

Last, prose. The word "certificate" is made to carry four meanings; one early table would spare the reader. Cut "makes trust automatic" — trust is earned, not automated. Omit needless words.
