Review of Submission 1048: "Shifting Testing Left for Repository-Level Code Generation via Double-Blind Cross-Validation"

Summary

DECO generates a repository from a natural-language requirement along two tracks: one writes code, the other writes tests, each reading only the requirement. An adjudicator cross-checks the tracks at the requirement, architecture, and code stages, so a misreading is caught before it spreads. On RealBench and DevEval, DECO beats seven baselines, including Claude Code.

Strengths

- The diagnosis is right and matters: a test grown from a wrong implementation blesses the wrong behavior. The nlsh example makes it concrete.
- Validating the architecture by simulating requirement-level tests against bare signatures is the paper's best idea, and the ablation shows this stage carries most of the gain.
- The impl-times-two control (a second implementer instead of a tester) is exactly the experiment a skeptic would ask for.
- Honest cost accounting and a useful taxonomy of the 332 architecture faults.

Weaknesses

- The novelty claim is broader than the evidence.
- Oracle migration can reward incomplete implementations.
- "Double-blind" is not double-blind.
- One commercial agent stands in for all agentic baselines.
- Single runs, small samples, no variance for the main tables.
- The prose needs a hard edit.

Detailed review

Novelty. Tests derived from intent rather than code are not new. TiCoder (Lahiri et al.) formalizes intent as tests; AgentCoder's test designer is deliberately decoupled from its programmer; CoCoEvo, which you cite, already grows programs and tests from the description alone. The claim that existing work generates tests only from the implementation is wrong as stated. What is new here: validating pre-code artifacts by simulated execution of requirement-level tests, and the de-sourcing adjudication protocol. Rewrite the introduction around that delta; it is enough.

Measurement. In the requirement-only setting, migration drops any reference test that does not resolve against the generated interface. A method that omits a feature loses the very tests that would fail it. Report the drop rate per method and score dropped tests as failures; without this, RQ1 may reward incompleteness.

Blindness. The test track reuses the implementation's skeleton, derives function-level tests from the validated implementation contract, and signatures are merged. The blindness is partial and asymmetric. Name it accurately or enforce it.

Adjudicator. You cite evidence that LLM execution reasoning is coarse [38, 39], yet never measure your adjudicator. Hand-label a sample of verdicts and report false accepts and rejects.

Baselines and statistics. Claude Code run as one end-to-end call is a weak configuration; add SWE-agent or OpenHands. With 56 + 9 projects and one run each, per-level cells rest on as few as nine projects. Run multiple seeds and give significance for the main tables, as you did for the ablation. RealBench shares authors with DevEval and SCoT; state the relationship.

Writing. "generate through two decoupled implementation and test track," "compatable," "Base on these two setting," "Both dataset provides." The abstract says pp; the conclusion says %. Edit throughout.
