Review of submission #1267: "TestDesigner: Decoupling Test Design from Input Generation via Synthesized Generators"

SUMMARY

The paper generates test inputs for programming problems by having an LLM extract a "Test Design Specification" (equivalence classes, boundaries, constraints) from the problem statement, synthesize a Python input generator and validator from it, and refine both with branch-coverage feedback from a reference solution. On 1,865 curated APPS problems it reports 95.0% fault detection, ahead of CodeContests+, the official APPS tests, and direct LLM prompting.

STRENGTHS

+ Clear pipeline; the taxi-fare example makes the method easy to follow.
+ The pairwise exclusive-detection table and the G1/G2/G3 breakdown of the refinement loop are more informative than the usual aggregate numbers.
+ Candid threats section; artifact released.

WEAKNESSES

- Novelty is overstated. The TDS is the category-partition method reborn: Ostrand and Balcer's TSL (CACM 1988) already gave machine-readable test specifications from which a tool generates test frames. Not cited. LLM-written input generators appear in TrickCatcher (ACL 2025), HardTests, rStar-Coder, and property-based-test synthesis (Vikram et al. 2023). Coverage-guided refinement of generators is Zest (ISSTA 2019) and targeted PBT; coverage-in-the-prompt loops are CoverUp (FSE 2025). The claim that no prior work combines black-box design with white-box coverage guidance is false.
- The buggy set is the set of programs failing the union of the compared suites, and TestDesigner contributes 82.5 inputs per problem against 19.6-27.2 for the baselines. The comparison confounds design quality with input budget. Cap every suite at the same size.
- No precision measurement. The validator is itself LLM-generated and never checked, so an invalid input can make a correct program look buggy. CodeContests+ and HardTests report true/false positive rates against real human submissions with verdicts; such submissions exist in CodeContests and TCGBench, which undercuts the anti-crawling excuse for using only LLM-made bugs.
- 3,135 of 5,000 APPS problems (63%) were dropped; selection bias is likely.
- The pivotal ablation is missing: same model, same budget, generator synthesized directly from the description with no TDS. Without it, the central decoupling claim rests on comparison with a differently engineered system. The refinement loop matters on only 39 of 1,865 problems (+0.7 points overall).
- One model, one run, a 1.6-point margin over CodeContests+, and no variance or significance testing.

DETAILED COMMENTS

The system is competently built and the writing is clean, but the paper needs: 

(1) honest positioning against four decades of specification-based test design and against Zest-style coverage-guided generation, arguing what LLM extraction adds; 

(2) size-matched suites evaluated on an independent set of real wrong human submissions, with both detection and false-alarm rates and significance tests; 

(3) the TDS-versus-direct-generator ablation; 

(4) evidence the template scales beyond scalar tuples to strings, graphs, and arrays with cross-element constraints, where Cartesian products explode; 

(5) multi-model results and cost. Say up front that a known-correct program is required: this is a benchmark-construction technique, not general testing. Fix these and the paper becomes a solid contribution.
