Review of Submission 4410: "VeriSAGE-CC: Schema-Guided and Verification-Driven Generation of Competitive Coding Problems"

Summary

VeriSAGE-CC turns a seed contest problem into a new problem package. It extracts a four-part schema (input structure, constraints, objective, invariants), applies a transformation rule, refines the statement with an LLM critic, builds validator, brute force, reference solution and checker, and adds tests that kill a pool of wrong solutions. On 194 TACO seeds it beats AutoCode, UniCode and direct prompting on rubric scores; on 200 TestCase-Eval problems targeted tests raise wrong-submission coverage from 0.929 to 0.975.

Strengths

- The whole package is produced and cross-checked. Most generators stop at statement plus reference solution.
- Checking the checker in both directions (accept trusted outputs, reject constructed illegal ones) is the right test for constructive and multi-answer problems.

Weaknesses

- The two new parts, the schema and the rule engine, are never shown: no rule list, no worked quadruple, no generated problem.
- The generated suites reject 26% of human-accepted solutions, and adding untargeted tests makes it worse; the difficulty numbers inherit that problem.
- Full-pipeline scores differ between Tables I and VII on the same seeds by about 3 points, the size of several claimed gaps.
- Generator model, judge model, rater count, agreement, rule count, yield and cost are absent.
- Stages 4 and 5 are close to AutoCode, Polygon practice, CodeContests-O and CodeHacker.

Detailed review

Novelty. AutoCode (Zhou et al., 2025) already has validator, generator and checker selection, brute-force dual verification, hack-style tests, mutant programs to select checkers, and seed-based variants. CodeContests-O refines tests from the behavior of incorrect solutions; CodeHacker (Feb 2026) calibrates its validator and checker with adversarial probes; Polygon has long let setters register solutions that must fail. What remains new is the quadruple, the rule planner with its distance D, and the negative-case checker test. Enough for a paper, but none of it is shown. Add a table of rules (count, applicability condition, one instance each), one seed with Q, Q', D and the final statement, and the fraction of seeds that die at each stage. EvoEval (Xia et al., 2024) evolves problems with named operators and belongs in the comparison.

Test correctness. On TestCase-Eval the non-targeted suite accepts 74% of correct human submissions and the size-controlled suite 72%. Expected outputs come from the pipeline's own reference solution, so in a quarter of problems that solution, the checker or the time limit is wrong. This is the central promise of Stage 4, and here it fails. Break the 26% down by verdict (WA or TLE) and by test class (brute-force-verified small versus reference-only large); give the size of Table V's "successful augmentation" subset; and run AutoCode's and HardTests' suites on the same problems. AutoCode reports 99% agreement with official verdicts.

Difficulty. Pass rates fall from 85.7% on small tests to 76.4% on large tests, so some "hard" problems may have wrong large-case answers. Report whether each failing model solution also fails a brute-force-verified test. That number decides whether 49% hard is real.

Judge reliability. Table I (84.64 / 79.63) and Table VII "full" (87.41 / 82.44) describe the same system on the same seeds. Explain the gap; if it is judge variance, give intervals, since the UniCode solvability gap and the no_rules gap sit inside that spread. Name the judge and generator models. AutoCode found little correlation between LLM and expert scores; yours track within five points in every cell. How many raters, of what rating, scored how many problems each, with what kappa, and did they see the LLM scores?

Novelty metric. D is computed on LLM-written labels, so a paraphrase counts as distance. Report D between a problem and a reworded copy of itself (the floor) and between random pairs (the ceiling); 0.35 must sit well above the floor. Re-extract Q from the final statement instead of reusing the planned Q'. Check each generated problem against the 9,000-problem corpus for near-copies.

RQ3. Constraints show R² 0.53 and 36% new labels in the last decile. That is not convergence. Claim closure for I and O only.

Smaller points. Justify the weights or test their sensitivity. Report cost per accepted problem. Release the code and the 194 packages.
