Summary

RAL-Bench asks a model to generate a complete Python application from a short requirement plus the expected public interface, then grades the result with black-box tests validated against a pinned reference repository, alongside five ISO/IEC 25010-inspired quality signals. Sixteen LLMs, evaluated zero-shot and single-turn, all score below 45% on functional tests. The execution is careful, but the benchmark overlaps prior work more than the paper admits, and two metric choices undercut the headline claims.

Strengths

+ Validating every test against a pinned reference commit, and recording the documented behavior each test traces to, is sound oracle discipline.
+ The failure taxonomy over 446 generated repositories is genuinely useful; the finding that most failures occur after the repository becomes executable is credible and actionable.
+ Rerun stability, prompt-sensitivity, and agentic supplements anticipate the obvious objections.

Weaknesses

- Section II claims repositories used in public benchmarks were excluded. This is false: pyjwt, requests, click, loguru, pypdf, cachetools, and tinydb appear in Commit0 (Zhao et al., ICML 2025), and requests and pyjwt in SWE-bench and SWE-bench-Live.
- No contamination analysis, though every subject is a famous repository and the sample requirement names its target ("the Stegano project"). A model can regurgitate memorized code; the "SWE-Bench Illusion" studies show this inflates exactly this kind of score.
- The non-functional scores are not on a common scale. A repository at parity with the reference earns 0.5 on maintainability (Eq. 1: g=b gives 0.5) but 1.0 on security, efficiency, and resource. RQ1's conclusion that "maintainability, efficiency, and resource scores are lower" than security and robustness is partly an artifact of this.
- Non-functional suites are graded independently of functional correctness. A near-empty repository that installs and imports gets a perfect security score, since it has fewer high-risk findings than the reference. Condition these scores on functional pass, or report them only for functionally passing tasks.
- Related work omits Commit0, DevBench (Li et al., 2024), ProjectEval, ChatDev/SRDD, and NoFunEval (COLM 2024), which established non-functional evaluation of code LMs.

Detailed comments

The novelty is modest. Spec-to-repository generation with reference-derived tests is Commit0's setting; non-functional evaluation is NoFunEval's. The defensible contribution is the combination under one executable protocol, and the reference-normalized scoring. Frame it that way, and compare against Commit0 directly, since seven tasks share its subjects.

The defect-injection check is circular: defects were injected into behaviors the tests cover, so 100% detection tells us little. Inject defects sampled from real bug-fix commits instead. Eleven functional tests per task is thin for celery or mitmproxy; say what fraction of documented behavior they touch.

GPT-3.5-Turbo, with a small context window, outscoring GPT-4o on functional tests wants an explanation; it suggests shallow tests or memorization. And the abstract's "no model exceeds 45%" holds only single-turn; your own Table VII has Live-SWE-Agent at 56-59%. Scope the claim.
