Paper 260: evidence behind the improvement review

1. Repair the assertions before interpreting robustness

The strongest finding is in tests/Stegano/robustness_test.py. Each of its four pytest tests contains an import-failure path ending in assert True and a return. The helper also treats an exception as success:

def _run_case(fn):
    try:
        fn()
        return True
    except Exception:
        return True

The tests write separate diagnostic JSON, but measure_generated.py computes robustness from pytest’s passed/total counts. Recording an import failure in JSON therefore does not prevent the released scoring path from awarding credit.

The Stegano performance test discards the return value of lsb.reveal and asserts only positive hide/reveal durations. A successful timing run consequently does not demonstrate that the message was recovered. The Folium robustness suite has genuine assertions, so the appropriate recommendation is an audit, not a blanket claim that all suites are ineffective.

Revision: establish the exact test revisions used for the paper, correct the outcome assertions, remeasure reference baselines, and rerun affected generated repositories. Report score and ranking changes. Independently selected mutations and deliberately incomplete implementations would test whether the corrected suites distinguish plausible wrong behavior, beyond the existing two-defect sanity check.

2. Novelty lies in the combination, not the generation setting

The closest comparisons do not make RAL-Bench redundant. They narrow its contribution to a particular input, oracle, and measurement protocol. That contribution would be stronger if the paper demonstrated useful distinctions between implementations after controlling for functional completeness.

Concrete overlap: the original Commit0 paper's Table 2 lists tinydb, pyjwt and cachetools; RepoCraft's Table 1 lists requests. All four occur in RAL-Bench's Table I. This contradicts the stated repository-exclusion policy, not necessarily task-level deduplication. It also does not establish inclusion in any model's training data. Commit0's §6 provides a relevant precedent for measuring generated/reference code overlap.

3. Distinguish execution, useful work, and measurement

• Quality comparisons: the paper already assigns zero when efficiency/resource suites fail. The missing safeguard in the inspected Stegano performance test is output validation. Report both unconditional results and comparisons on a shared, functionally validated workload, including how many implementations qualify.
• Reference anchors: for a positive reference MI, Eq. 1 gives (g/b)/(1+g/b) = 0.5 when g=b. The other reference scores are 1 under successful, defined measurements. Their percentages cannot be interpreted as a common degree of software quality. The release returns zero for nonpositive reference MI, which the printed equation leaves unspecified.
• Sampling: measure_generated.py checks its deadline, calls blocking proc.stdout.readline(), then samples RSS/CPU and sleeps. A quiet process can delay both sampling and the next deadline check. Validate a decoupled sampler with a long-running process that emits no output. This concern follows from the code path; the published measurement error has not been quantified.

The Radon documentation and Arie van Deursen's MI critique give additional reasons to validate maintainability signals empirically. The short review prioritizes the directly demonstrable normalization issue rather than dismissing MI merely because it is old.

4. Align the experimental contract and the release

In §IV-A, clarification changes GPT-5.2's functional score from 41.81% to 49.72%, a 7.91-percentage-point increase; paraphrasing yields 49.19%. This supports a controlled study of which assertions require information missing from, or unclear in, the original prompt. Traceability to upstream documentation is not the same as traceability to the text supplied to the model. It is not established that particular hidden requirements caused these gains.

The Stegano manifest includes signatures, defaults, and explicit paths such as stegano/tools/bititerator.py. The generation entry point places those paths under “Required files” and requests temperature=0.2. The manuscript instead describes removing layout hints and using greedy decoding. Publish the exact table-producing configuration; do not assume that the currently released entry point is that configuration.

Run isolation needs the same provenance check. The helper named _ensure_empty_dir only creates a directory if needed. The batch runner ignores the generation-success return value before loading a per-project result file. Without external cleanup, old files or results can survive a later failed run. Fresh run directories and explicit failure records would remove that ambiguity.

The “no model exceeds 45%” finding applies to the original single-turn comparison, not all settings studied: clarified prompts exceed it, and Table VII reaches 58.76% with Live-SWE-Agent/Gemini-3-Pro. Pair task-level confidence intervals with specified token budgets, truncation rates, feedback access, and strategy costs.

Evidence boundaries

The review does not infer training contamination from repository popularity, assume that 12 tests necessarily hide unstated requirements, or equate high robustness percentages with defective tests without inspecting the assertions. The artifact findings justify repairs and controlled reruns; they do not establish the magnitude of changes to the reported results. Google research covered twelve distinct sites, eleven accessible; OpenReview's blocked page was skipped at the user's request and replaced with the primary arXiv paper and official publication records.