Review of Submission 1805: "Contamination-Resilient Benchmarking for LLM Code Generation: Methods, Datasets, and Evaluation"

Summary

The paper rewrites competitive-programming problem statements with an LLM, keeps the original test suite, and accepts a rewrite when some LLM can write a program from the new statement that passes the old tests. On APPS and Luogu this yields CRAL, 1,839 English and 1,024 Chinese problems. Five models score 0.3 to 9.7 points lower on the rewritten problems, which the authors read as evidence of contamination.

Strengths

- Reusing the original tests as the equivalence oracle is cheap and avoids the LLM-judge step DyCodeEval relies on.
- The Luogu subset, with seven difficulty tiers and ten algorithm tags, is a real addition.
- The false-positive experiment (RQ1) asks the right question, and the scripts are released.

Weaknesses

- The rewritten set is selected by the very models it evaluates, so the measured drops are biased downward.
- Nothing separates contamination from ordinary paraphrase brittleness.
- Rewriting the statement does not remove contamination that lives in memorized solutions.
- The adequacy assumption behind the oracle is stated as a definition, not tested.
- Drops are averaged over untransformed problems, no significance test, and the conclusion overstates RQ2.

Detailed review

Novelty. Rewriting problems to dodge memorization is well trodden: EvoEval (2024) rewords HumanEval and measures with execution, ReCode (2022) paraphrases docstrings, DyCodeEval (ICML 2025) rewrites with a scenario and validates with an LLM plus humans. The new part is swapping the LLM validator for a pass-the-old-tests check and scaling to two competition sources. A useful engineering step, not a new idea; say so and drop "first work" from Section II-C. CodeMorph (ICSE 2025 Companion) perturbs code, not statements.

Selection bias. Datasets_Construction_CRAL-APPS.py runs the same five evaluated models on each rewrite, accepts the first rewrite that any of them solves, retries up to five times, and stores the acceptance run as the transformed-set result (futures = futures1). This deflates every drop, and most for the strongest model, which is most often the accepter. o4-mini's small drops (2.4 and 1.2 points) are what this artifact predicts. Fix: validate with a disjoint model set, or report drops only on problems where the given model was not the accepter, and rerun the evaluation.

Contamination versus brittleness. A few-point loss after paraphrase is the known robustness effect from ReCode and EvoEval, and it appears on unseen problems. A flat compilation rate says nothing about comprehension. Two controls would settle it: (a) run the same pipeline on problems released after the models' cutoffs (LiveCodeBench slices, or 2026 Luogu problems) and compare drops; (b) as DyCodeEval does, fine-tune an open model on the APPS originals and check whether CRAL scores stay flat. Yang et al. (2023) and Spiesberger et al. (Feb 2026) show that rephrased test items still transfer their advantage, so (b) may not go the way the title assumes. Riddell et al. (2024) locate most code contamination in memorized solutions, which the rewrite leaves untouched.

The oracle. Definition 4 (pass iff equivalent) holds for no finite suite; EvalPlus cuts HumanEval pass@k by 19 to 29 percent with added tests. The Case 1/Case 2 analysis omits the realistic failure: the rewrite drops or blurs a constraint, and a model that has memorized the original fills the gap with the memorized solution. RQ1 never tests this; its variants are minimal, well-specified edits. Add an adversarial set in which a constraint or edge-case rule is deleted. And 0 of 200 gives a 95 percent upper bound near 1.5 percent, about 26 problems in CRAL-APPS. A cheap strengthening: generate fresh inputs from the rewritten statement and compare the candidate against the APPS reference solutions.

Reporting. Table VI seems to average over the 100 and 331 untransformed problems, which contribute zero change; report drops on the transformed subsets. Give pass-to-fail and fail-to-pass counts per model with a McNemar test. RQ2 calls the effect "moderate"; the conclusion says "severe". Section III-C.3 analyzes N code samples per rewrite, but the script draws one sample per model and N counts rewrites. "0.45" should be 0.4^5. Low embedding similarity (0.659) is no virtue for problems meant to be equivalent. State that DeepSeek is both rewriter and subject.
