Review of ICSE 2027 submission 1479 (CECouple)

Summary

The paper keeps DPO unchanged and changes the training pairs. The chosen response is the fastest verified-correct program in a sampled pool; the rejected one is a failing program whose static similarity (AST histogram, control-flow signature, identifiers, API/constant sets) to it is closest to tau=0.3. Trained on 2,147 MBPP/LBPP-derived tasks, it reports Pass@1 and runtime gains on Mercury, EffiBench, HumanEval(+), MBPP(+), ClassEval, RepoExec and HumanEval-X.

Strengths

- Simple and cheap: one DPO run, no reward model, no new loss.
- Runtime is measured with 50 repetitions, IQM, and per-task confidence intervals (Fig. 3). Few papers bother.
- Ablations cover both sides of the pair; the tau and view-weight sweeps are useful.
- Contamination check with RapidFuzz plus Dolos.

Weaknesses

- Base MBPP numbers are off by 40 to 50 points, so the MBPP results, and probably the EffiBench Pass@1 jumps, cannot be trusted.
- Efficiency numbers are not comparable across tables; ablations rest on 9 to 19 tasks.
- The motivating "conflict" is not a conflict.
- "Structural degradation" of the negative is asserted, never measured.
- Modest novelty; the closest prior work is missing.
- Evaluation on very old and easy benchmarks such as HumanEval and MBPP.

Detailed review

Start with Table III. Qwen2.5-Coder-7B base scores 27.30 on MBPP and 30.42 on MBPP+. The Qwen2.5-Coder report (Table 3) gives 76.9 and 62.9 for the same checkpoint. DeepSeek-Coder-6.7B: 19.47/22.75 here, 70.2/56.6 published. DeepSeek-Coder-1.3B: 11.24/14.28 versus 55.6/46.9. Your HumanEval numbers match, so the MBPP prompt is broken, most likely missing the function signature or test example. Base MBPP+ exceeding MBPP in all five rows is another symptom. With a harness this far off, any DPO run on MBPP-derived seeds will "fix" it by teaching format, which is what 19.47 -> 35.14 looks like. The EffiBench jumps (34.90 -> 55.70, 36.50 -> 57.30 from about 2k LoRA triplets) and +10 Java Pass@1 from Python-only training deserve the same suspicion. Classify base failures into name/format errors versus wrong output.

Second, the "all-method common-correct set" changes with the methods in each table. CECouple on DeepSeek-1.3B/HumanEval has Eff.Gain 10.07 (Table III), 2.11 (IV), 3.79 (V), 2.79 (VI) and 11.35 (VII). Same model, same config. The Eff.WinRate denominators in Table IV are 19, 13, 11 and 9 tasks; StarCoder-1B in Table III has 17, 11, 23 and 19. You cannot draw tau or view-weight conclusions from nine tasks. Use pairwise base-vs-method sets, print |Pcc| in every table, and move the ablations to Mercury or EffiBench, whose inputs actually stress runtime (EvalPerf, ENAMEL and COFFE show HumanEval/MBPP tests do not). Fig. 4 illustrates the problem: dict versus list.index over ten fixed digits is not an asymptotic change.

Third, Fig. 1. A correct-but-slow program that beats an incorrect one and loses to a faster correct one is a consistent total order, exactly Eq. 1. DPO fits a scalar Bradley-Terry reward, so Dfunc plus Deff encodes this order without contradiction. The real argument is that a near-miss negative gives a harder gradient than a random failure, which is Focused-DPO's argument. Reframe, and build the decoupled baseline from your own pool. It is unclear whether the five external baselines were retrained on your 2,147 tasks. EffiCode falling to 25.00 Pass@1 on Mercury (base 50.39) looks like a failed reproduction, not a weak method.

Fourth, Eq. 9 restricts negatives to failing programs, so nothing checks that the negative is slower or structurally worse. And with AST cosine weighted 0.4, two solutions to one task usually score well above 0.3; your Fig. 2 negative would too. Plot the similarity distribution, show that tau=0.3 picks anything resembling a near miss, and report how often the negative has deeper nesting than the positive. Fig. 3 also shows 17/62, 20/70 and 20/66 tasks getting reliably slower. Analyze them.

On novelty: fastest-correct positives (Code-Optimise, EffiCode, Afterburner), PageRank test ranking (CodeDPO) and near-miss negatives (Focused-DPO, CodeLutra) all exist. The new part is the four-view similarity target for picking the negative: a reasonable heuristic, not a new formulation. Cite and discuss ACECode (joint efficiency/correctness reward, Dec 2024), Code-Optimise as the direct decoupled counterpart, MODPO, PLUM, ECCO, ENAMEL, EvalPerf and COFFE.

Fix the MBPP harness and rerun. The method may still hold up on Mercury and EffiBench; that would be a decent paper.
