Review: Paper 3126 - SpecShift: Measuring Specification Responsiveness in Code LLMs

Summary
-------
SpecShift rewrites HumanEval docstrings two ways: paraphrases that keep the meaning (Invariance Score, IS) and edits that change it (Responsiveness Score, RS, with tests derived by rule or two-model agreement), then counts how often two models disagree on a variant (DD). Five open models run on 164 problems plus 662 variants; the claim is that IS and RS separate models that pass@1 lumps together.

Strengths
---------
+ Good question; execution oracles are the right instrument.
+ The T1 derivation (re-run the reference solution under the swapped operator) is sound.
+ Honest about the 38.6% T2 acceptance rate and the n=1 off-by-one cell.
+ The matched-subset analysis (Table VI) is the right move.

Weaknesses
----------
- Prior work covers both halves. Docstring rewrites: ReCode (Wang et al., 2022), NLPerturbator (Chen et al., 2024). Semantic edits with derived oracles: Turbulence (Honarvar et al., ICST 2025), PPM (Chen et al., FSE 2024), EvoEval's "subtle" split (Xia et al., 2024). None is cited; the "first tool" claim fails.
- The two models that define the T2 oracle are two of the five under test. Table V shows the consequence: their T2 RS is 0.93 and 0.90 against 0.66 and 0.62 on T1; the other three gain far less. The paper calls constraint changes easier; the simpler reading is that the oracle came from their own greedy outputs. III.B's no-circularity claim is wrong.
- Same issue for IS. A paraphrase is kept only if Qwen-32B solves it, so its IS (0.83 vs pass@1 0.82) is protected by construction. And 492/492 acceptance is hard to believe when Qwen-32B scores 0 of 3 on many originals (std 0.35). Either the gate was not applied as described or paraphrases were resampled until one passed.
- Most T2 variants violate Definition 2: the original solution still passes T' (VII.b). There, RS is pass@1 on a reworded prompt.
- The quadrant story has no data behind it. All five models sit on the IS=RS diagonal in Fig. 3; no Memorizer, no Surface-noisy model. Qwen-7B and DS-V2-Lite differ on IS (0.56 vs 0.74) but also on RS (0.64 vs 0.80). That is one dimension.
- No noise floor. Sampling is n=3 at an unstated temperature, yet DD on the 164 originals is never reported, nor self-DD under resampling. Without those, the 14-57% disagreement cannot be credited to the variants.
- Qwen-32B gets 0.82 here; EvalPlus reports 92.1 greedy on the same problems. Ten missing points look like prompt formatting or code extraction, which here reads as surface sensitivity.

Detailed comments
-----------------
Novelty is modest. The parts exist; what is new is the pairwise DD framing, and the IS/RS quadrant is empty in your own data. Reframe around DD and the matched subsets, and compare with ReCode and EvoEval, both on HumanEval.

Fixes, by payoff:

(1) Use models outside the evaluated set for the paraphrase gate and the T2 oracle, then rerun Table V. If the oracle models' T2/T1 gap shrinks, that settles it.

(2) Report DD on the originals and self-DD under resampling. State the temperature. Greedy decoding plus HumanEval+ tests would make IS measure the docstring rather than the sampler.

(3) Enforce Definition 2: drop every T2 variant the canonical solution still passes; report how many survive.

(4) RQ4 pools five models, so r=0.80 mostly says Qwen-32B beats Mistral. Use within-model correlations.

(5) DD is a discordant-pair count; McNemar's test fits it, Mann-Whitney on IS does not.

(6) Add a less memorized benchmark (LiveCodeBench or EvoEval) so a Memorizer could appear.

Smaller things: Table III gives Qwen-32B RS 0.85, Table V 0.80. The Mistral vs Qwen-32B RS effect is r=0.75 (Table VIII), not 0.69. III.B says "three-way consensus"; elsewhere it is two models. RQ5 says T1 drives disagreement, RQ3 says T2. A non-significant difference does not "confirm" equal scales. [34] is the EvalPlus paper by Liu et al., already [15], with no GPT-4 judge. [50] is chain-of-thought, not RL from execution feedback. [7] LLMorph is an ICSME 2025 tool for NLP tasks, not code; [9] has the wrong title. The Zenodo link 404s without the token the PDF omits. Drop "supports reviewer credibility" from IV.A.
